Image display method, image display device, and program

Abstract

In an image display method for displaying a result of detecting a cell region from a stained image based on an immunostaining method, a display mode of a screen can be switched between a first display mode and a second display mode. In the first display mode, a first display image group including two or more superimposed images in which one dyed image in the dyed image group and the cell detection image are superimposed for each of two or more fields of view among the plurality of fields of view is arranged and displayed. In the second display mode, a second display image group including two or more superimposed images in which two or more dyed images in the dyed image group are superimposed on the cell detection image for one of the plurality of fields of view is arranged and displayed. As a result, it is possible to facilitate operator-based verification of the detection result of the cell region.

Description

Technical Field

[0001] The present invention relates to a technique for displaying the results of detecting cell regions from a staining image based on an immunostaining method.

[0002] [Reference to related applications]

[0003] This application claims the benefit of priority from Japanese patent application JP 2023-025321, filed on February 21, 2023, and incorporates all disclosure of that application into this application. Background Art

[0004] In recent years, in medical research and biological research, for the purpose of clarifying the mechanism of disease, the mechanism of organism, the mechanism of action of drug, etc., the analysis of single cell units of organism-derived specimens such as tissue specimens and cell specimens is implemented. For example, in immunostaining analysis, quantitative and qualitative analysis is performed by staining organism-derived substances such as proteins and measuring the staining state. In Japanese Patent Publication No. 2022-37567 (Document 1), it is proposed that in the stained image obtained by photographing the stained tissue specimen, the technology of determining the cell area by using the learning completion model produced by machine learning is proposed. In addition, in Japanese Patent Publication No. 2021-117886 (Document 2), it is proposed that when displaying the stained image of the tissue specimen, the stained image and the processed image obtained by processing the stained image based on the doctor's pathological findings are displayed on one screen, thereby assisting the doctor's diagnosis operation.

[0005] However, when using a learned model created through machine learning to detect cell regions from a stained image, as described in Reference 1, it is necessary to visually confirm in advance that the learned model has the desired detection accuracy. This confirmation of detection accuracy includes, for example, confirming that the target cells can be detected at a certain ratio or above (i.e., confirming the detection accuracy in terms of quantity), confirming that no analytically important cell types are missed even though the number of cells is small (i.e., confirming the detection accuracy of the stained image using a specific staining method), etc.

[0006] The operator verifying detection accuracy visually compares the cell region detection results of the learned model with stained images obtained using various staining methods across multiple (e.g., a dozen to 100+) visual fields set for each of multiple tissue specimens. Consequently, the operator compares numerous images from different viewpoints tailored to the required level of detection accuracy, requiring considerable time to verify the cell region detection results. Summary of the Invention

[0007] The present invention relates to an image display method for displaying the result of detecting a cell region from a stained image by immunostaining, and its object is to facilitate verification of the detection result of the cell region.

[0008] A first aspect of the present invention is an image display method for displaying the results of detecting a cell region from a stained image obtained by an immunostaining method, comprising: a) preparing, for each of a plurality of fields of view on a specimen, a staining image set including a plurality of staining images of the specimen obtained by the immunostaining method; b) preparing, for each of the plurality of fields of view, a cell detection image representing a cell region detected from the staining image set using a predetermined detection method; and c) displaying the staining image set on a screen so as to overlap the cell detection image. The display mode of the screen in step c) is switchable between a first display mode in which a first display image set is displayed side by side on the screen, the first display image set including two or more superimposed images formed by overlapping one staining image from the staining image set with the cell detection image for each of two or more fields of view; and a second display mode in which a second display image set is displayed side by side on the screen, the second display image set including two or more superimposed images formed by overlapping two or more staining images from the staining image set with the cell detection image for each of the plurality of fields of view.

[0009] According to the present invention, it is possible to easily verify the detection results of the cell region.

[0010] In mode 2 of the present invention, according to the image display method of mode 1, the process c) includes: d) a process of setting the display mode of the screen as the first display mode and displaying the first display image group on the screen; e) a process of selecting an image from the first display image group displayed in the process d); and f) a process of setting the field of view corresponding to the one image selected in the process e) as the one field of view of the second display mode, switching the display mode of the screen from the first display mode to the second display mode, and displaying the second display image group on the screen.

[0011] In mode 3 of the present invention, according to the image display method of mode 2, the two or more superimposed images included in the second display image group are displayed in the f) step with the same display magnification and display range as the display magnification of the one image selected in the e) step and the display range on the specimen.

[0012] In aspect 4 of the present invention, according to any one of aspects 1 to 3, in the image display method, in the second display mode, display magnifications of the two or more superimposed images included in the second display image group can be changed in a linked manner.

[0013] A fifth aspect of the present invention is an image display method according to any one of aspects 1 to 3 (or any one of aspects 1 to 4), wherein, in the second display mode, the display range on the specimen of the two or more superimposed images included in the second display image group can be changed in a linked manner.

[0014] In mode 6 of the present invention, according to any one of modes 1 to 3 (or any one of modes 1 to 5), an image display method is provided, wherein, in the second display mode, a position indication portion that is respectively displayed on the two or more superimposed images included in the second display image group and indicates the position on the image can move in conjunction.

[0015] In mode 7 of the present invention, according to any one of modes 1 to 3 (or any one of modes 1 to 6), the image display method is configured such that, in the first display mode, the display magnifications of the two or more superimposed images included in the first display image group can be changed in a linked manner.

[0016] In mode 8 of the present invention, according to any one of modes 1 to 3 (or any one of modes 1 to 7), an image display method is provided, wherein, in the first display mode, the processing of switching the one staining image overlapping with the cell detection image to other staining images included in the staining image group is performed in a linked manner in the two or more fields of view.

[0017] In mode 9 of the present invention, according to any one of modes 1 to 3 (or any one of modes 1 to 8), the image display method is characterized in that the detection method in step b) is a detection method using a learning model produced by machine learning.

[0018] In mode 10 of the present invention, according to any one of modes 1 to 3 (or any one of modes 1 to 9), an image display method is provided, wherein, for each of the multiple fields of view, another cell detection image representing a cell region detected from the staining image group by another detection method is prepared. The display mode of the screen in step c) can be switched between the first display mode, the second display mode, and a third display mode, wherein the third display mode is to arrange and display a third display image group on the screen, the third display image group including a superimposed image formed by superimposing one staining image from the staining image group and the cell detection image for one field of view among the multiple fields of view, and another superimposed image formed by superimposing the one staining image and the other cell detection image for the one field of view.

[0019] In aspect 11 of the present invention, according to the image display method of aspect 10, in the third display mode, display magnifications of the superimposed image and the other superimposed images can be changed in conjunction with each other.

[0020] In aspect 12 of the present invention, according to the image display method of aspect 10 (which may be aspect 10 or 11), in the third display mode, the display ranges on the specimen of the superimposed image and the other superimposed images can be changed in a linked manner.

[0021] In mode 13 of the present invention, according to the image display method of mode 10 (or any one of modes 10 to 12), in the third display mode, the position indication portion displayed on the superimposed image and the other superimposed image and indicating the position on the image can move in conjunction.

[0022] Mode 14 of the present invention is an image display device that displays the results of detecting a cell area from a staining image based on an immunostaining method, comprising: a display unit that displays an image; a storage unit that stores a staining image group of multiple staining images and a cell detection image, the staining image group including multiple staining images of the specimen obtained by the immunostaining method for each of multiple fields of view on the specimen, the cell detection image representing the cell area detected from the staining image group by a prescribed detection method for each of the multiple fields of view; and a display control unit that causes the staining image group and the cell detection image to overlap and display them on the screen of the display unit. Through the control of the display control unit, the display mode of the screen can be switched between a first display mode and a second display mode. The first display mode is to display a first display image group arranged on the screen, and the first display image group includes two or more superimposed images formed by overlapping one staining image in the staining image group with the cell detection image for each of two or more fields of view among the multiple fields of view; the second display mode is to display a second display image group side by side on the screen, and the second display image group includes two or more superimposed images formed by overlapping two or more staining images in the staining image group with the cell detection image for one field of view among the multiple fields of view.

[0023] Mode 15 of the present invention is a program for displaying the results of detecting a cell area from a staining image based on an immunostaining method, and the program is executed by a computer, thereby performing the following steps: a) preparing a staining image group containing multiple staining images of the specimen obtained by the immunostaining method for each of multiple fields of view on the specimen; b) preparing a cell detection image representing the cell area detected from the staining image group by a prescribed detection method for each of the multiple fields of view; and c) displaying the staining image group and the cell detection image on a screen in an overlapping manner. The display mode of the screen in the step c) can be switched between a first display mode and a second display mode, wherein the first display mode is to display a first display image group arranged on the screen, the first display image group including two or more superimposed images formed by overlapping one staining image in the staining image group with the cell detection image for each of two or more fields of view among the multiple fields of view; and the second display mode is to display a second display image group side by side on the screen, the second display image group including two or more superimposed images formed by overlapping two or more staining images in the staining image group with the cell detection image for one field of view among the multiple fields of view.

[0024] The above-mentioned object and other objects, features, aspects and advantages will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a diagram showing the structure of an image display device according to one embodiment.

[0026] Figure 2 This is a block diagram showing the functional structure of an image display device.

[0027] Figure 3 A diagram showing multiple fields of view on a specimen.

[0028] Figure 4 This figure conceptually shows a staining image group and a cell detection image.

[0029] Figure 5 This is a diagram showing the first stained image.

[0030] Figure 6 This is a diagram showing the second stained image.

[0031] Figure 7 This is a diagram showing the third stained image.

[0032] Figure 8 This is a diagram showing the fourth stained image.

[0033] Figure 9 It is a figure which shows the cell detection image.

[0034] Figure 10 is a diagram showing a first superimposed image.

[0035] Figure 11 is a diagram showing a second superimposed image.

[0036] Figure 12 is a diagram showing a third superimposed image.

[0037] Figure 13 This is a diagram showing an input and output superimposed image.

[0038] Figure 14 This is a diagram showing the flow of image display.

[0039] Figure 15 It is a diagram showing an example of a screen in the first display mode.

[0040] Figure 16 It is a diagram showing an example of a screen in the first display mode.

[0041] Figure 17 It is a diagram showing an example of a screen in the first display mode.

[0042] Figure 18 It is a diagram showing an example of a screen in the first display mode.

[0043] Figure 19 This is a diagram showing the flow of switching display modes.

[0044] Figure 20 It is a diagram showing an example of a screen in the second display mode.

[0045] Figure 21 It is a diagram showing an example of a screen in the second display mode.

[0046] Figure 22 It is a diagram showing an example of a screen in the second display mode.

[0047] Figure 23 It is a diagram showing an example of a screen in the third display mode.

[0048] Figure 24 It is a diagram showing an example of a screen in the third display mode.

[0049] Figure 25 It is a diagram showing an example of a screen in the third display mode. DETAILED DESCRIPTION

[0050] Figure 1 This diagram illustrates the configuration of a computer that functions as an image display device 1 according to one embodiment of the present invention. Image display device 1 displays the results of cell region detection from a stained image of a biological specimen obtained through immunostaining. Image display device 1 is used, for example, to verify the cell region detection results.

[0051] Figure 1 This diagram shows the structure of a computer functioning as an image display device 1. This computer has the structure of a typical computer system, including a CPU 51, ROM 52, RAM 53, a fixed disk 54, a display 55, an input unit 56, a reader 57, a communication unit 58, a GPU 59, and a bus 50. The CPU 51 performs various computations. The GPU 59 performs various computations related to image processing. The ROM 52 stores basic programs. The RAM 53 stores various information. The fixed disk 54 stores information. The display 55 is a display unit that displays various information, including images.

[0052] The input unit 56 includes a keyboard 56a and a mouse 56b for receiving input from the operator. The reader 57 reads information from a computer-readable recording medium 571, such as an optical disc, a magnetic disc, a magneto-optical disc, or a memory card. The display 55, keyboard 56a, mouse 56b, and reader 57 are connected to the bus 50 via an interface I / F. The communication unit 58 transmits and receives signals to and from devices external to the image display device 1. The bus 50 is a signal circuit that connects the CPU 51, GPU 59, ROM 52, RAM 53, fixed disk 54, display 55, input unit 56, reader 57, and communication unit 58.

[0053] In the image display device 1, a program 572 is read from a recording medium 571 via a reader 57 and stored in the fixed disk 54. The program 572 may also be stored in the fixed disk 54 via a network. The CPU 51 and the GPU 59 execute computational processing according to the program 572 while using the RAM 53 and the fixed disk 54. The CPU 51 and the GPU 59 function as a computation unit in the image display device 1. In addition to the CPU 51 and the GPU 59, other configurations may also be employed to function as a computation unit.

[0054] Figure 2 This block diagram shows the functional structure of the image display device 1 implemented by the computer 572 executing arithmetic operations and the like according to the program 572. This functional structure includes a storage unit 501 and a display control unit 502. All or part of these functions may be implemented by dedicated circuits. Alternatively, these functions may be implemented by multiple computers. Figure 2 The storage unit 501 in the functional configuration shown is mainly implemented by the RAM 53 and the fixed disk 54. The display control unit 502 is implemented by the CPU 51, GPU 59, ROM 52, RAM 53, the fixed disk 54, and their peripheral components.

[0055] The storage unit 501 pre-stores a set of stained images and cell detection images for each of a plurality of analytical fields of view on a biological specimen (hereinafter, also referred to as a "specimen"). The "multiple analytical fields of view" on the specimen (hereinafter, also referred to as a "field of view") refer to a plurality of segmented regions obtained by dividing an image of the stained specimen into a matrix. The specimen can be stained using various staining methods that use antibodies or by various staining methods that do not use antibodies.

[0056] A "stained image group" refers to a collection of multiple stained images corresponding to each of the multiple fields of view described above. These multiple stained images include multiple images obtained by staining the same specimen using multiple different staining methods. These multiple stained images include, for example, an extracted image obtained by extracting a portion of the captured image, a processed image obtained by performing a predetermined process on the extracted image, and a composite image obtained by synthesizing two or more extracted images (for example, an image in which the pixel value of each pixel in the two or more extracted images is the smallest RGB value). The number of images included in the stained image group can be appropriately set within a range of two or more.

[0057] The “cell detection image” refers to an image corresponding to each of the plurality of fields of view, and represents a cell region detected by a predetermined detection method from the staining image group corresponding to each field of view.

[0058] Figure 3 9 is a diagram showing a plurality of fields of view 90 set on a captured image (hereinafter also referred to simply as "specimen 9") obtained by capturing an image of a specimen 9 of a tissue 95. Figure 3 In the example shown, a plurality of substantially square fields of view 90 are arranged in a matrix on the specimen 9 adjacent to each other in the vertical and horizontal directions. Figure 3 In FIG, parallel oblique lines are drawn on three of the plurality of fields of view 90. In the following description, these three fields of view are also referred to as a "first field of view 90a", a "second field of view 90b", and a "third field of view 90c".

[0059] In the following description, it is assumed that only images related to the first field of view 90a, the second field of view 90b, and the third field of view 90c are displayed on the image display device 1. However, in practice, images related to all fields of view 90 on the specimen 9 can also be displayed on the image display device 1 to verify the detection results of the cell region. Furthermore, the image display device 1 does not necessarily need to be able to display images related to all fields of view 90 on the specimen 9. The fields of view 90 displayed on the image display device 1 can be appropriately set within two or more ranges.

[0060] Figure 4 : is a diagram conceptually showing the staining image groups 80a, 80b, 80c and the cell detection images 71a, 71b, 71c corresponding to the first field of view 90a, the second field of view 90b, and the third field of view 90c, respectively. Figure 4 In FIG. 8 , the dyed image groups 80 a , 80 b , and 80 c are respectively surrounded by two-dot chain lines.

[0061] exist Figure 4In the example shown, the stained image group 80a corresponding to the first field of view 90a includes a first stained image 81a, a second stained image 82a, and a third stained image 83a. The first stained image 81a is, for example, a single stained image obtained by extracting the region corresponding to the first field of view 90a from an image obtained by staining the specimen 9 using a first staining method. For example, the first staining method is a staining method using a CD20 antibody that detects the CD20 protein, one of the CD classifications. The second stained image 82a is, for example, a single stained image obtained by extracting the region corresponding to the first field of view 90a from an image obtained by staining the specimen 9 using a second staining method. For example, the second staining method is a staining method using a CD8 antibody that detects the CD8 protein. The third stained image 83a is, for example, a single stained image obtained by extracting the region corresponding to the first field of view 90a from an image obtained by staining the specimen 9 using a third staining method. For example, the third staining method is a staining method using a CD3 antibody that detects the CD3 protein.

[0062] The stained image group 80a corresponding to the first field of view 90a further includes a fourth stained image 84a. The fourth stained image 84a is a composite image obtained by superimposing the first stained image 81a, the second stained image 82a, and the third stained image 83a. Specifically, the fourth stained image 84a represents proteins stained with the CD20, CD8, and CD3 antibodies.

[0063] The stained image group 80b corresponding to the second field of view 90b includes a first stained image 81b, a second stained image 82b, a third stained image 83b, and a fourth stained image 84b. The first stained image 81b, the second stained image 82b, and the third stained image 83b of the second field of view 90b are single-stained images obtained by extracting the region corresponding to the second field of view 90b from an image obtained by staining the specimen 9 using the first to third staining methods described above. The fourth stained image 84b of the second field of view 90b is a composite image obtained by superimposing the first stained image 81b, the second stained image 82b, and the third stained image 83b of the second field of view 90b.

[0064] The stained image group 80c corresponding to the third field of view 90c includes a first stained image 81c, a second stained image 82c, a third stained image 83c, and a fourth stained image 84c. The first stained image 81c, the second stained image 82c, and the third stained image 83c of the third field of view 90c are single-stained images obtained by extracting the region corresponding to the third field of view 90c from an image obtained by staining the specimen 9 using the first to third staining methods described above. The fourth stained image 84c of the third field of view 90c is a composite image obtained by superimposing the first stained image 81c, the second stained image 82c, and the third stained image 83c of the third field of view 90c.

[0065] The first stained image 81a, second stained image 82a, and third stained image 83a of the stained image group 80a corresponding to the first field of view 90a are obtained, for example, by sequentially staining a single specimen 9 with a CD20 antibody and imaging, destaining the specimen 9, staining with a CD8 antibody and imaging, destaining the specimen 9, and staining with a CD3 antibody and imaging. Alternatively, the first stained image 81a, second stained image 82a, and third stained image 83a of the stained image group 80a corresponding to the first field of view 90a may be obtained by collectively staining a single specimen 9 with the CD20, CD8, and CD3 antibodies, and then color-decomposing the resulting multi-color image according to the color corresponding to each antibody. The same applies to the stained image group 80b corresponding to the second field of view 90b and the stained image group 80c corresponding to the third field of view 90c.

[0066] The cell detection image 71a is an image corresponding to the first field of view 90a, and represents a cell region detected from the stained image group 80a corresponding to the first field of view 90a using a predetermined detection method. This predetermined detection method is, for example, a detection method using a learned model created through machine learning. The cell detection image 71a is output, for example, as a result of inputting an input image included in the stained image group 80a corresponding to the first field of view 90a, or an input image created based on an image included in the stained image group 80a, into the learned model (i.e., an output image). This input image is, for example, the fourth stained image 84a included in the stained image group 80a.

[0067] In cell detection image 71a, the cell regions detected from the input image by the learned model are displayed in segments. Specifically, cell detection image 71a displays the cell regions detected by the learned model from the first stained image 81a, the second stained image 82a, and the third stained image 83a without distinguishing which stained image they were detected from. For example, in cell detection image 71a, the detected cell regions are displayed filled in with a predetermined color.

[0068] The above-mentioned learning model is created by performing machine learning on the initial model for detecting the cell region using a learning dataset as a set of learning stained images. This machine learning is, for example, deep learning using a neural network. Learning based on this deep learning is performed using, for example, U-Net. In addition, this machine learning can also be performed using methods other than deep learning.

[0069] Cell detection image 71b is an image corresponding to second field of view 90b, and represents a cell region detected from stained image group 80b corresponding to second field of view 90b using the same detection method as cell detection image 71a (e.g., a detection method using a learned model created through machine learning). Cell detection image 71c is an image corresponding to third field of view 90c, and represents a cell region detected from stained image group 80c corresponding to third field of view 90c using the same detection method as cell detection image 71a (e.g., a detection method using a learned model created through machine learning).

[0070] Figures 5 to 8 81a, the second stained image 82a, the third stained image 83a, and the fourth stained image 84a included in the stained image group 80a of the first field of view 90a are shown in FIG. Figures 5 to 8 , regarding the stained cells 91 in the first stained image 81 a , the second stained image 82 a , the third stained image 83 a , and the fourth stained image 84 a , the contours of the cells 91 are indicated by thin lines.

[0071] exist Figure 5 In the illustrated first stained image 81a, a relatively large number of cells 91 are dispersed in a substantially circular shape in the upper left portion of the image (i.e., the upper left portion of the first field of view 90a). Figure 6 In the illustrated second stained image 82a, a relatively large number of cells 91 are dispersed in a substantially rectangular shape in the lower right portion of the image (ie, the lower right portion of the first field of view 90a). Figure 7 In the illustrated third stained image 83a, three cells 91 are located near the lower left corner of the image (i.e., near the lower left corner of the first field of view 90a). The cells 91 in the third stained image 83a are, for example, cells of a type that is small in number but important for analysis. Figure 8 In the fourth stained image 84a shown as an example, Figures 5 to 7 Cells 91 are shown in the first stained image 81a, the second stained image 82a, and the third stained image 83a.

[0072] Figure 9 is a diagram showing an example of a cell detection image 71a of the first field of view 90a. Figure 9 In the image 71a, the cell region detected from the fourth stained image 84a (i.e., detected from the first stained image 81a, the second stained image 82a, and the third stained image 83a) by the above-described detection method is filled in black for display. Furthermore, in the cell detection image 71a, the detected cell region 92 (hereinafter also referred to as "detected cell region 92") does not necessarily need to be filled in for display; it may be displayed by other methods such as emphasizing its outline with a thick line.

[0073] In the actual cell detection image 71a, the color representing the detected cell region 92 is preferably set to a color different from the color representing the cells 91 in the first stained image 81a, the second stained image 82a, the third stained image 83a, and the fourth stained image 84a. More preferably, it is set to a color that is highly recognizable relative to the color representing the cells 91. Specifically, the color representing the detected cell region 92 and the color representing the cells 91 are preferably set so that the distance between them increases in an equal color space such as CIELAB.

[0074] As will be described later, in the image display device 1, a superimposed image related to the first visual field 90a in which the first stained image 81a, the second stained image 82a, the third stained image 83a and / or the fourth stained image 84a are superimposed on the cell detection image 71a is displayed on the display 55 (see Figure 1 ). Figure 10 3 is a diagram showing a first superimposed image 61 a obtained by superimposing the first stained image 81 a and the cell detection image 71 a . Figure 11 3 is a diagram showing a second superimposed image 62 a obtained by superimposing the second stained image 82 a and the cell detection image 71 a . Figure 12 3 is a diagram showing a third superimposed image 63 a obtained by superimposing the third stained image 83 a and the cell detection image 71 a . Figure 13 1 and 2 are diagrams showing an input-output superimposed image 64a obtained by superimposing the fourth stained image 84a and the cell detection image 71a (ie, obtained by superimposing the input image and the output image in the above-mentioned learned model).

[0075] exist Figure 13 In the illustrated input / output superimposed image 64a, most of the cells 91 in the fourth stained image 84a overlap with the detection cell region 92 and are filled with the detection cell region 92. Therefore, most of the cells 91 cannot be visually identified. On the other hand, some of the cells 91 in the fourth stained image 84a do not overlap with the detection cell region 92 (i.e., they are not detected by the learned model). Figure 13 . When the number of visually identifiable cells 91 in the input / output superimposed image 64a is relatively small, the learned model has high detection accuracy for the stained cells 91 in the fourth stained image 84a (i.e., the input image). On the other hand, when the number of visually identifiable cells 91 in the input / output superimposed image 64a is relatively large, the learned model has low detection accuracy for the stained cells 91 in the fourth stained image 84a.

[0076] Furthermore, in the detection of cells 91 based on the above-described learned model, an area that is not actually a cell 91 may be mistakenly detected as a cell 91 and filled in as a detected cell area 92 in the cell detection image 71a. The presence or absence of such over-detection can be checked by, for example, adjusting the transparency of the cell detection image 71a in the input / output superimposed image 64a to be higher, and the operator observing the presence or absence of cells 91 overlapping with the detected cell area 92. Alternatively, the input / output superimposed image 64a and the fourth staining image 84a can be displayed side by side (see FIG. 1 ). Figure 8 ), the operator observes the two images and compares them to confirm whether there is any over-detection.

[0077] exist Figure 10 In the illustrated first superimposed image 61a, most of the cells 91 in the first stained image 81a overlap with the detection cell region 92 and cannot be visually identified. On the other hand, a portion of the cells 91 in the first stained image 81a do not overlap with the detection cell region 92 and are visually identified. When the number of visually identified cells 91 in the first superimposed image 61a is relatively small, the learned model detects cells 91 with high accuracy using the antibody CD20 staining corresponding to the first stained image 81a. On the other hand, when the number of visually identified cells 91 in the first superimposed image 61a is relatively large, the learned model detects cells 91 with low accuracy using the antibody CD20 staining corresponding to the first stained image 81a. Furthermore, similarly to the input / output superimposed image 64a, the aforementioned overdetection in the first superimposed image 61a can be confirmed by adjusting the transparency of the cell detection image 71a in the first superimposed image 61a or by displaying the first superimposed image 61a and the first stained image 81a side by side.

[0078] exist Figure 11In the illustrated second superimposed image 62a, most of the cells 91 in the second stained image 82a overlap with the detection cell region 92 and cannot be visually identified. On the other hand, a portion of the cells 91 in the second stained image 82a do not overlap with the detection cell region 92 and are visually identified. When the number of visually identified cells 91 in the second superimposed image 62a is relatively small, the learned model detects cells 91 with high accuracy using the CD8 antibody staining corresponding to the second stained image 82a. On the other hand, when the number of visually identified cells 91 in the second superimposed image 62a is relatively large, the learned model detects cells 91 with low accuracy using the CD8 antibody staining corresponding to the second stained image 82a. Furthermore, similarly to the input / output superimposed image 64a, the aforementioned overdetection in the second superimposed image 62a can be confirmed by adjusting the transparency of the cell detection image 71a in the second superimposed image 62a or by displaying the second superimposed image 62a and the second stained image 82a side by side.

[0079] exist Figure 12 In the illustrated third superimposed image 63a, two of the three cells 91 in the third stained image 83a overlap with the detection cell region 92 and are therefore not visually recognizable. On the other hand, one of the three cells 91 in the third stained image 83a does not overlap with the detection cell region 92 and is therefore visually recognizable. When the number of visually recognizable cells 91 in the third superimposed image 63a is relatively small, the learned model detects cells 91 with high accuracy after staining with the antibody CD3 corresponding to the third stained image 83a. On the other hand, when the number of visually recognizable cells 91 in the third superimposed image 63a is relatively large, the learned model detects cells 91 with low accuracy after staining with the antibody CD3 corresponding to the third stained image 83a. In addition, similarly to the case of inputting and outputting the superimposed image 64a, the above-mentioned overdetection in the third superimposed image 63a can be confirmed by adjusting the transparency of the cell detection image 71a in the third superimposed image 63a or displaying the third superimposed image 63a and the third stained image 83a side by side.

[0080] The first stained image 81b, the second stained image 82b, the third stained image 83b, the fourth stained image 84b, and the cell detection image 71b of the second field of view 90b, and the first stained image 81c, the second stained image 82c, the third stained image 83c, the fourth stained image 84c, and the cell detection image 71c of the third field of view 90c are the same except that the arrangement of the cells 91 and the detection cell area 92 is different. Figures 5 to 9 The examples shown are roughly the same image.

[0081] In addition, in the image display device 1, Figures 10 to 13 The above-mentioned superimposed images related to the first field of view 90a shown (i.e., the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a and the input-output superimposed image 64a) are similarly displayed on the display 55. Similarly, the superimposed images related to the second field of view 90b and the superimposed images related to the third field of view 90c can also be displayed on the display 55.

[0082] Next, refer to Figure 14 An example of the flow of displaying an image in the image display device 1 will be described. First, in the image display device 1, the staining image groups 80a, 80b, 80c corresponding to the first field of view 90a, the second field of view 90b, and the third field of view 90c of the specimen 9 are displayed. Figure 4 ) is input to the image display device 1 via the communication unit 68 and the reading device 67 and stored in the storage unit 501 (refer to Figure 2 ), thereby preparing (step S11). In addition, the cell detection images 71a, 71b, 71c corresponding to the first field of view 90a, the second field of view 90b and the third field of view 90c (see Figure 4 ) is input to the image display device 1 via the communication unit 68 and the reading device 67 and stored in the storage unit 501, thereby preparing (step S12).

[0083] Next, the display control unit 502 (see Figure 2 ) is controlled to generate a superimposed image by superimposing the images included in the stained image groups 80a, 80b, and 80c with the cell detection images 71a, 71b, and 71c, and display it on the screen of the display 55 (step S13). In step S13, the images are displayed in various display modes, and the display can be switched between multiple display modes.

[0084] Figure 15 1 is a diagram showing an example of a screen 551 of the display 55 displaying an image in the first display mode. Figure 15 In the example shown, the input / output superimposed images 64a, 64b, and 64c corresponding to the first field of view 90a, the second field of view 90b, and the third field of view 90c, respectively, are displayed on screen 551 as a first display image group, arranged non-overlappingly (i.e., arranged flatly in a tiled pattern). Therefore, by observing the first display image group on screen 551, the operator can simultaneously verify whether the learned model can detect cells 91 to be detected at a predetermined ratio or above (i.e., verify the quantitative detection accuracy) for multiple fields of view on specimen 9. As a result, the quantitative detection accuracy of the learned model can be easily and quickly verified.

[0085] In the image display device 1, it is preferred that the display magnifications of the multiple images included in the first display image group in the first display mode (in the above example, the input and output superimposed images 64a, 64b, and 64c of the first field of view 90a, the second field of view 90b, and the third field of view 90c) can be changed in a linked manner.

[0086] For example, when the operator places a mouse pointer (not shown) on one of the multiple images included in the first display image group (for example, the input / output superimposed image 64a of the first field of view 90a), and operates the mouse wheel or the like to change the display magnification of the one image, as shown in FIG. Figure 16 As shown, the display magnifications of images other than the one image on screen 551 (for example, the input and output superimposed images 64b and 64c of the second field of view 90b and the third field of view 90c) are also changed to the same display magnification as that of the one image. When the display magnifications of the plurality of images included in the first display image group are greater than the initial state, only a portion of the first field of view 90a, the second field of view 90b, and the third field of view 90c are displayed in each of the plurality of images.

[0087] Thus, when the operator observes an image other than the one image (e.g., input / output superimposed image 64a) included in the first display image group, the operator can observe the image using the display magnification selected when observing the one image without re-adjusting the display magnification. As a result, the observation quality of the multiple images included in the first display image group can be easily made uniform.

[0088] In the first display mode, for the first to third fields of view 90-90c, instead of or in addition to the input / output superimposed images 64a-64c, a first superimposed image formed by superimposing the first stained images 81a-81c and the cell detection images 71a-71c, a second superimposed image formed by superimposing the second stained images 82a-82c and the cell detection images 71a-71c, and / or a third superimposed image formed by superimposing the third stained images 83a-83c and the cell detection images 71a-71c may be displayed on the screen 551. That is, in the first display mode, a first display image group including two or more superimposed images corresponding to each of two or more fields of view is displayed side by side on the screen 551. The number of images displayed on the screen 551 in the first display mode can be changed as appropriate.

[0089] In the first display mode, as Figure 17As shown, the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a and the input / output superimposed image 64a of the first field of view 90a can also be displayed in a stacked manner. Stacked display refers to a display method in which multiple images are displayed in layers, and the entire top image can be visually recognized, while only a portion of the other images can be visually recognized. Figure 17 In the example shown, the input / output superimposed image 64 a is located at the top layer, and its entirety can be visually recognized.

[0090] For example, the operator moves the mouse pointer to another superimposed image located below the input / output superimposed image 64a on the screen 551 and clicks the mouse 56b (see Figure 1 ) by selecting the other superimposed image, thereby moving the other superimposed image to the top layer and displaying the entire superimposed image. In this way, changing the superimposed image displayed on the top layer among the plurality of superimposed images (e.g., the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a, and the input / output superimposed image 64a) displayed in a stacked manner is similar to switching a stained image displayed superimposed on the cell detection image 71a to another stained image included in the stained image group 80a.

[0091] Moving a superimposed image from a lower layer of a plurality of superimposed images displayed in a stack to the top layer can also be performed by various other methods. For example, a list of a plurality of superimposed images displayed in a stack can be displayed on screen 551, and by selecting a superimposed image from the list, the superimposed image is moved to the top layer of the stack. Alternatively, among the plurality of superimposed images displayed in a stack, buttons (e.g., a "next" button and a "previous" button) can be provided on screen 551 for switching the superimposed image located at the top layer one by one, and by clicking the button, the desired superimposed image is moved to the top layer. In addition, instead of the above-mentioned buttons for switching superimposed images, a shortcut key for switching superimposed images can be provided.

[0092] exist Figure 17 In the example shown, the first superimposed image 61b, the second superimposed image 62b, the third superimposed image 63b, and the input-output superimposed image 64b of the second field of view 90b, and the first superimposed image 61c, the second superimposed image 62c, the third superimposed image 63c, and the input-output superimposed image 64c of the third field of view 90c are also stacked and displayed in the same manner as the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a, and the input-output superimposed image 64a corresponding to the first field of view 90a.

[0093] In the first display mode, when the operator selects a superimposed image in a lower layer of the stacked display of superimposed images for one of the first, second, and third fields of view 90a, 90b, and 90c, as described above, the selected superimposed image is moved to the top layer. Furthermore, in the stacked display of superimposed images for fields of view other than the selected superimposed image, the superimposed images corresponding to the selected superimposed image are also moved to the top layer in conjunction with the selected superimposed image. Specifically, for example, when third superimposed image 63a in first field of view 90a is selected and moved to the top layer, third superimposed image 63b in second field of view 90b and third superimposed image 63c in third field of view 90c are also moved to the top layer in conjunction with the selected superimposed image.

[0094] Thus, for the first to third fields of view 90 to 90c, an operator who observes one superimposed image (e.g., input and output superimposed images 64a to 64c) can quickly perform the next observation if he or she wants to observe another superimposed image (e.g., third superimposed images 63a to 63c) based on the observation results.

[0095] In the first display mode, in addition to the above-mentioned superimposed image, the staining image and cell detection image included in the staining image group can also be displayed. Figure 18 In the example shown, the fourth stained image 84a of the first field of view 90a is displayed in a non-overlapping manner to the right of the input-output superimposed image 64a of the first field of view 90a. Furthermore, the fourth stained image 84b of the second field of view 90b is displayed in a non-overlapping manner to the right of the input-output superimposed image 64b of the second field of view 90b. Thus, the fourth stained images 84a and 84b and the input-output superimposed images 64a and 64b, which are formed by overlapping the detected cell region 92 with the fourth stained images 84a and 84b, can be observed in parallel for the first and second fields of view 90a, 90b. As a result, for example, it is possible to easily examine the configuration and characteristics of cells 91 that were not detected by the learned model. Furthermore, it is easy to confirm whether the detected region (i.e., the detected cell region 92) is the actual cell 91 that should be detected (i.e., whether there is overdetection as described above).

[0096] In addition, Figure 18 In the example shown, the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a can be displayed stacked below the input / output superimposed image 64a of the first field of view 90a, and the first stained image 81a, the second stained image 82a, and the third stained image 83a can be displayed stacked below the fourth stained image 84a. The same applies to the input / output superimposed image 64b and the fourth stained image 84b of the second field of view 90b.

[0097] exist Figure 18In the example shown, by clicking a page switch button (not shown) on screen 551, the first display image group associated with the first field of view 90a and the second field of view 90b is switched to the first display image group associated with the third field of view 90c. Within the first display image group associated with the third field of view 90c, the fourth stained image 84c of the third field of view 90c is displayed to the right of the input / output superimposed image 64c of the third field of view 90c, without overlapping.

[0098] In the image display device 1, the display control unit 502 (see Figure 2 ) is controlled, the display mode of the screen 551 can be switched between the first display mode mentioned above and the second display mode described below.

[0099] Figure 19 FIG. 1 is a diagram showing an example of a process for switching the display mode in the above-mentioned step S13. Figure 17 As shown in FIG. 5 , the display mode of the screen 551 is set to the first display mode, and the first display image group is displayed on the screen 551 (step S21). Figure 17 On the screen 551 of the first display mode shown, move the mouse pointer (not shown) to the input / output superimposed image 64a and click the mouse 56b (see Figure 1 ) to select the input / output superimposed image 64a (step S22). Thereafter, the operator clicks the display mode switching button (omitted from the figure) on the screen 551. Figure 20 As shown, the display mode of screen 551 is switched to a second display mode (step S23) in which a plurality of superimposed images of the field of view corresponding to the input / output superimposed image 64a, i.e., the first field of view 90a (i.e., the field of view corresponding to an image selected in step S22) are arranged and displayed as a second display image group.

[0100] Figure 20 5 is a diagram showing an example of a screen 551 of the display 55 displaying an image in the second display mode. Figure 20 In the illustrated example, regarding the first field of view 90a, the input / output superimposed image 64a, the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a are arranged and displayed on the screen 551 as a second display image group so as not to overlap each other.

[0101] Therefore, by observing the second display image group on screen 551, the operator can simultaneously confirm which stained image contains cells 91 that failed to detect the learned model, for the first stained image 81a, the second stained image 82a, and the third stained image 83a. As a result, detection based on the learned model can easily and quickly confirm that no analytically important types of cells 91, which are rare but not detected (e.g., cells 91 contained in third stained image 83a), have been missed. In other words, by observing the second display image group on screen 551, the operator can easily and quickly confirm the qualitative detection accuracy of the learned model.

[0102] In the image display device 1, when the display of the screen 551 is switched from the first display mode to the second display mode in the above step S23, the display magnification of one image selected in the first display mode (in the above example, the input / output superimposed image 64a of the first field of view 90a) and the display range on the specimen 9 are temporarily stored in the above storage unit 501 (refer to Figure 2 ). Then, when the display mode of the screen 551 is set to the second display mode and the second display image group is displayed, the display magnification and display range on the specimen 9 of the superimposed images included in the second display image group (i.e., the input / output superimposed image 64a, the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a of the first field of view 90a) are set to be the same as the display magnification and display range on the specimen 9 temporarily stored in the storage unit 501.

[0103] Thus, for the area in the first field of view 90a focused on in the first display mode, the operator does not need to re-search the area in the second display mode. In addition, there is no need to adjust the display magnification of the area, and the area can be observed at the desired magnification immediately after switching to the second display mode.

[0104] In the image display device 1, it is preferred that the display magnifications of the plurality of images included in the second display image group (in the above example, the input / output superimposed image 64a of the first field of view 90a, the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a) in the second display mode can be changed in a linked manner.

[0105] For example, when the operator places a mouse pointer (not shown) on one of the plurality of images included in the second display image group (for example, the input / output superimposed image 64a) and operates the mouse wheel or the like to change the display magnification of the one image, as shown in FIG. Figure 21As shown, the display magnifications of images other than the one image (e.g., the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a) on the screen 551 are also changed to the same display magnification as that of the one image. When the display magnifications of the plurality of images included in the second display image group are greater than the initial state, only a portion of the first field of view 90a is displayed in each of the plurality of images.

[0106] Thus, for an area in the first field of view 90a focused on an image in the second display image group (for example, the input-output overlay image 64a), the operator can immediately observe the area at the desired magnification without having to readjust the display magnification of the area on other images.

[0107] In addition, in the image display device 1, it is preferred that the display range on the specimen 9 of the multiple images included in the second display image group in the second display mode (in the above example, the input and output superimposed image 64a of the first field of view 90a, the first superimposed image 61a, the second superimposed image 62a and the third superimposed image 63a) can be changed in a linked manner.

[0108] For example, as described above, in a state where the display magnification of the plurality of images included in the second display image group is greater than the initial state and only a portion of the first field of view 90a is displayed, the operator places the mouse pointer on one of the plurality of images (for example, the input / output superimposed image 64a) and operates the mouse button to change the display range of the specimen 9 of the one image (that is, to change the position of the display range of the one image within the first field of view 90a). Figure 22 As shown, the display range on the specimen 9 of images other than the one image on the screen 551 (for example, the first superimposed image 61 a , the second superimposed image 62 a , and the third superimposed image 63 a ) is also changed to be the same as the display range of the one image.

[0109] Thus, the operator can immediately observe the region in the first field of view 90a that is of interest in one image (eg, the input / output superimposed image 64a) in the second display image group without having to search for the region again in other images.

[0110] In the image display device 1, as Figure 20As shown, in the second display mode, a position indicator 553 indicating the position on each image can be displayed on each of the multiple images included in the second display image group (in the above example, the input / output superimposed image 64a of the first field of view 90a, the first superimposed image 61a, the second superimposed image 62a, and the third superimposed image 63a). The position indicator 553 is, for example, a cross mark, a rectangular frame, or the like that can be moved by the mouse 56b or the like. Preferably, the multiple position indicators 553 displayed on the multiple images included in the second display image group can move in conjunction with the movement of the mouse 56b or the like. As a result, the operator can easily focus on approximately the same position indicated by the position indicator 553 in the multiple images included in the second display image group.

[0111] In the second display mode, instead of or in addition to the input / output superimposed image 64a, first superimposed image 61a, second superimposed image 62a, and third superimposed image 63a of the first field of view 90a, the input / output superimposed image 64b, first superimposed image 61b, second superimposed image 62b, and third superimposed image 63b of the second field of view 90b, and / or the input / output superimposed image 64c, first superimposed image 61c, second superimposed image 62c, and third superimposed image 63c of the third field of view 90c may be displayed on the screen 551. In other words, in the second display mode, a second display image group including two or more superimposed images corresponding to one field of view is displayed side by side on the screen 551. The number of images displayed on the screen 551 in the second display mode can be changed as appropriate.

[0112] When switching the display of the screen 551 from the second display mode to the first display mode, the operator, for example, Figure 20 On the screen 551 of the second display mode shown, move the mouse pointer (not shown) to the input / output superimposed image 64a and click the mouse 56b (see Figure 1 ) to select the input / output superimposed image 64a, and then click the display mode switching button (omitted from the figure) on the screen 551. Figure 15 or Figure 17 As shown, the display mode of the screen 551 is switched to the first display mode in which the input / output superimposed images 64a, 64b, and 64c of the first field of view 90a, the second field of view 90b, and the third field of view 90c are displayed side by side.

[0113] In addition, when switching from the second display mode to the first display mode, the operator selects Figure 20 In the case of the first superimposed image 61a in the first display mode, the first superimposed images 61a, 61b, 61c of the first field of view 90a, the second field of view 90b and the third field of view 90c are displayed. Figure 15 or Figure 17 The input / output superimposed images 64a, 64b, and 64c are displayed side by side on the screen 551 in substantially the same manner.

[0114] In addition, when the operator selects the second superimposed image 62a, in the first display mode, the second superimposed images 62a, 62b, 62c of the first field of view 90a, the second field of view 90b, and the third field of view 90c are displayed. Figure 15 or Figure 17 The input and output superimposed images 64a, 64b, and 64c are arranged and displayed on the screen 551 in a substantially similar manner. When the operator selects the third superimposed image 63a, in the first display mode, the third superimposed images 63a, 63b, and 63c of the first field of view 90a, the second field of view 90b, and the third field of view 90c are also arranged and displayed on the screen 551. Figure 15 or Figure 17 The input / output superimposed images 64 a , 64 b , and 64 c are arranged and displayed on the screen 551 in substantially the same manner.

[0115] In the above example, the image display device 1 is used to verify the detection results of the cell area based on one learned model, but it is not limited to this. For example, the image display device 1 can also be used to compare the detection results of the cell area based on multiple learned models. When comparing multiple learned models, the display mode of the screen 551 in step S13 is switched to Figure 23 The third display mode shown. Switching from the first display mode or the second display mode to the third display mode is performed, for example, by the operator clicking a display mode switching button (not shown) on the screen 551. Switching from the third display mode to the first display mode or the second display mode is substantially the same.

[0116] The following describes a comparison between the previously described learned model and another learned model of a different type. In the following description, the previously described learned model is also referred to as the "first learned model," and the other learned model is also referred to as the "second learned model." The second learned model is created, for example, by performing machine learning, such as deep learning, on an initial model for detecting cell regions using a learning dataset, which is a collection of stained images used for learning. The first and second learned models differ in the type of initial model, the content of the learning dataset, and the machine learning method.

[0117] exist Figure 23In the example shown, a first superimposed image 61a, a second superimposed image 62a, a third superimposed image 63a, and an input / output superimposed image 64a related to the first learning completed model are displayed in a stacked manner on the left side of the screen 551 of the display 55. The top layer of this stacked display is the input / output superimposed image 64a. The first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a, and the input / output superimposed image 64a are generated by the first learning completed model and are related to the first visual field 90a (refer to FIG. Figure 3 ) corresponding cell detection image 71a (refer to Figure 9 ) are respectively superimposed on the first dyed image 81a, the second dyed image 82a, the third dyed image 83a and the fourth dyed image 84a corresponding to the first field of view 90a (refer to Figures 5 to 8 ) is an image.

[0118] On the right side of the screen 551 of the display 55, the first superimposed image 41a, the second superimposed image 42a, the third superimposed image 43a, and the input / output superimposed image 44a related to the second learning completed model are displayed in a stacked manner. The top layer of this stacked display is the input / output superimposed image 44a. The first superimposed image 41a, the second superimposed image 42a, the third superimposed image 43a, and the input / output superimposed image 44a are generated by the second learning completed model and are related to the first visual field 90a (refer to FIG. Figure 3 ) are overlapped with the first stained image 81a, the second stained image 82a, the third stained image 83a and the fourth stained image 84a corresponding to the first visual field 90a (refer to Figures 5 to 8 ) is an image.

[0119] exist Figure 23 In the illustrated third display mode, the first overlay image 61a, second overlay image 62a, third overlay image 63a, and input / output overlay image 64a related to the first learning completion model, and the first overlay image 41a, second overlay image 42a, third overlay image 43a, and input / output overlay image 44a related to the second learning completion model constitute a third display image group displayed on screen 551. The input / output overlay image 64a and the input / output overlay image 44a located at the top of each stack are arranged on screen 551 so as not to overlap each other (i.e., arranged in a tiled manner).

[0120] For example, the operator moves the mouse pointer to another superimposed image located below the input / output superimposed image 64a on the screen 551 and clicks the mouse 56b (see Figure 1) by pressing a mouse button or the like to select the other superimposed image, thereby moving the other superimposed image to the top layer and displaying the whole superimposed image. In this way, the superimposed image displayed on the top layer among the plurality of superimposed images (for example, the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a, and the input / output superimposed image 64a) displayed in a stacked manner is changed, and one stained image displayed superimposed on the cell detection image 71a is switched to the stained image group 80a (see Figure 4 The same is true for another stained image included in ).

[0121] In the third display mode, when the operator selects an overlay image in a lower layer of the stacked display of overlay images associated with one of the first and second completed models, as described above, the selected overlay image is moved to the top layer. Furthermore, in the stacked display of overlay images associated with other completed models, the overlay images corresponding to the selected overlay image are also moved to the top layer. Specifically, for example, when the third overlay image 63a of the first completed model is selected and moved to the top layer, the third overlay image 43a of the second completed model is also moved to the top layer in a linked manner.

[0122] The operator can easily and quickly compare the detection results of the cells 91 based on the first learned model with the detection results of the cells 91 based on the second learned model by observing the third display image group on the screen 551. Figure 23 As shown, by comparing the input / output superimposed image 64a with the input / output superimposed image 44a, the first learned model and the second learned model can be compared to see whether the cells 91 to be detected can be detected at a ratio above a certain level (i.e., the detection accuracy in terms of quantity). Figure 12 ) is compared with the third superimposed image 43a, and the first learning completion model and the second learning completion model can be compared to determine whether there is any missed detection of a type of cell 91 that is small in number but important in analysis (i.e., the detection accuracy in terms of quality).

[0123] In addition, Figure 23 In the example shown, the superimposed image corresponding to the first field of view 90a is displayed on the screen 551, but in the third display mode, the stained image corresponding to the first field of view 90a (for example, Figures 5 to 8The first dyed image 81a, the second dyed image 82a, the third dyed image 83a, and the fourth dyed image 84a shown in the figure can also be displayed on the screen 551. In addition, in the third display mode, instead of the superimposed image corresponding to the first field of view 90a, or in addition to the superimposed image corresponding to the first field of view 90a, superimposed images corresponding to fields of view other than the first field of view 90a can also be displayed on the screen 551.

[0124] In the image display device 1, it is preferred that the display magnifications of multiple images included in the third display image group (for example, the input-output superimposed image 64a corresponding to the first learning completion model and the input-output superimposed image 44a corresponding to the second learning completion model) in the third display mode can be changed in a linked manner.

[0125] For example, when the operator places a mouse pointer (not shown) on one of the images included in the third display image group (for example, the input / output superimposed image 64a) and operates the mouse wheel or the like to change the display magnification of the one image, as shown in FIG. Figure 24 As shown, the display magnifications of images other than the one image (e.g., input / output overlay image 44a) on screen 551 are also changed to the same display magnification as that of the one image. When the display magnifications of the plurality of images included in the third display image group are greater than the initial state, only a portion of the first field of view 90a is displayed in each of the plurality of images.

[0126] Thus, for an area in the first field of view 90a focused on an image in the third display image group (for example, the input / output overlay image 64a), the operator can immediately observe the area at the desired magnification without having to readjust the display magnification of the area on other images.

[0127] In addition, in the image display device 1, it is preferred that, in the third display mode, the display range on the specimen 9 of the plurality of images included in the third display image group (for example, the input-output overlay image 64a corresponding to the first learning completion model and the input-output overlay image 44a corresponding to the second learning completion model) can be changed in a linked manner.

[0128] For example, as described above, in a state where the display magnification of the plurality of images included in the third display image group is greater than the initial state and only a portion of the first field of view 90a is displayed, the operator places the mouse pointer on one of the plurality of images (for example, the input / output superimposed image 64a) and operates the mouse button to change the display range of the specimen 9 of the one image (that is, to change the position of the display range of the one image within the first field of view 90a). Figure 25As shown, the display range on the specimen 9 of images other than the one image (for example, the input / output superimposed image 44 a ) on the screen 551 is also changed to be the same as the display range of the one image.

[0129] Thus, the operator can immediately observe the region in the first field of view 90a focused on one image (eg, the input / output superimposed image 64a) in the third display image group without having to search for the region again on other images.

[0130] In the image display device 1, as Figure 23 As shown, in the third display mode, a position indicator 553 indicating the position on each image may be displayed on each of the multiple images included in the third display image group (for example, the input / output superimposed image 64a corresponding to the first learning completion model and the input / output superimposed image 44a corresponding to the second learning completion model). The position indicator 553 may be, for example, a cross mark, a rectangular frame, or the like that can be moved using the mouse 56b or the like. Preferably, the multiple position indicators 553 displayed on the multiple images included in the third display image group can be moved in conjunction with the movement of the mouse 56b or the like. This allows the operator to easily focus on approximately the same position indicated by the position indicator 553 in the multiple images included in the third display image group.

[0131] As described above, the above-mentioned image display method for displaying the result of detecting a cell area from a staining image based on an immunostaining method includes: a process of preparing a staining image group 80a to 80c including a plurality of staining images 81a to 84a, 81b to 84b, and 81c to 84c of the specimen 9 obtained by the immunostaining method for each of a plurality of fields of view 90a to 90c on the specimen 9 (step S11); a process of preparing a cell detection image 71a to 71c representing a cell area 92 (i.e., a detected cell area 92) detected from the staining image group 80a to 80c by a prescribed detection method for each of the plurality of fields of view 90a to 90c (step S12); and a process of displaying the staining image group 80a to 80c and the cell detection images 71a to 71c superimposed on each other on the screen 551 (step S13).

[0132] The display mode of screen 551 in step S13 can be switched between a first display mode and a second display mode. In the first display mode, a first display image group is displayed on screen 551. The first display image group includes two or more superimposed images (e.g., input / output superimposed images 64a-64c) formed by overlapping one stained image from the stained image group 80a-80c with the cell detection images 71a-71c for each of two or more fields of view 90a-90c. In the second display mode, a second display image group is displayed on screen 551. The second display image group includes two or more superimposed images (e.g., first superimposed image 61a, second superimposed image 62a, third superimposed image 63a, and input / output superimposed image 64a) formed by overlapping two or more stained images from the stained image group 80a-80c with the cell detection images 71a-71c for each of the fields of view 90a-90c. As described above, this facilitates verification of the detection results of the cell region 92 by the operator.

[0133] Preferably, the above-mentioned step S13 includes: a process of setting the display mode of the screen 551 as the first display mode and displaying the first display image group on the screen 551 (step S21); a process of selecting an image (for example, the input-output overlay image 64a) from the first display image group displayed in step S21 (step S22); and a process of setting the field of view (for example, the first field of view 90a) corresponding to the one image selected in step S22 as the above-mentioned one field of view of the second display mode, switching the display mode of the screen 551 from the first display mode to the second display mode, and displaying the second display image group (for example, the first overlay image 61a, the second overlay image 62a, the third overlay image 63a and the input-output overlay image 64a) on the screen 551 (step S23).

[0134] Thus, for a single visual field that the operator is focusing on in the first display mode, multiple superimposed images corresponding to multiple staining methods can be displayed on the screen 551 through simple operation, and these multiple superimposed images can be observed simultaneously. As a result, verification of the detection results of the cell region 92 can be facilitated.

[0135] As described above, it is preferred that, in step S23, two or more superimposed images included in the second display image group (e.g., the first superimposed image 61a, the second superimposed image 62a, the third superimposed image 63a, and the input / output superimposed image 64a) are displayed with the same display magnification and display range on the specimen 9 as those of the one image selected in step S22 (e.g., the input / output superimposed image 64a).

[0136] In this way, the display magnification and display range of the superimposed image are maintained when switching display modes. This allows the operator to observe the area of ​​interest in the first display mode at the desired magnification immediately after switching to the second display mode. As a result, verification of the detection results of the cell region 92 can be facilitated.

[0137] In the above-described image display method, preferably, in the second display mode, the display magnifications of the two or more superimposed images included in the second display image group can be changed in a coordinated manner. Thus, for a region of interest in one image in the second display image group, the operator can immediately observe that region at the desired magnification without having to readjust the display magnification of that region in other images. This makes it easier to verify the detection results of the cell region 92.

[0138] In the above-described image display method, preferably, in the second display mode, the display range of specimen 9 in two or more superimposed images included in the second display image set can be changed in a coordinated manner. This allows the operator to immediately observe a region of interest in one image in the second display image set without having to search for that region again in other images. This facilitates verification of the detection results of cell region 92.

[0139] In the above-described image display method, preferably, in the second display mode, the position indicators 553, each displayed on one or more superimposed images included in the second display image group and indicating a position on the image, can move in conjunction with each other. This allows the operator to easily focus on substantially the same position indicated by the position indicators 553 in the two or more superimposed images included in the second display image group. As a result, verification of the detection results of the cell region 92 can be facilitated.

[0140] In the above-described image display method, preferably, in the first display mode, the display magnifications of the two or more superimposed images included in the first display image group can be changed in a coordinated manner. Thus, when observing the two or more superimposed images, the operator does not need to individually adjust the display magnifications of each, and can observe at the desired display magnification. As a result, verification of the detection results of the cell region 92 can be facilitated.

[0141] In the above-mentioned image display method, it is preferred that, in the first display mode, the processing of switching one staining image (for example, the fourth staining image 84a to 84c) overlapping with the cell detection images 71a to 71c to other staining images (for example, the third staining image 83a to 83c) included in the staining image group 80a to 80c is performed in conjunction with the above-mentioned two or more fields of view.

[0142] Thus, an operator who has observed one superimposed image (e.g., input / output superimposed images 64a-64c) for a plurality of fields of view 90a-90c and then wishes to observe another superimposed image (e.g., third superimposed images 63a-63c) based on the observation result can quickly perform the next observation. As a result, verification of the detection results of the cell region 92 can be facilitated.

[0143] As described above, the detection method in step S12 (i.e., the method for detecting the cell region 92) is preferably a detection method using a learned model created through machine learning. This allows for highly accurate detection of cell regions 92 having various shapes, sizes, configurations, and staining patterns.

[0144] Regarding the above-mentioned image display method, it is preferred that other cell detection images representing the cell area detected from the staining image group 80a to 80c by other detection methods are prepared for each of the multiple fields of view 90a to 90c. Furthermore, it is preferred that the display mode of the screen 551 in step S13 can be switched between a first display mode, a second display mode, and a third display mode. In the third display mode, a third display image group is arranged and displayed on the screen 551, and the third display image group includes: a superimposed image (for example, input-output superimposed image 64a) formed by superimposing one staining image in the staining image group and the cell detection image for one field of view among the multiple fields of view 90a to 90c; and another superimposed image (for example, input-output superimposed image 44a) formed by superimposing the above-mentioned one staining image and other cell detection images for the one field of view. Thus, the operator can easily compare the detection results of the cell area 92 based on multiple detection methods.

[0145] In the above-described image display method, preferably, in the third display mode, the display magnifications of the aforementioned overlay image and other overlay images (e.g., input / output overlay image 64a and input / output overlay image 44a) can be changed in conjunction with each other. Thus, for a region of interest in one image in the third display image group, the operator can immediately observe that region at the desired magnification without having to readjust the display magnification of that region in other images. As a result, it is easier to compare detection results of cell region 92 based on multiple detection methods.

[0146] In the above-described image display method, preferably, in the third display mode, the display ranges of the specimen 9 in the aforementioned overlay image and other overlay images (e.g., input / output overlay image 64a and input / output overlay image 44a) can be changed in tandem. This allows the operator to immediately observe a region of interest in one image of the third display image group without having to search for that region again in other images. This makes it easier to compare detection results of cell region 92 using multiple detection methods.

[0147] In the above-described image display method, it is preferred that, in the third display mode, the position indicator 553, which is displayed on the aforementioned superimposed image and the other superimposed image (e.g., input / output superimposed image 64a and input / output superimposed image 44a) and indicates the position on the image, be movable in conjunction with each other. This allows the operator to easily focus on the substantially identical position indicated by the position indicator 553 in the two or more superimposed images included in the third display image group. Consequently, the detection results of the cell region 92 based on multiple detection methods can be more easily compared.

[0148] The image display device 1 includes a display unit (i.e., display 55), a storage unit 501, and a display control unit 502. The display 55 displays images. The storage unit 501 stores a set of stained images 80a-80c, including a plurality of stained images 81a-84a, 81b-84b, and 81c-84c of the specimen 9 obtained by immunostaining for each of a plurality of fields 90a-90c on the specimen 9. The cell detection images 71a-71c represent cell regions 92 (i.e., detected cell regions 92) detected from the stained image sets 80a-80c using a predetermined detection method for each of the plurality of fields 90a-90c. The display control unit 502 superimposes the stained image sets 80a-80c and the cell detection images 71a-71c on the screen 551 of the display 55.

[0149] In the image display device 1, the display mode of the screen 551 can be switched between a first display mode and a second display mode under the control of the display control unit 502. In the first display mode, a first display image group is displayed on the screen 551. The first display image group includes two or more superimposed images (e.g., input / output superimposed images 64a-64c) formed by superimposing one stained image from the stained image group 80a-80c and the cell detection images 71a-71c for each of two or more fields of view 90a-90c. In the second display mode, a second display image group is displayed on the screen 551. The second display image group includes two or more superimposed images (e.g., first superimposed image 61a, second superimposed image 62a, third superimposed image 63a, and input / output superimposed image 64a) formed by superimposing two or more stained images from the stained image group 80a-80c and the cell detection images 71a-71c for each of the fields of view 90a-90c. This makes it easier for the operator to verify the detection results of the cell region 92 as described above.

[0150] As described above, the following steps are performed by executing the program 572 on a computer: a step of preparing, for each of the multiple fields of view 90a to 90c on the specimen 9, a staining image group 80a to 80c including multiple staining images 81a to 84a, 81b to 84b, and 81c to 84c of the specimen 9 obtained by the immunostaining method (step S11); a step of preparing, for each of the multiple fields of view 90a to 90c, cell detection images 71a to 71c representing the cell area 92 (i.e., the detected cell area 92) detected from the staining image group 80a to 80c by a prescribed detection method (step S12); and a step of displaying the staining image group 80a to 80c and the cell detection images 71a to 71c on the screen 551 so that they overlap (step S13).

[0151] The display mode of screen 551 in step S13 can be switched between a first display mode and a second display mode. In the first display mode, a first display image group is displayed on screen 551. The first display image group includes two or more superimposed images (e.g., input / output superimposed images 64a-64c) formed by superimposing one stained image from the stained image group 80a-80c and the cell detection images 71a-71c for each of two or more fields of view 90a-90c. In the second display mode, a second display image group is displayed on screen 551. The second display image group includes two or more superimposed images (e.g., first superimposed image 61a, second superimposed image 62a, third superimposed image 63a, and input / output superimposed image 64a) formed by superimposing two or more stained images from the stained image group 80a-80c and the cell detection images 71a-71c for each of the fields of view 90a-90c. As described above, this facilitates verification of the detection results of the cell region 92 by the operator.

[0152] Various changes can be made to the above-described image display method, image display device 1 , and program 572 .

[0153] For example, in the first display mode, the second display mode, and the third display mode, the display magnifications of the plurality of superimposed images displayed on the screen 551 may be individually changeable.

[0154] In the second and third display modes, the display ranges of the specimen 9 in the multiple superimposed images displayed on the screen 551 may be individually changeable. Furthermore, in the second and third display modes, the position indicators 553 displayed on the multiple superimposed images displayed on the screen 551 may be individually movable. Furthermore, the display of the position indicators 553 may be omitted.

[0155] In the first display mode, the process of switching one staining image (for example, the fourth staining image 84a to 84c) overlapping with the cell detection images 71a to 71c to another staining image (for example, the third staining image 83a to 83c) can also be performed separately in multiple fields of view 90a to 90c.

[0156] When the display of screen 551 is switched from the first display mode to the second display mode in step S23, the display magnification and display range of the multiple superimposed images included in the second display image group may not necessarily be the same as the display magnification and display range of an image in the first display image group selected in step S22.

[0157] In addition, when the display of screen 551 switches from the first display mode to the second display mode, it is not necessary to select an image from the first display image group (i.e., select the field of view corresponding to the one image). A field of view different from the field of view displayed in the first display image group can also be selected, and the second display image group corresponding to the field of view is displayed on screen 551.

[0158] In the image display device 1 , when comparison of detection results of the cell region 92 based on a plurality of detection methods is not performed, it is not necessarily necessary to provide a switching function to the third display mode.

[0159] In the image display device 1 , for example, the learned model may be stored in the storage unit 501 , and the image display device 1 may detect the cell region 92 from the stained image group 80 a to 80 c .

[0160] The detection method of the cell region 92 is not necessarily limited to the detection method using a learned model created by machine learning, and various modifications such as a detection method based on rules may be used.

[0161] In the image display device 1 , when a plurality of displays 55 are arranged in the vertical direction and / or the horizontal direction to serve as one display unit, the above-mentioned screen 551 is a collection of screens of the plurality of displays 55 .

[0162] The above-mentioned image display method, image display device 1 and program 572 do not necessarily need to be used for verification of cell region detection results, and can be used for various purposes. In addition, the specimen 9 does not necessarily need to be a specimen of biological origin, and can also be other specimens.

[0163] The configurations in the above-described embodiment and various modifications may be appropriately combined as long as they do not contradict each other.

[0164] While the invention has been described and illustrated in detail, the above description is illustrative and not restrictive, and therefore, it can be understood that various modifications and variations are possible without departing from the scope of the invention.

[0165] Description of Reference Numerals

[0166] 1 Image display device

[0167] 9 specimens

[0168] 41a, 61a~61c first superimposed image

[0169] 42a, 62a~62c second superimposed image

[0170] 43a, 63a~63c third superimposed image

[0171] 44a, 64a~64c input and output superimposed images

[0172] 55 monitor

[0173] 71a~71c cell detection images

[0174] 80a~80c stained image group

[0175] 81a~81c First stained images

[0176] 82a~82c Second stained images

[0177] 83a~83c Third stained images

[0178] 84a~84c Fourth stained images

[0179] 90 Field of View

[0180] 90a First Vision

[0181] 90b Second Vision

[0182] 90c Third Horizon

[0183] 91 cells

[0184] 92 cell region

[0185] 501 Storage Department

[0186] 502 Display Control Unit

[0187] 551 screens

[0188] 572 Program

[0189] Steps S11~S13, S21~S23.

Claims

1. An image display method for displaying the result of detecting a cell area from a staining image based on an immunostaining method, characterized in that: The image display method comprises: a) preparing a staining image set including a plurality of staining images of the specimen obtained by immunostaining for each of a plurality of fields of view on the specimen; b) preparing, for each of the plurality of fields of view, a cell detection image representing a cell region detected from the stained image group by a predetermined detection method; as well as c) a step of displaying the staining image group and the cell detection image superimposed on each other on a screen, The display mode of the screen in step c) can be switched between a first display mode and a second display mode. The first display mode is to arrange and display a first display image group on the screen, the first display image group including two or more superimposed images formed by superimposing one staining image in the staining image group and the cell detection image for each of two or more fields of view among the plurality of fields of view; The second display mode is to display a second display image group side by side on the screen, the second display image group including two or more superimposed images formed by overlapping two or more staining images in the staining image group with the cell detection image for one field of view among the multiple fields of view.

2. The image display method according to claim 1, wherein: The step c) comprises: d) setting the display mode of the screen as the first display mode and displaying the first display image group on the screen; e) a step of selecting an image from the first display image group displayed in step d); as well as f) a step of switching the display mode of the screen from the first display mode to the second display mode, using the field of view corresponding to the one image selected in step e) as the one field of view of the second display mode, and displaying the second display image group on the screen.

3. The image display method according to claim 2, wherein: In the step f), the two or more superimposed images included in the second display image group are displayed at the same display magnification and display range on the specimen as those of the one image selected in the step e).

4. The image display method according to any one of claims 1 to 3, wherein: In the second display mode, display magnifications of the two or more superimposed images included in the second display image group can be changed in a linked manner.

5. The image display method according to any one of claims 1 to 3, wherein: In the second display mode, the display ranges on the specimen of the two or more superimposed images included in the second display image group can be changed in a linked manner.

6. The image display method according to any one of claims 1 to 3, wherein: In the second display mode, position indicators displayed on the two or more superimposed images included in the second display image group and indicating positions on the images are movable in conjunction with each other.

7. The image display method according to any one of claims 1 to 3, wherein: In the first display mode, display magnifications of the two or more superimposed images included in the first display image group can be changed in a linked manner.

8. The image display method according to any one of claims 1 to 3, wherein: In the first display mode, a process of switching the one stained image superimposed on the cell detection image to another stained image included in the stained image group is performed in conjunction with the two or more fields of view.

9. The image display method according to any one of claims 1 to 3, wherein: The detection method in the step b) is a detection method using a learned model created by machine learning.

10. The image display method according to any one of claims 1 to 3, wherein: For each of the plurality of fields of view, another cell detection image is prepared that represents a cell region detected from the stained image group by another detection method. The display mode of the screen in the step c) can be switched between the first display mode, the second display mode, and the third display mode. The third display mode is to arrange and display a third display image group on the screen, and the third display image group includes a superimposed image formed by overlapping one staining image in the staining image group with the cell detection image for one field of view among the multiple fields of view, and other superimposed images formed by overlapping one staining image with the other cell detection images for the one field of view.

11. The image display method according to claim 10, wherein: In the third display mode, the display magnifications of the superimposed image and the other superimposed images can be changed in conjunction with each other.

12. The image display method according to claim 10, wherein: In the third display mode, the display ranges of the superimposed image and the other superimposed images on the specimen can be changed in a linked manner.

13. The image display method according to claim 10, wherein: In the third display mode, position indicators displayed on the superimposed image and the other superimposed image and indicating positions on the images are movable in conjunction with each other.

14. An image display device for displaying the result of detecting a cell region from a staining image based on an immunostaining method, characterized in that: The image display device comprises: a display unit that displays an image; a storage unit storing a staining image group including a plurality of staining images of the specimen obtained by immunostaining for each of a plurality of fields of view on the specimen, and a cell detection image representing a cell region detected from the staining image group by a predetermined detection method for each of the plurality of fields of view; as well as a display control unit that causes the staining image group and the cell detection image to be superimposed and displayed on the screen of the display unit; The display mode of the screen can be switched between a first display mode and a second display mode under the control of the display control unit. The first display mode is to arrange and display a first display image group on the screen, the first display image group including two or more superimposed images formed by superimposing one staining image in the staining image group and the cell detection image for each of two or more fields of view among the plurality of fields of view; The second display mode is to display a second display image group side by side on the screen, the second display image group including two or more superimposed images formed by overlapping two or more staining images in the staining image group with the cell detection image for one field of view among the multiple fields of view.

15. A program for displaying the result of detecting a cell area from a staining image based on an immunostaining method, characterized in that: The program is executed by a computer, thereby performing the following steps: a) preparing a staining image set including a plurality of staining images of the specimen obtained by immunostaining for each of a plurality of fields of view on the specimen; b) preparing, for each of the plurality of fields of view, a cell detection image representing a cell region detected from the stained image group by a predetermined detection method; as well as c) a step of displaying the staining image group and the cell detection image superimposed on each other on a screen, The display mode of the screen in step c) can be switched between a first display mode and a second display mode. The first display mode is to arrange and display a first display image group on the screen, the first display image group including two or more superimposed images formed by superimposing one staining image in the staining image group and the cell detection image for each of two or more fields of view among the plurality of fields of view; The second display mode is to display a second display image group side by side on the screen, the second display image group including two or more superimposed images formed by overlapping two or more staining images in the staining image group with the cell detection image for one field of view among the multiple fields of view.

Images