Analysis assistance method, program, and analysis assistance device

Abstract

An analysis assistance method for assisting analysis of a stained image of a specimen comprises: a step (step S11) for preparing a plurality of stained images which are obtained by performing a plurality of types of stains on a single specimen and respectively correspond to the plurality of types of stains; a step (step S13) for dividing each of the plurality of dyed images into a plurality of divided regions by a predetermined division method; a step (step S14) for acquiring, in each of the dyed images, a uniformity level indicating the uniformity of the dyeing state in each of the plurality of divided regions by using a learned model generated by machine learning (step S14); and a step (step S15) for determining, on the basis of the uniformity level of each of the plurality of dyed images, whether or not a predetermined analysis method is suitable for each of the plurality of divided regions. As a result, a region suitable for analysis, which is common to the plurality of dyed images, can be presented with high precision.

Description

Technical Field

[0001] The present invention relates to a technology for assisting in analyzing a stained image of a specimen.

[0002] [Reference to related applications]

[0003] This application claims the benefit of priority of Japanese patent application JP2022-118443 filed on July 26, 2022, and all disclosed contents of that patent application are incorporated herein by reference. Background Art

[0004] In recent years, in medical research and biological research, the analysis of single-cell units of specimens derived from organisms such as tissue specimens or cell specimens has been carried out for the purpose of clarifying disease mechanisms, biological mechanisms, and drug action mechanisms. For example, in multiple immunostaining analysis, substances derived from organisms such as proteins are stained by various methods and the staining state is measured, thereby performing quantitative and qualitative analysis. In Japanese Patent Publication No. 2017-502362 (Document 1), a technology is proposed: in multiple digital images obtained by staining tissue sections of patients by different staining methods, the position of the images based on the tissue structure is aligned.

[0005] In addition, in the above-mentioned multiple immunostaining analysis, sometimes the staining image used for analysis contains artifacts caused by the staining technique (i.e., shaking or unexpected staining results caused by the instability of staining, etc.). In this case, as long as an area with a uniform staining state is selected as the analysis field of view in the staining image, the staining image can also be used for analysis. On the other hand, in multiple staining images that have been subjected to different staining, even if the analysis field of view of one staining image contains artifacts, the analysis accuracy may be reduced. Therefore, it is necessary to select an area with a uniform staining state as the analysis field of view in all staining images.

[0006] However, since there are many causes for the above-mentioned artifacts, it is difficult to consistently select an appropriate analytical field of view in a stained image, and the current situation is that a skilled operator selects an analytical field of view based on experience. However, it is not easy for a skilled operator to select an appropriate analytical field of view by considering the staining states of multiple stained images. In addition, most operators will magnify the stained image to observe it, in which case it is difficult to visually confirm whether the staining state is uniform. Summary of the invention

[0007] The present invention focuses on an analysis support method for supporting analysis of a stained image of a specimen, and aims to present a region suitable for analysis that is common to a plurality of stained images with high accuracy.

[0008] The first mode of the present invention is an analysis assisting method for assisting in analyzing a staining image of a specimen, comprising: step a, preparing a plurality of staining images corresponding to the plurality of types of staining obtained by performing a plurality of types of staining on a specimen; step b, dividing each of the plurality of staining images into a plurality of segmented areas in a predetermined segmentation manner; step c, using a learned model generated by machine learning to obtain a uniformity level representing the uniformity of the staining state of each of the plurality of segmented areas in each of the staining images; and step d, based on the uniformity level of each of the plurality of staining images, determining whether a predetermined analysis method is suitable for each of the plurality of segmented areas.

[0009] According to the present invention, it is possible to accurately present a region suitable for analysis that is common to a plurality of stained images.

[0010] The analysis supporting method according to a second aspect of the present invention is the analysis supporting method according to the first aspect, wherein based on the determination result in the step d, segmented areas suitable for the analysis method, ie, areas suitable for analysis, are displayed on the plurality of stained images.

[0011] The third aspect of the analysis assistance method of the present invention is an analysis assistance method according to the second aspect, wherein the display magnification of each of the plurality of stained images can be changed, and when the magnification of each of the stained images is changed, the display representing the suitable analysis area on each of the stained images is also changed in magnification together with each of the stained images.

[0012] The fourth mode of the analysis assistance method of the present invention is an analysis assistance method described in the first mode (or any one of the first to third modes), wherein based on the judgment result in the step d, segmented areas that are not suitable for the analysis method, i.e., areas not suitable for analysis, are displayed on the plurality of stained images.

[0013] The fifth mode of the analysis assistance method of the present invention is an analysis assistance method according to the fourth mode, wherein the display magnification of each of the plurality of stained images can be changed, and when the magnification of each of the stained images is changed, the display of the unsuitable analysis area on each of the stained images is also changed in magnification together with each of the stained images.

[0014] The sixth aspect of the analysis assistance method of the present invention is an analysis assistance method described in the first aspect (or any one of the first to fifth aspects), wherein based on the judgment result in the step d, a segmented area, i.e., an intermediate area, is displayed on the plurality of stained images, of which it is uncertain whether it is suitable for the analysis method.

[0015] The seventh aspect of the analysis assistance method of the present invention is an analysis assistance method according to the sixth aspect, wherein the display magnification of each of the plurality of stained images can be changed, and when the magnification of each of the stained images is changed, the display of the middle area on each of the stained images is also changed in magnification together with the stained images.

[0016] The eighth aspect of the analysis assisting method of the present invention is the analysis assisting method according to any one of the first to seventh aspects, further comprising, between the step a and the step c, a step of performing a resolution reduction process on each of the plurality of stained images.

[0017] A ninth aspect of the present invention is the analysis supporting method according to the eighth aspect, wherein, in each of the plurality of stained images after the resolution reduction process, an average diameter of cells in the specimen is equal to or larger than a pixel size.

[0018] The tenth aspect of the analysis assistance method of the present invention is an analysis assistance method recorded in any one of the first to seventh aspects (it may also be any one of the first to ninth aspects), wherein, in the step c, the learned model used to obtain the uniformity level of each of the multiple stained images is common.

[0019] The eleventh aspect of the analysis assistance method of the present invention is an analysis assistance method described in any one of the first to seventh aspects (it may also be any one of the first to tenth aspects), wherein the uniformity level of the learning image used to generate the learned model is determined based on the brightness values ​​of the positive cells and the negative cells in the learning image.

[0020] The present invention also focuses on a program for assisting in analyzing a stained image of a specimen. The twelfth mode of the present invention is a program for assisting in analyzing a stained image of a specimen, wherein the program is executed by a computer to perform: step a, preparing a plurality of stained images corresponding to a plurality of types of staining obtained by performing a plurality of types of staining on a specimen; step b, segmenting each of the plurality of stained images into a plurality of segmented areas in a predetermined segmentation method; step c, using a learned model generated by machine learning to obtain a uniformity level representing the uniformity of the staining state of each of the plurality of segmented areas in each of the stained images; and step d, based on the uniformity level of each of the plurality of stained images, determining whether a predetermined analysis method is suitable for each of the plurality of segmented areas.

[0021] The present invention also focuses on an analysis assistance device for assisting in analyzing a staining image of a specimen. The thirteenth mode of the present invention is an analysis assistance device for assisting in analyzing a staining image of a specimen, comprising: a segmentation unit that segments each of a plurality of staining images corresponding to a plurality of types of staining obtained by performing a plurality of types of staining on a specimen into a plurality of segmented areas in a predetermined segmentation method; a level acquisition unit that uses a learned model generated by machine learning to acquire a uniformity level representing the uniformity of the staining state of each of the plurality of segmented areas in each of the staining images; and a determination unit that determines whether a predetermined analysis method is suitable for each of the plurality of segmented areas based on the uniformity level of each of the plurality of staining images.

[0022] The above-mentioned object and other objects, features, aspects and advantages will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram showing the structure of an analysis support device according to one embodiment.

[0024] Figure 2 This is a block diagram showing the functions of the analysis support device.

[0025] Figure 3 This is a diagram showing an example of a stained image.

[0026] Figure 4 It is a diagram showing a plurality of segmented regions of a stained image.

[0027] Figure 5 This is a diagram showing an example of an enlarged learning image.

[0028] Figure 6 This is a diagram showing an example of an enlarged learning image.

[0029] Figure 7 This is a diagram showing the flow of processing in the analysis support device.

[0030] Figure 8 This is a diagram showing an example of the category display of each divided area.

[0031] Fig. 9 This is a diagram showing an image in which a stained image and a mask are superimposed. DETAILED DESCRIPTION

[0032] Figure 11 is a diagram showing the configuration of a computer functioning as the analysis support device 1 according to one embodiment of the present invention. The analysis support device 1 is a device that supports analysis of a stained image of a specimen, and is used, for example, to support multiplex immunostaining analysis of a specimen derived from a living body.

[0033] The analysis support device 1 is a general computer, and includes a processor 31, a memory 32, an input / output unit 33, and a bus 34. The bus 34 is a signal circuit that connects the processor 31, the memory 32, and the input / output unit 33. The memory 32 stores various information. The memory 32 reads and stores, for example, a program 39 pre-stored in a storage medium 30, i.e., a program product. The storage medium 30 is, for example, a USB or a CD-ROM. The processor 31 performs various processes (for example, numerical calculations) while using the memory 32, etc., according to the above-mentioned program 39, etc. stored in the memory 32. The input / output unit 33 includes: a keyboard 35 and a mouse 36 for accepting inputs from an operator, and a display 37 for displaying outputs from the processor 31, etc. The input / output unit 33 also includes a sending unit 38 for sending outputs from the processor 31, etc.

[0034] Figure 2 39 is a block diagram showing functions realized by the computer executing the program 39 constituting the analysis support device 1. As functions realized by the computer, the analysis support device 1 includes a storage unit 40, an image processing unit 41, a segmentation unit 42, a level acquisition unit 43, a determination unit 44, and a display control unit 45. The storage unit 40 is mainly implemented by the memory 32 (see Figure 1 ) is implemented to store multiple staining images used for multiple immunostaining analysis, etc.

[0035] Figure 3 A diagram showing an example of the stained image 9 stored in the storage unit 40. Figure 3 The illustrated staining image 9 shows a stained, substantially rectangular specimen 91. The specimen 91 is a specimen derived from a living organism that is analyzed by a predetermined analysis method such as multiplex immunostaining analysis. Figure 3 The dyeing state is uniform in the approximately central area (i.e., the area surrounded by the rectangular frame 92), and this area 92 is a suitable area for analysis. In addition, the area 92 is larger than the analysis field when analyzing the dyed image 9. Figure 3 In FIG. 9 , a rectangle 95 indicating the size of the analysis field of view is shown in the lower right region of the stained image 9 .

[0036] Although in Figure 3 In the sample 91 shown Figure 3In the upper left region of the image (i.e., the region surrounded by the rectangular frame 93), the staining state is not uniform but rather presents a gradient pattern, but the change in concentration per unit distance (i.e., the concentration gradient) is small. Therefore, the determination of whether region 93 is suitable for analysis varies depending on the type of analysis performed using the staining image 9. Figure 3 The right end region (i.e., the region surrounded by the rectangular frame 94) is not uniformly dyed but rather appears gradually dyed, and the concentration gradient is large. Therefore, the region 94 is not suitable for analysis.

[0037] Figure 2 The image processing unit 41, the segmentation unit 42, the level acquisition unit 43, the determination unit 44, and the display control unit 45 are mainly operated by the processor 31 (see Figure 1 The image processing unit 41 performs preprocessing on the stained image 9 to emphasize whether the staining state of the specimen 91 is uniform. This preprocessing is, for example, resolution reduction processing.

[0038] The segmentation unit 42 segments the stained image 9 into a plurality of segmented regions using a predetermined segmentation method. Figure 4 9 is a diagram showing a stained image 9 divided into a plurality of divided regions 96. Figure 4 In the example shown, the stained image 9 is divided into a plurality of substantially rectangular divided areas 96 arranged in the vertical direction and the horizontal direction (ie, arranged in a matrix). The divided areas 96 are, for example, squares with a side of 256 pixels. Figure 4 In FIG. 9 , six segmented regions 96 are arranged in the vertical direction, and ten segmented regions 96 are arranged in the horizontal direction. In addition, the segmentation method of the stained image 9 is not limited to Figure 4 As shown, various changes can also be made. For example, the stained image 9 can also be divided into Figure 4 The divided regions 96 shown are generally rectangular divided regions of different sizes. The shape of each divided region 96 is not limited to a generally rectangular shape, and various changes may be made.

[0039] Figure 2The level acquisition unit 43 shown acquires a uniformity level using a learned model generated by deep learning, which is a type of machine learning. The uniformity level indicates the uniformity of the coloring state of each segmented region 96. For example, when the coloring state of the segmented region 96 is uniform as in the region 92 described above, the uniformity level of the segmented region 96 is set to "level 0". In addition, when the coloring state of the segmented region 96 is a gradient pattern but has a small concentration gradient as in the region 93 described above, the uniformity level of the segmented region 96 is set to "level 1". When the coloring state of the segmented region 96 is a gradient pattern and has a large concentration gradient as in the region 94 described above, the uniformity level of the segmented region 96 is set to "level 2". In this example, as the value of the level increases, the uniformity of the coloring state of the segmented region 96 decreases, and the suitability for analysis also decreases.

[0040] The above-mentioned learned model is generated in advance by a device other than the analysis assisting device 1, for example, and is stored in advance in the storage unit 40 of the analysis assisting device 1. The generation of the learned model is performed in the following manner: using a large number of learning images obtained by photographing various staining states in various staining images 9, the initial model learns the learning data representing the relationship between each learning image and the uniformity level through deep learning. As the learning image, for example, an image obtained by cutting out a part of the staining image 9 of the specimen 91, and an image after the image has been rotated or reversed, etc. can be used. The shape and size of the cut learning image can be the same as or different from the segmented area 96. The learning image can also be obtained from a plurality of staining images 9 corresponding to a plurality of types of staining used for multiple immunostaining analysis, etc. In addition, the learning image can also be obtained from a plurality of staining images 9 corresponding to a plurality of specimens 91. As the learning image, for example, an image other than the staining image 9 used for the above-mentioned analysis can also be used. In addition, the above-mentioned learned model can also be generated by machine learning other than deep learning.

[0041] like Figure 2 As shown, the analysis support device 1 may further include a learning unit 46 that executes the program 39 (see Figure 1 ) to realize and function, to generate the above-mentioned learned model. The learning unit 46 is mainly realized by the processor 31. The learning unit 46 uses the above-mentioned numerous learning data to perform machine learning such as deep learning on the initial model, thereby generating the above-mentioned learned model. The learning image used for machine learning is obtained in advance using, for example, a shooting device provided in the analysis assisting device 1 or a shooting device independent of the analysis assisting device 1, and is stored in the storage unit 40 of the analysis assisting device 1.

[0042] The learning data used for machine learning in the learning unit 46 is created by the operator associating a learning image with a level indicating the uniformity level of the learning image. For example, the operator enlarges the learning image and compares the staining methods of adjacent positive cells and negative cells. Generally, since positive cells are stained more densely and negative cells are stained more lightly (or not stained), as long as the difference in concentration (i.e., the difference in brightness value) is clearly determined by adjacent positive cells and negative cells, it can be judged that the staining state is uniform, and the uniformity level of the staining state of the learning image is set to level 0.

[0043] On the other hand, when the concentration of positive cells in the lighter area of ​​the learning image is lighter than the concentration of negative cells in the darker area, or when the concentration of positive cells in the lighter area of ​​the learning image is the same as the concentration of negative cells in the darker area, the staining state of the learning image is judged to be a gradient. Moreover, an area corresponding to the above-mentioned analytical field of view 95 is cut out from the learning image, and when there is light or dark in the area, the concentration gradient judged to be a gradient is large, and the uniformity level of the staining state of the learning image is set to level 2. In addition, when there is substantially no light or dark in the cut-out area, the concentration gradient judged to be a gradient is small, and the uniformity level of the staining state of the learning image is set to level 1. In this way, the uniformity level of the staining state of the learning image used to generate the above-mentioned learned model will be determined based on the concentration (i.e., brightness value) of the positive cells and negative cells in the learning image.

[0044] Figure 5 8 is a diagram showing an example of a learning image 81 that is determined to be level 0 in an enlarged manner. The numerous circles 82a and 82b in the learning image 81 represent cells. Figure 5 The cells outlined by the dotted lines in FIG. 8 represent negative cells 82a. In addition, the cells outlined by the solid lines represent positive cells 82b. Figure 5 In the example shown, negative cells 82a and positive cells 82b are stained normally. Positive cells 82b are stained more densely, and negative cells 82a are stained more lightly (or substantially not stained). Figure 5 In the figure, the positive cells 82b are marked with parallel oblique lines, while the negative cells 82a are not marked with parallel oblique lines.

[0045] exist Figure 5In the upper part of the illustrated learning image 81, since negative cells 82a are densely packed, when the learning image 81 is not magnified, the overall density will be viewed as lighter. In the lower part of the learning image 81, since positive cells 82b are densely packed, when the learning image 81 is not magnified, the overall density will be viewed as darker. In addition, in the central part of the learning image 81 in the vertical direction, since positive cells 82b exist sparsely, when the learning image 81 is not magnified, the overall density will be viewed as intermediate. Therefore, although the learning image 81 can be viewed as a gradient when it is not magnified, when the learning image 81 is not magnified, the gradient will be viewed as darker. Figure 5 When enlarged as shown, since the concentration difference between the adjacent positive cells 82b and negative cells 82a is clear, the uniformity level of the staining state is set to level 0 as described above.

[0046] Figure 6 8 is a diagram showing an example of an enlarged learning image 81 determined to be level 1 or level 2. Figure 6 In FIG. 1 , the density of the parallel oblique lines added to the negative cells 82 a represented by the dotted lines and the positive cells 82 b represented by the solid lines represents the concentration of the stained cells. The higher the density of the parallel oblique lines, the higher the concentration. Figure 6 The upper portion of the illustrated learning image 81 is a pale color region that is insufficiently colored, and the lower portion of the learning image 81 is a dark color region that is excessively colored. Figure 6 The illustrated learning image 81 shows a gradient appearance when not enlarged. Figure 6 In the illustrated learning image 81, the concentration of positive cells 82b in the upper light-colored area is about the same as the concentration of negative cells 82a in the lower dark-colored area. Therefore, the learning image 81 is judged to actually show a gradation pattern, and the uniformity level of the staining state is set to level 1 or level 2.

[0047] The determination unit 44 determines Figure 4 For each of the plurality of divided regions 96 shown, whether or not a predetermined analysis method such as multiple immunostaining analysis is suitable is determined based on the uniformity level of the staining state acquired by the level acquisition unit 43 . The display control unit 45 displays the determination result of the determination unit 44 on the display 37 .

[0048] Next, refer to Figure 7The flow of processing of the analysis assisting device 1 is explained. In the analysis assisting device 1, first, preparation is performed by storing a plurality of staining images 9 in the storage unit 40 (step S11). The plurality of staining images 9 are obtained in the following manner: a plurality of types (for example, nine types) of staining corresponding to a predetermined analysis method (for example, multiple immunostaining analysis) is performed on a specimen 91 and photographed, thereby obtaining a plurality of staining images 9. The plurality of staining images 9 respectively correspond to the above-mentioned plurality of types of staining. The specimen 91 is, for example, a slide specimen. The thickness of the specimen 91 is, for example, less than 4 μm. Each staining image 9 is, for example, an image obtained by photographing the entirety of an immunostained specimen 91. The resolution of the staining image 9 is, for example, about 0.46 μm / pixel. The diameter of the cells contained in the specimen 91 in the staining image 9 is about 10 to 30 pixels.

[0049] Next, the image processing unit 41 performs the above-mentioned preprocessing on each of the plurality of stained images 9 (step S12). In step S12, the stained images 9 are preprocessed in such a way that the boundaries of the cells in each of the stained images 9 are blurred. However, this preprocessing is performed within a range where the presence of each cell can be visually confirmed, so that it can be determined that the area with a high concentration in the stained image 9 is a group of positive cells 82b (see Figure 5 ) or over-dyed areas of intense color (see Figure 6 ).

[0050] The preprocessing is, for example, a resolution reduction process for reducing the resolution of each staining image 9. In the resolution reduction process, for example, each staining image 9 is processed in such a way that the resolution becomes about one-fourth. Preferably, in each of the multiple staining images 9 after the resolution reduction process, the average diameter of the cells (i.e., negative cells and positive cells) in the specimen 91 is greater than the pixel size. The pixel size refers to the length of one side of a pixel in the staining image 9 after the resolution reduction process. In addition, the so-called average diameter of the cells refers to the arithmetic mean of the diameters of all the cells in the specimen 91. In addition, the above-mentioned preprocessing is not limited to the resolution reduction process, and various changes can be made. For example, blurring processing using a filter or the like can also be performed on each staining image 9 as preprocessing.

[0051] Next, the segmentation unit 42 segments each of the plurality of stained images 9 into a plurality of segmented regions 96 (see Figure 4)(Step S13). The segmentation method of the plurality of stained images 9 is the same, and the shapes and positions of the plurality of segmented regions 96 in one stained image 9 are the same even in other stained images 9. In addition, step S13 may be performed in parallel with step S12, or may be performed between step S11 and step S12. That is, steps S12 and S13 are performed between step S11 and step S14 described later.

[0052] When steps S11 to S13 are completed, the uniformity level indicating the uniformity of the dyeing state of each of the plurality of segmented regions 96 is obtained by the level acquisition unit 43 in each dyed image 9 (step S14). As described above, the uniformity level of each segmented region 96 obtained by the level acquisition unit 43 is obtained using a learning model generated by machine learning such as deep learning. Thus, the uniformity level of each segmented region 96 can be obtained with high accuracy without being affected by the proficiency of the operator.

[0053] In step S14, for example, the learned model used to obtain the uniformity level of each of the plurality of stained images 9 is common. In this case, the learning images used to generate the learned model include, for example, images obtained from the plurality of stained images 9. By setting the learned model to be common to the plurality of stained images 9, step S14 can be performed more quickly than in a case where the learned model is changed for each type of stained image 9.

[0054] Next, for Figure 4 Each of the illustrated plurality of segmented regions 96 is judged by the judging unit 44 whether the above-mentioned predetermined analysis method is suitable (step S15). In step S15, based on the uniformity level obtained in each of the plurality of stained images 9, it is judged whether each segmented region 96 is suitable for use in the predetermined analysis method.

[0055] Specifically, the uniformity level obtained in each of the plurality of stained images 9 is confirmed by the determination unit 44 for one segmented region 96. Next, for example, when the uniformity levels of all the plurality of stained images 9 are level 0, the one segmented region 96 is classified as "category 0", which means that the staining state is uniform in all the stained images 9 and the one segmented region 96 is a suitable analysis region suitable for the above-mentioned predetermined analysis method. In addition, when the uniformity level obtained in each of the plurality of stained images 9 includes one or more level 2, the one segmented region 96 is classified as "category 2", which means that the staining state is not uniform and the one segmented region 96 is an unsuitable analysis region unsuitable for the above-mentioned predetermined analysis method. On the other hand, when the uniformity level obtained in each of the plurality of stained images 9 includes one or more level 1 and does not include level 2, the one segmented region 96 is classified as "category 1", which means that the staining state is slightly non-uniform and the one segmented region 96 is an intermediate region for which it is uncertain whether it is suitable for the above-mentioned predetermined analysis method.

[0056] In step S15, the above-mentioned classification is sequentially performed on the plurality of segmented regions 96, thereby determining the category of each segmented region 96. In addition, the criteria for the above-mentioned classification can also be changed in various ways. For example, when the value of level 2 contained in the uniformity level obtained in each of the plurality of stained images 9 for one segmented region 96 is 0 and the value of level 1 is 1 or less, the segmented region 96 can also be classified as category 0, when the value of level 2 contained in the uniformity level obtained in each of the plurality of stained images 9 for one segmented region 96 is 1 or more, the segmented region 96 can also be classified as category 2, and in other cases, the segmented region 96 can also be classified as category 1.

[0057] When step S15 is completed, the display control unit 45 displays the type of each divided area 96 on the display 37 based on the determination result in step S15 (step S16). Figure 8 FIG. 4 is a diagram showing an example of a display of categories of each divided area on the display 37. Figure 8The suitable analysis area 96a (i.e., the segmented area 96 of category 0) is shown without parallel oblique lines. The suitable analysis area 96a is a segmented area suitable for the above-mentioned predetermined analysis method. In addition, the unsuitable analysis area 96c (i.e., the segmented area 96 of category 2) is shown with parallel oblique lines of higher density. The unsuitable analysis area 96c is a region that is not suitable for the predetermined analysis method. Furthermore, the middle area 96b (i.e., the segmented area 96 of category 1) is represented with parallel oblique lines of lower density than the unsuitable analysis area 96c. The middle area 96b is a region that is uncertain whether it is suitable for the predetermined analysis method. Thus, the operator can easily visually confirm the areas suitable for analysis, the areas unsuitable for analysis, and the areas that are uncertain whether they are suitable for analysis in the stained image 9. In the following description, an image in which multiple segmented areas 96 of each category can be visually confirmed will also be referred to as a "mask 97." In Figure 8 In the example shown, an example is shown below the shield 97 (in Fig. 9 The same also applies to ).

[0058] Fig. 9 The mask 97 is displayed superimposed (ie, covered) on one stained image 9 on the display 37. This allows the operator to more easily visually confirm areas suitable for analysis, areas unsuitable for analysis, and areas whose suitability for analysis is uncertain in the stained image 9.

[0059] exist Fig. 9 In the stained image 9, it is most difficult to visually confirm the specimen 91 in the region overlapping with the unsuitable analysis region 96c, and it is easiest to visually confirm the specimen 91 in the region overlapping with the suitable analysis region 96a. In addition, in the region overlapping with the middle region 96b in the stained image 9, the specimen 91 is easier to visually confirm than in the region overlapping with the unsuitable analysis region 96c, and is less visually confirmable than in the region overlapping with the suitable analysis region 96a. In other words, in the stained image 9, the unsuitable analysis region 96c is shielded in a manner that is most difficult to visually confirm, and the suitable analysis region 96a is extracted from the plurality of segmented regions 96 in a manner that is easiest to visually confirm. Thus, in the analysis using the stained image 9, the operator sets the analysis field of view 95 (see Figure 3 ), the analysis field of view 95 can be easily set on the suitable analysis area 96a.

[0060] In the analysis support device 1, the suitable analysis region 96a, the middle region 96b, and the unsuitable analysis region 96c can be overlapped and displayed on each of the above-mentioned multiple staining images 9. One staining image 9 on which the suitable analysis region 96a, the middle region 96b, and the unsuitable analysis region 96c are overlapped may be displayed on the display 37, or two or more staining images 9 on which the suitable analysis region 96a, the middle region 96b, and the unsuitable analysis region 96c are overlapped may be displayed on the display 37 at the same time.

[0061] In addition, the suitable analysis area 96a, the middle area 96b, and the unsuitable analysis area 96c may also be formed by Figure 8 The suitable analysis area 96a, the intermediate area 96b, and the unsuitable analysis area 96c may be displayed on the display 37 in a manner different from each other by hatching as shown, or may be displayed in different colors by being colored on the display 37. In addition, the suitable analysis area 96a, the intermediate area 96b, and the unsuitable analysis area 96c may be displayed on the display 37 in a manner different from each other by various methods other than hatching and color.

[0062] In the analysis support device 1, the display magnification of each stained image 9 displayed on the display 37 can be changed by the display control unit 45 (see Figure 2 When the display magnification of each stained image 9 is changed, the display of the suitable analysis area 96a, the middle area 96b, and the unsuitable analysis area 96c on each stained image 9 is also changed in magnification together with each stained image 9 by the display control unit 45.

[0063] In addition, in step S16, the mask 97 does not necessarily need to be overlapped with the stained image 9 to be represented, and can also be as follows Figure 8 As shown, the image is displayed separately on the display 37 with a mask 97. In step S16, only one or two types of regions among the suitable analysis region 96a, the intermediate region 96b, and the unsuitable analysis region 96c may be overlapped on the stained image 9 and displayed on the display 37, or may be selectively displayed on the display 37 without being overlapped.

[0064] In addition, other masks different from the mask 97 may be displayed on the display 37, either superimposed or not superimposed on the stained image 9. For example, a mask indicating the uniformity level of each segmented region 96 of a stained image 9 may be displayed by superimposing it on the stained image 9. In the mask, for example, parallel oblique lines are not attached to the segmented regions 96 of level 0, parallel oblique lines with a higher density are attached to the segmented regions 96 of level 2, and parallel oblique lines with a lower density than the segmented regions 96 of level 2 are attached to the segmented regions 96 of level 1. Thus, it is possible to easily indicate to the operator in the stained image 9 the region suitable for analysis, the region unsuitable for analysis, and the region uncertain whether or not it is suitable for analysis.

[0065] In the analysis assisting method based on the above-mentioned analysis assisting device 1, category 1 may be omitted in step S15, and each segmented region 96 may be classified into category 0 or category 2. For example, when all the uniformity levels obtained for one segmented region 96 in each of the plurality of stained images 9 are level 0, the one segmented region 96 may be classified into category 0 indicating a suitable analysis region, and in other cases, the one segmented region 96 may be classified into category 2 indicating an unsuitable analysis region. The basis for the category classification may also be changed in various ways. For example, when the uniformity level obtained for one segmented region 96 in each of the plurality of stained images 9 contains a value of level 2 of 0 and a value of level 1 of 2 or less, the one segmented region 96 may be classified into category 0, and in other cases, the one segmented region 96 may be classified into category 2.

[0066] As described above, the above-mentioned analysis assisting method for assisting in analyzing the staining image 9 of the specimen 91 comprises: a process of preparing a plurality of staining images 9 corresponding to the plurality of stainings obtained by performing a plurality of types of staining on a specimen 91 (step S11); a process of segmenting each of the plurality of staining images 9 into a plurality of segmented regions 96 in a predetermined segmentation manner (step S13); a process of obtaining a uniformity level representing the uniformity of the staining state of each of the plurality of segmented regions 96 in each staining image 9 using a learned model generated by machine learning (step S14); and a process of determining whether a predetermined analysis method is suitable for each of the plurality of segmented regions 96 based on the uniformity level of each of the plurality of staining images 9 (step S15).

[0067] Thus, as described above, the uniformity level of the staining state can be obtained with high accuracy for each segmented area 96 in the plurality of stained images 9. Therefore, whether or not the segmented area 96 is suitable for analysis can be determined with high accuracy. As a result, a region suitable for analysis that is common in the plurality of stained images 9 can be presented with high accuracy.

[0068] As described above, in the analysis assisting method, it is preferable that, based on the determination result in step S15, the segmented area 96 suitable for the above-mentioned analysis method, that is, the analysis-suitable area 96a, is displayed on the plurality of stained images 9. Thus, the operator or the like who performs the analysis can easily visually confirm the area suitable for analysis in the stained image 9, and can easily set the analysis field of view 95 for the area. As a result, the accuracy of the analysis using the plurality of stained images 9 can be improved.

[0069] As described above, in the analysis assisting method, it is preferred that the display magnification of each of the plurality of staining images 9 can be changed, and when the magnification of each staining image 9 is changed, the display indicating the suitable analysis area 96a on each staining image 9 is also changed in magnification together with each staining image 9. Thus, even when the display magnification of the staining image 9 is changed, the operator or the like can easily visually confirm the area suitable for analysis in the staining image 9.

[0070] As described above, in the analysis assisting method, it is preferable that, based on the determination result in step S15, the segmented area 96 not suitable for the above analysis method, that is, the unsuitable analysis area 96c, is displayed on the plurality of stained images 9. Thus, the operator or the like who performs the analysis can easily visually confirm the unsuitable analysis area in the stained image 9, and can easily set the analysis field of view 95 by avoiding the area. As a result, the accuracy of the analysis using the plurality of stained images 9 can be improved.

[0071] As described above, in the analysis support method, it is preferred that the display magnification of each of the plurality of staining images 9 can be changed, and when the magnification of each staining image 9 is changed, the display indicating the unsuitable analysis region 96c on each staining image 9 is also changed in magnification together with each staining image 9. Thus, even when the display magnification of the staining image 9 is changed, the operator or the like can easily visually confirm the unsuitable analysis region in the staining image 9.

[0072] As described above, in the analysis assisting method, it is preferred that, based on the determination result in step S15, a segmented area 96, i.e., an intermediate area 96b, is indicated on a plurality of stained images 9, which is uncertain whether it is suitable for the above-mentioned analysis method. Thus, an operator or the like who performs the analysis can easily visually confirm the area in the stained image 9, which is uncertain whether it is suitable for analysis. Therefore, in the case of performing an analysis in which the area can be used, the range in which the analysis field of view 95 can be set on the stained image 9 can be increased, and as a result, the degree of freedom of the analysis can be improved. In addition, in the case of performing an analysis in which the area cannot be used, the analysis field of view 95 can be easily set by avoiding the area, and as a result, the accuracy of the analysis can be improved.

[0073] As described above, in the analysis support method, it is preferred that the display magnification of each of the plurality of stained images 9 can be changed, and when the magnification of each stained image 9 is changed, the display representing the middle area 96b on each stained image 9 is also changed in magnification together with each stained image 9. Thus, even when the display magnification of the stained image 9 is changed, the operator or the like can easily visually confirm the area in the stained image 9 that is uncertain whether it is suitable for analysis.

[0074] As described above, in this analysis support method, preferably, between step S11 and step S14, a step of performing resolution reduction processing (step S12) on each of the plurality of stained images 9 is further included. This allows the uniformity level to be acquired with high accuracy in step S14.

[0075] As described above, it is preferable that the average diameter of cells in the sample 91 is equal to or larger than the pixel size in each of the plurality of stained images 9 after the resolution reduction process. Thus, by performing the resolution reduction process to a degree that allows cells to be individually identified, the uniformity level in step S14 can be appropriately acquired.

[0076] As described above, in step S14, it is preferable that the learned model used to obtain the uniformity level of each of the plurality of stained images 9 is common. Thus, step S14 can be performed more quickly than in the case where the learned model is changed for each type of stained image 9. In addition, the time and effort required to generate the learned model corresponding to the plurality of stained images 9 can be reduced.

[0077] As described above, the uniformity level of the learning image 81 used to generate the learned model is preferably determined based on the brightness values ​​of the positive cells 82b and the negative cells 82a in the learning image 81. This allows the learned model to be appropriately generated.

[0078] As described above, by executing the program 39 on a computer, the following steps are performed: preparing a plurality of stained images 9 corresponding to the plurality of types of staining obtained by performing a plurality of types of staining on a specimen 91 (step S11); dividing each of the plurality of stained images 9 into a plurality of segmented regions 96 in a predetermined segmentation manner (step S13); obtaining a uniformity level indicating the uniformity of the staining state of each of the plurality of segmented regions 96 in each stained image 9 using a learned model generated by machine learning (step S14); and determining whether a predetermined analysis method is suitable for each of the plurality of segmented regions 96 based on the uniformity level of each of the plurality of stained images 9 (step S15). Thus, as described above, it is possible to accurately present a region suitable for analysis that is common to the plurality of stained images 9.

[0079] The analysis support device 1 described above includes: a segmentation unit 42 that segments each of a plurality of stained images 9 corresponding to a plurality of types of staining obtained by performing a plurality of types of staining on a specimen 91 into a plurality of segmented regions 96 in a predetermined segmentation method; a level acquisition unit 43 that acquires a uniformity level indicating uniformity of a staining state of each of the plurality of segmented regions 96 in each stained image 9 using a learned model generated by machine learning; and a determination unit 44 that determines whether a predetermined analysis method is appropriate for each of the plurality of segmented regions 96 based on the uniformity level of each of the plurality of stained images 9. Thus, similarly to the above, a region suitable for analysis that is common to the plurality of stained images 9 can be presented with high accuracy.

[0080] Various changes can be made to the above-mentioned analysis support method, program 39, and analysis support device 1.

[0081] For example, the uniformity level of the staining state of the learning image 81 used to generate the above-mentioned learned model does not necessarily need to be determined based on the brightness values ​​of the positive cells 82b and the negative cells 82a in the learning image 81, and may be determined based on various other criteria.

[0082] In step S14, the learned model used to obtain the uniformity level of each of the plurality of stained images 9 does not necessarily need to be common. For example, a dedicated learned model may be used for each of the plurality of stained images 9. Alternatively, the learned model may be common only to two or more stained images 9 of a part of the plurality of stained images 9.

[0083] In the above example, the categories indicating the suitability of the analysis for each segmented region 96 are classified into three types: 0, 1, and 2. However, the classification may be finer. For example, when the numerical value of the plurality of stained images 9 is 9, when all the uniformity levels obtained for one segmented region 96 in each of the plurality of stained images 9 are level 0, when the one segmented region 96 is classified as category 0 and the numerical value of level 2 contained in the uniformity levels obtained in each of the plurality of stained images 9 is 0 and the numerical value of level 1 is 1, the one segmented region 96 is classified as "category 0.2". In addition, when the value of level 2 included in the uniformity level obtained in each of the plurality of stained images 9 is 0 and the value of level 1 is 2, the one segmented region 96 is classified as "class 0.4", when the value of level 2 included in the uniformity level obtained in each of the plurality of stained images 9 is 0 and the value of level 1 is 3, the one segmented region 96 is classified as "class 0.6", and when the value of level 2 included in the uniformity level obtained in each of the plurality of stained images 9 is 0 and the value of level 1 is 4, the one segmented region 96 is classified as "class 0.8". Furthermore, when the value of level 2 included in the uniformity level obtained in each of the plurality of stained images 9 is 0 and the value of level 1 is 5 or more, the one segmented region 96 is classified as "class 1", and when the value of level 2 included in the uniformity level obtained in each of the plurality of stained images 9 is 1 or more, the one segmented region 96 is classified as "class 2".

[0084] In step S12, in each of the plurality of stained images 9 after the resolution reduction process, the average diameter of the cells in the specimen 91 may be smaller than the pixel size. In addition, as described above, in step S12, a preprocessing other than the resolution reduction process may be performed instead of the resolution reduction process, or in addition to the resolution reduction process.

[0085] The above-mentioned analysis support method, program 39 and analysis support device 1 do not necessarily need to be used to support multiplex immunostaining analysis, but may be used to support various analyses other than multiplex immunostaining analysis. In addition, the specimen 91 does not necessarily need to be a specimen derived from a living body, but may be other specimens.

[0086] The configurations in the above-described embodiment and various modifications may be appropriately combined as long as they do not contradict each other.

[0087] Although the present invention has been described and illustrated in detail, the above description is only illustrative and not restrictive. Therefore, it is contemplated that various changes and modifications are possible without departing from the scope of the present invention.

[0088] Description of Reference Numerals

[0089] 1 Analysis auxiliary device

[0090] 9 Stained images

[0091] 39 Programs

[0092] 42 Division

[0093] 43 Level Acquisition Department

[0094] 44 Judgment Department

[0095] 81 Learning Images

[0096] 82a negative cells

[0097] 82b positive cells

[0098] 91 specimens

[0099] 96 Split Area

[0100] 96a is suitable for analysis area

[0101] 96b Middle Area

[0102] 96c Not suitable for parsing area

[0103] Steps S11 to S16.

Claims

1. A method for assisting analysis of a stained image of a specimen, characterized in that: The analysis assistance method has: Step a, preparing a plurality of staining images respectively corresponding to the plurality of types of staining obtained by performing a plurality of types of staining on a specimen; Step b, dividing each of the plurality of stained images into a plurality of segmented areas in a predetermined segmentation manner; Step c, using a learned model generated by machine learning to obtain a uniformity level representing the uniformity of the dyeing state of each of the plurality of divided regions in each of the dyed images; The step d is to determine whether a predetermined analysis method is appropriate for each of the plurality of divided regions based on the uniformity level of each of the plurality of stained images.

2. The analysis assistance method according to claim 1, characterized in that: Based on the determination result in the step d, the segmented regions suitable for the analysis method, that is, the analysis-suitable regions are displayed on the plurality of stained images.

3. The analysis assistance method according to claim 2, characterized in that: The display magnification of each of the plurality of stained images can be changed. When the magnification of each of the stained images is changed, the magnification of the display indicating the suitable analysis region on each of the stained images is also changed together with the magnification of each of the stained images.

4. The analysis assistance method according to claim 1, characterized in that: Based on the determination result in the step d, segmented regions that are not suitable for the analysis method, that is, regions not suitable for analysis, are displayed on the plurality of stained images.

5. The analysis assistance method according to claim 4, characterized in that: The display magnification of each of the plurality of stained images can be changed. When the magnification of each of the stained images is changed, the display indicating the unsuitable analysis region on each of the stained images is also changed in magnification together with the stained images.

6. The analysis assistance method according to claim 1, characterized in that: Based on the determination result in the step d, the intermediate regions, which are segmented regions for which it is uncertain whether or not they are suitable for the analysis method, are displayed on the plurality of stained images.

7. The analysis assistance method according to claim 6, characterized in that: The display magnification of each of the plurality of stained images can be changed; When the magnification of each of the stained images is changed, the display representing the intermediate region on each of the stained images is also changed in magnification together with the stained images.

8. The analysis assistance method according to any one of claims 1 to 7, characterized in that: The method further includes, between the step a and the step c, performing a resolution reduction process on each of the plurality of stained images.

9. The analysis assistance method according to claim 8, characterized in that: In each of the plurality of stained images after the resolution reduction process, an average diameter of cells in the specimen is larger than a pixel size.

10. The analysis assistance method according to any one of claims 1 to 7, characterized in that: In the step c, the learned model used to obtain the uniformity level of each of the plurality of stained images is common.

11. The analysis assistance method according to any one of claims 1 to 7, characterized in that: The uniformity level of the learning image used to generate the learned model is determined based on the brightness values ​​of positive cells and negative cells in the learning image.

12. A program for assisting in analyzing a stained image of a specimen, characterized in that: The program is executed by a computer to: Step a, preparing a plurality of staining images respectively corresponding to the plurality of types of staining obtained by performing a plurality of types of staining on a specimen; Step b, dividing each of the plurality of stained images into a plurality of segmented areas in a predetermined segmentation manner; Step c, using a learned model generated by machine learning to obtain a uniformity level representing the uniformity of the dyeing state of each of the plurality of divided regions in each of the dyed images; The step d is to determine whether a predetermined analysis method is appropriate for each of the plurality of divided regions based on the uniformity level of each of the plurality of stained images.

13. An analysis assisting device for assisting analysis of a stained image of a specimen, characterized in that: The analysis assisting device comprises: a segmentation unit that segments each of a plurality of stained images corresponding to the plurality of types of staining obtained by performing a plurality of types of staining on one specimen into a plurality of segmented regions in a predetermined segmentation manner; a level acquisition unit that acquires, in each of the stained images, a uniformity level indicating uniformity of a staining state of each of the plurality of divided regions using a learned model generated by machine learning; A determination unit determines whether a predetermined analysis method is appropriate for each of the plurality of divided regions based on the uniformity level of each of the plurality of stained images.

Images