Region detection method, program, and region detection device

Abstract

This method improves the accuracy of detecting target regions based on staining using a pre-trained model. [Solution] The first color component of interest, which is the saturation of multiple training images used to generate the trained model, or the representative value of the first color component of interest corresponding to the color of the stain or a color similar to that color, is obtained as the representative value of the trained color component (steps S14, S15). The representative value of the first color component of at least one target image is obtained as the representative value of the target color component (steps S16, S17). A correction function is obtained from the representative value of the trained color component and the representative value of the target color component, the second color component of the target image is corrected using the correction function, and the corrected target image is input to the trained model, thereby detecting the target region from the target image.

Description

Technical Field

[0001] The present invention relates to a technique for detecting a target region to be detected based on staining from a biologically stained image using a trained model.

Background Art

[0002] Conventionally, single-cell analysis has been performed to acquire the expression level of biological substances including proteins on a per-cell basis from digital images of stained pathological tissue specimens. In single-cell analysis, individual cell regions in an image are detected, and the staining intensity or the like in each cell is acquired based on information of the cell regions. Thereby, for example, the presence or absence of protein expression in each cell is specified, and an analysis result is acquired.

[0003] Images of pathological tissues differ in tissue state and staining condition even for the same site and the same staining, depending on differences in the site from which the tissue specimen is collected and the type of staining. Therefore, it is difficult to perform appropriate analysis with so-called rule-based image processing in which a person determines processing conditions in advance. Accordingly, analysis using machine learning has also been proposed, and generating a trained model after preprocessing training images has also been proposed.

[0004] For example, Patent Document 1 discloses that in a technique for determining ovarian toxicity using machine learning, a training process is executed after preprocessing images used for training so that the images have the same size scale, the same color scale or the same chroma scale. In addition, Patent Document 2 discloses a technique for generating a positive segmentation mask and a negative segmentation mask using a trained model from an image including a staining pattern of a biomarker. A learning algorithm is executed after preprocessing is performed on training images and verification input images so that the images have the same size scale, the same color scale or the same chroma scale.

Prior Art Literature

Patent Literature

[0005] [Patent Document 1] Special Publication No. 2022-529259 [Patent Document 2] Special Publication No. 2023-538309 [Overview of the project] [Problems that the invention aims to solve]

[0006] Incidentally, when extracting target regions to be detected based on staining from stained images (hereinafter referred to as "target regions") using a pre-trained model, the desired target regions are indicated (i.e., annotations are performed) on the images used for training, and the pre-trained model is generated using these images. At this time, the staining density of the sample is usually not constant, and there are images that are heavily stained and images that are lightly stained. Therefore, if the staining density of the images used for training differs from the staining density of the images in which the target regions are to be detected using the pre-trained model, the detection accuracy of the target regions will decrease. Furthermore, even if preprocessing is performed on the images used for training, as in conventional techniques, the detection accuracy of the target regions may not be sufficiently improved due to differences in staining that exceed expectations.

[0007] This invention has been made in view of the above problems, and aims to improve the accuracy of detecting target regions using a trained model. [Means for solving the problem]

[0008] One aspect of the present invention is a region detection method for detecting a target region to be detected based on staining from a biological staining image using a trained model, comprising: a) obtaining a representative value of the first color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from a plurality of training images used to generate the trained model, as a representative value of the trained color component; b) obtaining a representative value of the first color component of at least one target image, which is a biological staining image that is to be input to the trained model and from which a target region is to be detected, as a representative value of the target color component; c) obtaining a correction function from the representative value of the trained color component and the representative value of the target color component; d) correcting the second color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from the at least one target image using the correction function; and e) detecting a target region from the at least one target image by inputting the corrected at least one target image to the trained model.

[0009] Aspect 2 of the present invention is a region detection method according to aspect 1, wherein in step a), a representative value of the first color component of a pixel in the plurality of learning images whose third color component, which is saturation, or whose third color component corresponds to a color corresponding to dyeing or a color similar to said color, is greater than or equal to a predetermined threshold is obtained as the learning color component representative value, and in step b), a representative value of the first color component of a pixel in at least one target image whose third color component is greater than or equal to a predetermined threshold is obtained as the target color component representative value.

[0010] A third aspect of the present invention is a region detection method according to aspect 1 (or aspect 1 or 2), wherein the at least one target image is each of the groups obtained by dividing a plurality of target images into a plurality of groups.

[0011] Aspect 4 of the present invention is a region detection method according to aspect 1 (which may be aspect 1 or 2), wherein the number of at least one target image is 1.

[0012] Aspect 5 of the present invention is a region detection method according to Aspect 1 (which may be any one of Aspects 1 to 4), further comprising, after step d), e) a step of displaying a selected target image selected from the at least one target image; f) a step of modifying the correction function; g) a step of correcting the selected target image using the modified correction function; and h) a step of displaying the selected target image.

[0013] Aspect 6 of the present invention is a region detection method according to any one of aspects 1 to 5, wherein the target region is a cellular-level region.

[0014] Aspect 7 of the present invention is a computer-readable program that causes a computer to detect a target region to be detected based on staining from a biological staining image using a trained model, wherein the execution of the program by the computer causes the computer to perform the following steps: a) obtain a representative value of the first color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, of a plurality of training images used to generate the trained model, as a representative value of the trained color component; b) obtain a representative value of the first color component of at least one target image, which is a biological staining image that is to be input to the trained model and from which the target region is to be detected, as a representative value of the target color component; c) obtain a correction function from the representative value of the trained color component and the representative value of the target color component; d) correct the second color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, of the at least one target image using the correction function; and e) input the corrected at least one target image to the trained model and thereby detect the target region from the at least one target image.

[0015] Aspect 8 of the present invention is a region detection apparatus for detecting a target region to be detected based on staining from a stained biological image, comprising: a trained model that detects a target region from a target image when the target image, which is a stained biological image, is input thereto; a correction function acquisition unit that acquires, as a trained color component representative value, a representative value of a first target color component that is chroma saturation, or a first target color component corresponding to a color corresponding to staining or a color approximate to said color, from a plurality of training images used for generating said trained model, acquires, as a target color component representative value, a representative value of said first target color component of at least one target image for which a target region is to be detected by being input to said trained model, and acquires a correction function from said trained color component representative value and said target color component representative value; and a correction unit that corrects, using said correction function, a second target color component that is chroma saturation, or a second target color component corresponding to a color corresponding to staining or a color approximate to said color, of said at least one target image. Effects of the Invention

[0016] According to the present invention, the accuracy of detecting a target region using a trained model can be improved. Brief Description of Drawings

[0017] [Figure 1] It is a diagram showing the configuration of a computer. [Figure 2] It is a block diagram showing the functional configuration of a region detection apparatus. [Figure 3] It is a diagram showing the flow of a region extraction method. [Figure 4] It is a diagram showing the flow of a region extraction method. [Figure 5] It is a diagram illustrating an example of a training image. [Figure 6] It is a diagram illustrating an example of a chroma saturation image. [Figure 7] It is a diagram illustrating an example of a mask image. [Figure 8] It is a diagram illustrating an example of a target image before correction. [Figure 9] It is a diagram illustrating an example of a target image after correction. [Figure 10]It is a diagram showing a display example of the display unit. [Figure 11] It is a diagram showing the flow of correction of the correction function. MODE FOR CARRYING OUT THE INVENTION

[0018] Figure 1 is a diagram showing the configuration of a computer 4 that functions as a region detection device according to an embodiment of the present invention. The region detection device is a device that detects a target region to be detected based on staining from a biostained image using a trained model. The target region is a stained region, a region surrounded by stained regions, a region including a stained region, or the like. In the present embodiment, the target region is a stained cell region. "Region detection" refers to segmentation that separates a region to be detected from other regions in an image. Detection of a target region by a trained model is used, for example, in the field of image cytometry. In the present embodiment, the target image from which the target region is to be detected is assumed to be an image of a specimen in multiple immunostaining analysis, but the application of the computer 4 is not limited to such specimen analysis. Further, the target region does not have to be a cell region. The computer 4 also functions as a device that generates a trained model.

[0019] The computer 4 has a configuration of a general computer system including a CPU 41, a GPU 42, a ROM 43, a RAM 44, a fixed disk 45, a display 46, an input unit 47, a reading device 48, a communication unit 49, and a bus 40. The CPU 41 performs various arithmetic processing. The GPU 42 performs various arithmetic processing related to image processing at high speed. The ROM 43 stores basic programs. The RAM 44 stores various types of information. The fixed disk 45 performs information storage. The display 46 displays various types of information such as images.

[0020] The input unit 47 includes a keyboard 47a and a mouse 47b that accept input from the operator. The reader 48 reads information from a computer-readable recording medium 9 such as an optical disk, magnetic disk, magneto-optical disk, or memory card. The display 46, keyboard 47a, mouse 47b, and reader 48 are connected to the bus 40 via an interface (I / F). The communication unit 49 sends and receives signals to and from external devices of the computer 4. The bus 40 is a signal circuit that connects the CPU 41, GPU 42, ROM 43, RAM 44, fixed disk 45, display 46, input unit 47, reader 48, and communication unit 49.

[0021] In computer 4, program 91 is read in advance from recording medium 9 via reader 48 and stored in fixed disk 45. Program 91 may also be stored in fixed disk 45 via a network. The CPU 41 and GPU 42 perform arithmetic processing using RAM 44 and fixed disk 45 according to the computer-readable program 91. The CPU 41 and GPU 42 function as arithmetic units. Other configurations besides the CPU 41 and GPU 42 that function as arithmetic units may also be employed.

[0022] Figure 2 is a block diagram showing the functional configuration of the region detection device 1, which is realized when the computer 4 described above performs calculations and other processing according to program 91. All or part of each functional configuration may be realized by dedicated electrical circuits. Furthermore, these functional configurations may be realized by multiple computers. Program 91 is a computer-readable program that instructs the computer 4 to detect target regions from biological staining images using a trained model.

[0023] Of the functional configurations shown in Figure 2, the display unit 11 primarily displays information to the operator, based on the functions of the display 46 in Figure 1. The arithmetic unit 12 is implemented by the CPU 41, GPU 42, ROM 43, RAM 44, fixed disk 45, and their peripheral components. The operation unit 13 receives input from the operator and is primarily implemented by the input unit 47 in Figure 1. The storage unit 14 is primarily implemented by the RAM 44 and fixed disk 45. Various devices may be used as the storage unit 14, as long as they are capable of storing information.

[0024] The memory unit 14 is pre-stored with data for multiple target images. In Figure 2, the data for the target images is shown as "target image data 51". The processing of the target images in the following description is, more precisely, processing of the target image data 51. The same applies to other images. The target images are biological staining images that are to be input into the trained model 125 described later, in which the target region will be detected. Detection of the target region is synonymous with "extraction of the target region".

[0025] The memory unit 14 also stores training image data. Training images are images used to generate the trained model 125. In Figure 2, the training image data is shown as "training image data 52". The processing of training images in the following description is, more precisely, processing of training image data 52. Annotation image data 53 is associated with training image data 52. Annotation image data 53 is image data in which the region to be detected as a target region in the training image is indicated. Multiple combinations of training image data 52 and annotation image data 53 are stored in the memory unit 14 as a training dataset 520.

[0026] The arithmetic unit 12 comprises, functionally, a conversion unit 121, a correction function acquisition unit 122, a correction unit 123, an inverse conversion unit 124, and a learning unit 126. The arithmetic unit 12 also stores a trained model 125. The trained model 125 is a model generated when the learning unit 126 trains the learning model using the training dataset 520, thereby determining the values ​​of various parameters included in the learning model. The training here is so-called deep learning, but other machine learning methods may be used. The trained model 125 has the function of outputting region detection image data 54 in which a target region has been detected when image data is input. In the region detection device 1, the learning unit 126 may be omitted, in which case training is performed by a separately provided learning device to generate the trained model 125, and the trained model 125 and the training dataset 520 are taken into the region detection device 1.

[0027] Figures 3 and 4 show the flow of the region extraction method performed by computer 4. Before the processes shown in Figures 3 and 4 are executed, a trained model 125 is generated using the training dataset 520 and prepared in the calculation unit 12.

[0028] First, as shown in Figure 2, the storage unit 14 is prepared with training image data 52, which is data of multiple training images, and target image data 51, which is data of at least one target image (step S11). The number of target images may be one or two or more. Next, one training image is selected, and the conversion unit 121 converts the pixel values ​​of each pixel in the training image into an HSV color component group (step S12).

[0029] In this embodiment, the pixel values ​​of the training image before conversion are the values ​​of the red (R), green (G), and blue (B) color components. The pixel values ​​after conversion, that is, the values ​​of the three color components, are the hue (H), saturation (S), and lightness (V) color components. In this embodiment, the three color components are referred to as the "color component group." Also, in this embodiment, as will be described later, we focus on saturation among the converted color component group, so saturation will also be referred to as the "color component of interest" below. The training image converted to the values ​​of the color component group including the color component of interest will be referred to below as the "training-converted image."

[0030] Next, if there are unconverted training images (step S13), the unconverted training images are selected and converted (step S12). By performing step S12 for each of the multiple training images, multiple trained converted images are obtained (steps S12, S13).

[0031] Figure 5 illustrates a training image 61. The training image 61 includes tissue regions 611 and non-tissue regions 612. The tissue region 611 also includes a stained region 613. Figure 6 illustrates a training transformation image 62 in which the saturation value is used as a pixel value (hereinafter referred to as the "saturation image"). In the saturation image 62, the region 622 corresponding to the non-tissue region 612 in Figure 5 appears with extremely low saturation, while the region 623 corresponding to the stained region 613 appears as a region with high saturation. In addition, the region 621 corresponding to the tissue region 611 contains a region 624 with low saturation.

[0032] The correction function acquisition unit 122 acquires a representative value of the saturation of pixels in multiple training transformation images that have a saturation value equal to or greater than a predetermined threshold, as the training saturation representative value (indicated as "training color component representative value" in Figure 3) (step S14). Specifically, first, a mask image is generated in the saturation image 62 of Figure 6 that masks pixels whose saturation value is less than the threshold. Figure 7 is an example of the mask image 63, where the mask region 630 is defined as the region with lower saturation than the region 623 of Figure 6 corresponding to the stained region (regions 621, 622, 624 of Figure 6). In Figure 7, the region corresponding to the stained region is denoted by the code 633. Then, the saturation image 62 is masked with the mask image 63, and the average value of the saturation of the unmasked pixels is obtained. Furthermore, the average of the above average values ​​of all saturation images is obtained as the training saturation representative value. By using a mask to exclude low-saturation areas from the calculation, the influence of insufficiently stained areas or areas that do not show tissue on the calculation of the learned saturation representative value is suppressed.

[0033] In the above description, the representative value is the mean, but the representative value may be various summary statistics such as the 50th percentile (median), 75th percentile (third quartile), and 95th percentile. The same applies to the representative value of target saturation described later. In this embodiment, the representative value is the mean, but the mean in the following description may be replaced with other representative values.

[0034] Furthermore, in calculating the learned saturation representative value, the representative value of saturation at the target pixel in the saturation image may be determined, and the value obtained from the above representative values ​​of all saturation images (i.e., the representative value of the representative values) may be determined as the learned saturation representative value, or the representative value of saturation at the target pixel in all saturation images may be determined as the learned saturation representative value. The same applies to the target saturation representative value described later. Furthermore, the above definition of representative values, and the explanation of the method for determining the learned saturation representative value and target saturation representative value are also the same for the learned color component representative value and target color component representative value described later.

[0035] Next, one target image is selected. This target image is transformed by the transformation unit 121 in the same way as the training image (step S15). That is, the pixel value (value of the RGB color component group) of each pixel of the target image is converted to the value of the color component group (HSV color component group) which includes saturation, which is the color component of interest. Hereinafter, the transformed target image will be referred to as the "transformed target image".

[0036] Next, if there are unconverted target images (step S16), the unconverted target images are selected and converted (step S15). By performing step S15 for each of the multiple target images, multiple converted target images are obtained (steps S15, S16). Note that the number of target images may be as small as one.

[0037] The correction function acquisition unit 122 performs the same processing as in step S14 for all target images (at least one target image) to obtain a representative value of the target saturation. That is, it obtains the average value of the saturation of pixels in at least one target transformation image that have a saturation value equal to or greater than a predetermined threshold as the representative value of the target saturation (indicated as "representative value of the target color component" in Figure 3) (step S17). Specifically, first, in the saturation image, which has the saturation value of each pixel of the target transformation image as a pixel value, a mask image is generated that masks pixels whose saturation value is less than the threshold. The saturation image is then masked with the mask image, and the average value of the saturation of the unmasked pixels is obtained. The average of the above average values ​​of all saturation images is obtained as the representative value of the target saturation.

[0038] Next, the correction function acquisition unit 122 acquires a correction function from the learned saturation representative value and the target saturation representative value (Figure 4: Step S21). Preferably, the correction function is a function that adjusts the saturation value of each pixel of the target image to a representative saturation value of multiple learning images. In other words, the correction function is a function that brings the saturation value of each pixel of the target image closer to a representative saturation value of multiple learning images. In this embodiment, the value obtained by dividing the learned saturation representative value by the target saturation representative value is obtained as a correction coefficient, and a linear function obtained by multiplying the saturation value of each pixel of the target transformed image by the correction coefficient is obtained as the correction function. If the corrected value exceeds the upper limit, the upper limit is used. Note that the correction function may be acquired in any way as long as it functions substantially as a function, and in this embodiment, the correction coefficient is acquired substantially as the correction function.

[0039] Furthermore, the correction function may be a function that makes the saturation value of each pixel in the target image greater than the representative saturation value of multiple training images. In other words, the correction function may enhance the saturation of the target image compared to the saturation of the training images.

[0040] The processing up to step S21 is a preparatory process for detecting a target region from the target image using the trained model 125. Once the preparatory process is complete, one target image is selected and its saturation is corrected (step S22). For saturation correction, the target image may be converted to an HSV color component group, but since the converted target image has already been obtained in step S15, the saturation correction is actually performed on this converted target image. That is, the saturation value of each pixel is multiplied by a correction coefficient. As a result of the correction, the saturation of the converted target image may become higher or lower.

[0041] The corrected target image is inversely transformed in the inverse transformation unit 124, and the HSV color component group is converted to the RGB color component group (step S23). In other words, the color component group is inversely transformed back to the initial color component group. This results in a target image with corrected saturation. Figure 8 is an example of the target image 71 before correction. The target image 71 includes tissue regions 711 and non-tissue regions 712. The tissue region 711 also includes a stained region 713. If the saturation of the target image 71 is lower than the saturation of multiple training images, the correction increases the saturation of the target image 71, and in the corrected target image 72, the stained region 723 becomes clearly visible as shown in Figure 9. In other words, the saturation of the target image is corrected to be equivalent to the saturation of multiple training images.

[0042] The corrected target image is input to the trained model 125, and the target region is detected (step S24). That is, an image showing the target region is obtained. If there are unprocessed target images (step S25), the next target image is selected, and steps S22 to S25 are repeated. As a result, the saturation of at least one target image 71 is corrected using the correction function to obtain a corrected target image 71, which is then input to the trained model 125, and the target region is detected from that at least one target image 71.

[0043] As explained above, in the region detection device 1, the saturation of the target image is corrected using multiple training images used to generate the trained model 125, and then the target image is input to the trained model 125. This reduces the influence of the staining state shown by the training images and the staining state shown by the target image, and improves the accuracy of detecting the target region from the target image using the already existing trained model 125.

[0044] In the above explanation, it was assumed that the operations shown in Figures 3 and 4 are performed on all target images. However, the target images may be grouped based on factors that affect the staining state, such as staining conditions, the equipment used for staining, and the person who performed the staining. In this case, the operations shown in Figures 3 and 4 are performed on each of the multiple groups of target images. This can further improve the accuracy of detecting the target area.

[0045] In the region detection device 1, the operations shown in Figures 3 and 4 may be performed for each target image. That is, the operations shown in Figures 3 and 4 may be performed assuming that there is only one target image. This operation allows for the appropriate detection of the target region even when the staining state differs significantly between target images.

[0046] Next, the correction of the correction function and the threshold for the mask will be described. Figure 10 shows an example of the display of the display unit 11 used to correct the correction coefficient, which is the correction function. That is, the operator determines the final correction coefficient and threshold while referring to the display in Figure 10. In Figure 10, selected training images are displayed on the left, and selected target images are displayed on the right. For example, if multiple training images are displayed side by side on the screen, and the operator selects a training image via one of the operation units 13, that training image will be displayed on the left side of the screen. Similarly, if multiple target images are displayed side by side on the screen, and the operator selects one target image, that target image will be displayed on the right side of the screen. Multiple training images may be selected and displayed side by side. Multiple target images may be selected and displayed side by side.

[0047] In Figure 10, training images are labeled with code 61 and target images with code 71. Above these images, the "Overall Image Saturation Representative Value" is displayed. In this embodiment, the Overall Image Saturation Representative Value above training image 61 is the average value of the saturation of each individual training image (average value after masking) across all training images. Similarly, the Overall Image Saturation Representative Value above target image 71 is the average value of the saturation of each individual target image (average value after masking) across all target images.

[0048] The "Selected Group" indicates the group selected from a set of predefined groups for multiple target images. In this embodiment, the "Group Saturation Representative Value" is the average value of the saturation of the individual target images belonging to the selected group, averaged across all target images in the group. In the "Selected Group," you can select "Group1," "Group2," etc. If "Group1" is selected, the group saturation representative value will be the average value of the saturation of the individual target images belonging to Group1, averaged across all target images in Group1. The group saturation representative value for each group is predetermined.

[0049] In this embodiment, the "Saturation Representative Value" displayed below the training image 61 in Figure 10 is the average saturation value of the displayed training image 61, and the "Saturation Threshold" below it is the threshold value for the mask used when calculating the average saturation value of the training image during the calculation in step S14. In this embodiment, the "Saturation Representative Value" displayed below the target image 71 is the average saturation value of the displayed target image 71, and the "Saturation Threshold" below it is the threshold value for the mask used when calculating the average saturation value of the target image 71 during the calculation in step S17.

[0050] The "Correction Coefficient" below the target image 71 is the coefficient of the correction function, which is a proportional function. The initial value of the correction coefficient is obtained using the learned saturation representative value and the target saturation representative value mentioned above. The "Corrected Saturation Representative Value" is the value obtained by multiplying the saturation representative value of the displayed target image 71 by the correction coefficient. The target image 71 displayed is the image after the saturation has been corrected.

[0051] Here, the operator can change the saturation threshold of the training image 61, as well as the saturation threshold, correction coefficient, and corrected saturation representative value of the target image 71, via the operation unit 13.

[0052] For example, if the saturation threshold of training image 61 is changed, the correction function acquisition unit 122 recalculates the representative saturation value of training image 61. By comparing the representative saturation value of all images with the representative saturation value of the selected training image, it is possible to confirm whether the initial saturation threshold was appropriate. Similarly, if the saturation threshold of target image 71 is changed, the correction function acquisition unit 122 recalculates the representative saturation value of target image 72. By comparing the representative saturation value of all images and the group saturation representative value with the representative saturation value of the selected target image, it is possible to confirm whether the initial saturation threshold was appropriate. The saturation threshold of training image 61 and the saturation threshold of target image 71 may be changed so that when one is changed, the other is always equal. In addition, saturation images (see Figure 6) and mask images based on saturation thresholds (see Figure 7) can be displayed for training images and target images.

[0053] If the correction coefficient of the target image 71 is changed, the corrected saturation representative value is updated, and if the corrected saturation representative value is changed, the correction coefficient is updated. When a change occurs in the correction coefficient, a process is performed to change the saturation of the data of the currently displayed target image 71, and the corrected target image 71 (corresponding to target image 72 in Figure 9) is displayed. The operator looks at the displayed target image 71, the overall saturation representative value, and the group saturation representative value and determines whether the correction coefficient or the corrected saturation representative value is appropriate. Through the above work by the operator, the final saturation threshold and correction coefficient are determined.

[0054] Figure 11 shows the operation flow of the region detection device 1 in correcting the correction coefficient (correction function) described above. The steps shown in Figure 11 may be performed at any stage after step S21 (acquisition of the correction function) in Figure 4.

[0055] First, the system accepts the operator's selection of a target image, and the selected target image 71 is displayed on the display unit 11 (step S31). Hereinafter, the selected target image 71 will be referred to as the "selected target image". As illustrated in Figure 10, the display unit 11 displays the current representative saturation value, saturation threshold, correction coefficient, and corrected saturation threshold of the selected target image 71.

[0056] Next, when the operator changes the correction coefficient or the corrected saturation threshold via the control unit 13, the other value is appropriately changed to match the changed value, as described above. Note that the process of changing either value can be considered as a modification of the correction coefficient (correction function) (step S32).

[0057] The correction unit 123 applies the corrected correction coefficient to the selected image 71 to obtain a selected image with corrected saturation (step S33). Specifically, it converts the color component group of the selected image 71 from RGB to HSV, corrects the saturation using the corrected correction coefficient, and converts the color component group back to RGB. Then, the corrected selected image 71 is displayed on the display unit 11 (step S34).

[0058] If the process shown in Figure 11 is performed immediately after step S21, steps S22 to S25 are executed using the modified correction function. If the process shown in Figure 11 is performed after step S25, steps S22 to S25 are executed again using the modified correction function, and the detection result of the target region is updated.

[0059] As described above, the region detection device 1 allows the operator to modify the correction function while confirming the detection results via the display unit 11. This enables the detection of more appropriate target regions. Note that the correction function may be modified only for some of the target images (each group or individual target images). Furthermore, if the operator changes the saturation threshold for the training images, the representative saturation value for all images may be recalculated using the changed saturation threshold. Similarly, if the operator changes the saturation threshold for the target images, the representative saturation value for all images and the representative saturation value for groups may be recalculated using the changed saturation threshold.

[0060] Next, we will explain an example in which color components other than saturation are corrected as part of the correction in step S22. When an image is color-separated into a group of color components that includes color components other than saturation, saturation information is not obtained, but color information close to the dyed color is obtained.

[0061] The operation flow of the region detection device 1 when using color components other than saturation as the color component of interest is the same as in Figures 3, 4, and 11, except for some details, and the operation is the same except that saturation is replaced with a color component of interest other than saturation. In the following explanation of processing with respect to a color component of interest other than saturation, unless otherwise specified, a color component of interest other than saturation will simply be referred to as the "color component of interest".

[0062] As a color decomposition performed on the training image and target image, for example, the "Color Deconvolution" function is executed using the image analysis software "ImageJ," and [H AEC] is selected as the "Vectors" to decompose the pixel values ​​of the RGB color component group into "H color component," "AEC color component," and "other color component." Then, the AEC color component is used as the color component of interest. The color decomposition is not limited to the above example, but in this embodiment, since the dyeing color is the AEC color component, the above color decomposition is used.

[0063] In order to perform color separation and correct the intensity of the stained color in an image, the color component of interest must be the color corresponding to the stain or a color that approximates that color. In other words, the color component group obtained by converting from the initial color component group RGB of each pixel in the training image includes the color component of interest that corresponds to the color corresponding to the stain or a color that approximates that color. The trained transformed image obtained in step S12 is an image obtained by converting the values ​​of the initial color component group (pixel values) of each pixel to the values ​​of the color component group that includes the color component of interest. Then, the conversion unit 121 performs step S12 for each of the multiple training images, thereby obtaining multiple trained transformed images (step S13).

[0064] In step S14, the correction function acquisition unit 122 acquires representative values ​​of the color components of interest from multiple trained transformed images as the trained color component representative values. The trained color component representative values ​​correspond to the explanation of the trained saturation representative values ​​above, where saturation is replaced with the color component of interest (AEC color component). In this case, unlike the case of saturation, a mask image using a threshold (see Figure 7) is not generated. That is, the average value of the color component of interest from each trained image is simply calculated, and the value obtained by averaging these average values ​​over all trained images is the trained color component representative value. When correcting saturation, not only the dyed color but also various other colors are corrected simultaneously, and a threshold (saturation threshold) is used to exclude undyed areas. Of course, a threshold may also be used when calculating the trained color component representative values.

[0065] The conversion unit 121 and the correction function acquisition unit 122 perform the same processing on at least one target image to obtain representative values ​​of the target color components (steps S15 to S17). That is, the initial color component group values ​​(pixel values) of each pixel of the target image are converted to the values ​​of the color component group that includes the color component of interest to obtain a target converted image, and at least one target converted image is obtained by executing step S15 for each of the at least one target image (step S16). Then, the average value of the color component of interest of the at least one target converted image is obtained as the representative value of the target color component (step S17). In principle, thresholds and mask images are not used in the above processing either.

[0066] Subsequently, similar to the case of saturation, the correction function acquisition unit 122 acquires a correction function from the representative values ​​of the learned color components and the representative values ​​of the target color components (step S21), and the correction unit 123 corrects the color component of interest of the selected target transformed image using the correction function (step S22). The correction function is a function that adjusts the value of the color component of interest of each pixel of the target image to the representative value of the color component of interest of multiple learned images. In other words, the correction function is a function that brings the value of the color component of interest of each pixel of the target image closer to the representative value of the color component of interest of multiple learned images. Furthermore, the inverse transformation unit 124 inversely transforms the color component group in the corrected target transformed image back to the initial color component group, thereby acquiring the corrected target image (step S23).

[0067] Furthermore, the correction function may be a function that makes the value of the color component of interest for each pixel of the target image greater than the representative value of the color component of interest for multiple training images. In other words, the correction function may be one that emphasizes the color component of interest for the target image more than the color component of interest for interest for the training images.

[0068] The corrected target image is input to the trained model 125, and the target region is detected from the target image (step S24). Steps S22 to S24 are performed for each of at least one target image (step S25), so that the target region is detected from all target images.

[0069] As explained above, in the region detection device 1, the target image is input to the trained model 125 after the color component of the target image has been corrected using multiple training images used to generate the trained model 125. This reduces the influence of the staining state shown in the training images and the staining state shown in the target image, and improves the accuracy of detecting the target region from the target image using the already existing trained model 125. By using a color component other than saturation instead of saturation, the accuracy of detecting the target region can be further improved.

[0070] Even when the color component of interest is something other than saturation, the correction function may be modified in the same manner as the display screen in Figure 10 and the operation flow in Figure 11. However, a threshold may or may not be set. The modification of the correction function is the same as described above, but with saturation replaced by the color component of interest. By modifying the correction function while checking the detection results via the display unit 11, an appropriate correction function can be obtained quickly.

[0071] In the above embodiment, we have described cases where saturation is used as the color component of interest and cases where a color component other than saturation is used as the color component of interest. However, when correcting the saturation of the training image or target image, color components other than saturation may also be used. For example, although the correction is performed on saturation, color components other than saturation may be used when obtaining the mask image when acquiring the training saturation representative value (step S14) or the target saturation representative value (step S17).

[0072] Specifically, in step S12, the pixel value of each pixel in the training image is converted to the value of a group of color components that includes the color of interest corresponding to the color of staining or a color that approximates that color, thereby obtaining a training-transformed image. For example, the RGB color component group is converted to "H color component," "AEC color component," and "other color component." Then, in step S13, multiple training-transformed images are obtained. Subsequently, in step S14, the average value of the saturation of pixels in the multiple training-transformed images that have a value of the color of interest (other than saturation) that is above a predetermined threshold is obtained as the training saturation representative value. Note that since the saturation of the training image is needed in step S14, in step S12, the training image is also converted to the HSV color component group (conversion to a group of color components that includes saturation), and a saturation image is obtained.

[0073] Similarly, in step S15, the target image is converted to a target converted image by converting the pixel value of each pixel in the target image to a value of the color component group that includes the aforementioned color component of interest, excluding saturation. Then, in step S16, multiple target converted images are obtained. Subsequently, in step S17, the average value of the saturation of pixels in the multiple target converted images that have a value of the color component of interest above a predetermined threshold is obtained as the target saturation representative value. Note that since the saturation of the target image is required in step S17, in step S15, the target image is also converted to the HSV color component group (conversion to a color component group that includes saturation), and a saturation image is obtained.

[0074] Once the learned saturation representative value and the target saturation representative value are obtained, a correction function is acquired through the process shown in Figure 4, and the target region is detected from the corrected target image. Through the above process, a highly accurate saturation correction function is obtained by utilizing color components other than saturation, while high-speed and highly accurate target region detection is achieved by utilizing high-speed conversion between RGB and HSV.

[0075] In the embodiments described above, examples were explained in which a mask image is generated using saturation as the color of interest component, a correction function is obtained using saturation as the color of interest component, and the target image is corrected using saturation as the color of interest component; an example is described in which a correction function is obtained using color of interest components other than saturation, and the color of interest components other than saturation of the target image are corrected; and an example is described in which a mask image is generated using color of interest components other than saturation, a correction function is obtained using saturation as the color of interest component, and the target image is corrected using saturation as the color of interest component. However, the color of interest components when obtaining the correction function, the color of interest components when making corrections, and the color of interest components when generating the mask image may be combined arbitrarily.

[0076] In this case, in step S12, the conversion unit 121 generates images of the required color components from the training image, including the color component representing saturation and the color component representing the color corresponding to the dye or a color similar to that color (hereinafter also referred to as "non-saturation color component"). Similarly, in step S15, the conversion unit 121 generates images of the required color components from the target image, including the color component representing saturation and the non-saturation color component.

[0077] For example, if an image of the color component of interest (saturation) and an image of the color component of interest (non-saturation) are to be used in a later process, in step S12, the pixel value of each pixel in the training image is converted to a value in the color component group that includes the color component of interest (saturation), thereby obtaining a training-transformed image. Furthermore, another training-transformed image is obtained by converting the pixel value of each pixel in the training image to a value in the color component group that includes the color component of interest (non-saturation). Also, in step S15, the target-transformed image is obtained by converting the pixel value of each pixel in the target image to a value in the color component group that includes the color component of interest (saturation), and another target-transformed image is obtained by converting the pixel value of each pixel in the target image to a value in the color component group that includes the color component of interest (non-saturation).

[0078] Then, in step S14, the representative value of the focus color component of multiple trained transformed images is obtained as the trained color component representative value, using either the focus color component that is saturation or the focus color component that is not saturation as the focus color component. Similarly, in step S17, the representative value of the focus color component of at least one target transformed image is obtained as the target color component representative value, using either the focus color component that is saturation or the focus color component that is not saturation as the focus color component. Furthermore, the correction function is obtained in step S21.

[0079] Subsequently, in steps S22 to S25, the color component of interest, whether it is saturation or not, is treated as the color component of interest, and the color component of interest in the target transformed image corresponding to that color component is corrected. After the inverse transform, the target region is detected by the trained model. The transformation unit 121, the correction unit 123, and the inverse transform unit 124 are used to correct the color component of interest. If these functions are considered as a correction unit in a broad sense, the correction unit corrects the color component of interest in the target image.

[0080] In the above operation, if we refer to the color component of interest when determining the correction function (the color component of interest that is saturation, or the color component of interest that corresponds to the color of the dye or a color that approximates that color) as the "first color component of interest", then in step S14, representative values ​​of the first color component of the multiple training images used to generate the trained model are obtained as the training color component representative values. In step S17, representative values ​​of the first color component of at least one target image are obtained as the target color component representative values. Then, in step S21, the correction function is obtained from the training color component representative values ​​and the target color component representative values.

[0081] On the other hand, if we refer to the color component to be corrected (the color component of interest which is saturation, or the color component of interest which corresponds to the color of the dye or a color that approximates that color) as the "second color component of interest", then in steps S22 to S25, the second color component of interest of at least one target image is corrected using the correction function, and by inputting the corrected at least one target image into the trained model, the target region is detected from that at least one target image. The first color component of interest may be the same as or different from the second color component of interest.

[0082] Furthermore, if a mask image is used in steps S14 and S17, the color component of interest used to generate the mask image may also be a color component of interest that is saturation, or a color component of interest that corresponds to a color or a color that approximates that color. If this color component of interest is called the "third color component of interest," then in step S14, the representative value of the first color component of the pixels in multiple training images whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the representative value of the training color component. In step S17, the representative value of the first color component of the pixels in at least one target image whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the representative value of the target color component. The third color component of interest may be the same as or different from the first or second color component of interest.

[0083] As described above, there are four possible combinations of the first and second color components of interest, and if a mask image is used, there are eight possible combinations of the first to third color components of interest. The explanations given with reference to Figures 1 to 11 apply to all of these combinations. In any combination, the influence of the staining state shown in the training image and the staining state shown in the target image can be reduced, and the accuracy of detecting the target region from the target image using the already existing trained model 125 can be improved.

[0084] The configuration and operation of the region detection device 1 may be modified in various ways.

[0085] In Figures 3 and 4, the order of each step may be changed as appropriate, provided that equivalent processing is performed. For example, step S22 may be performed on all target images, then step S23 may be performed on all target images, and then step S24 may be performed on all target images.

[0086] The mask image used to determine the learned saturation representative value and the target saturation representative value may be generated using an upper threshold in addition to a lower threshold for saturation. That is, not only pixels with low saturation but also pixels with high saturation may be excluded when determining the learned saturation representative value and the target saturation representative value. The learned saturation representative value and the target saturation representative value may be determined after masking obviously unnecessary areas in advance. In this way, the representative value of saturation when determining the correction function may be obtained as an approximate representative value from appropriately selected pixels. Even when determining the correction function based on color components of interest other than saturation, the learned color component representative value and the target color component representative value may be obtained using lower thresholds and upper thresholds for the color components.

[0087] The correction function is not limited to multiplying by a correction coefficient; it may also add, subtract, or divide a constant by the color component values. The correction function is not limited to a linear function.

[0088] When determining the learned saturation representative value or learned color component representative value, color components other than the color component of interest (including saturation) are not used. Therefore, only the value of the color component of interest is obtained, and the values ​​of other color components do not need to be obtained. Regarding training images, the concept of obtaining the values ​​of a group of color components by transforming pixel values ​​includes cases where only the value of a single color component is obtained.

[0089] The vital staining images in which the target region is detected can be various images as long as they show cells. The training images and target images are not limited to images showing pathological tissue. For example, they may be images showing a collection of cells that do not bind to each other.

[0090] In the region detection device 1, corrections are made when the target image is input to the already existing trained model 125. Therefore, it is preferable that no corrections are made to the saturation or color component of the training image when generating the trained model 125. However, corrections to the training image may be made when generating the trained model 125.

[0091] The configurations in the above embodiments and each modified example may be combined as appropriate, as long as they do not contradict each other. [Explanation of Symbols]

[0092] 1. Area detection device 4 Computers 61 Training Images 71 Target Images 91 Programs 121 Conversion section 122 Correction function acquisition section 123 Correction section 124 Inverse Transform Section 125 Pre-trained Models S12-S17, S21-S25, S31-S34 Step

Claims

1. A region detection method for detecting target regions that should be detected based on staining from a biologically stained image using a trained model, a) A step of obtaining a representative value of the first color component of interest, which is the saturation of multiple training images used to generate the trained model, or a representative value of the first color component of interest that corresponds to the color corresponding to the dye or a color similar to said color, as the representative value of the trained color component. b) A step of obtaining a representative value of the first color component of at least one target image, which is a biological staining image in which the target region is to be detected by inputting it into the trained model, as the representative value of the target color component, c) A step of obtaining a correction function from the learned color component representative value and the target color component representative value, d) A step of correcting the second color component of at least one target image, which is saturation, or the second color component of the color corresponding to the dye or a color similar to said color, using the correction function; e) A step of detecting a target region from the at least one target image by inputting the corrected at least one target image into the trained model, A region detection method comprising the following features.

2. A region detection method according to claim 1, In step a) above, a representative value of the first color component of pixels in the plurality of learning images whose third color component of interest is saturation, or whose third color component of interest corresponds to a color corresponding to dyeing or a color similar to said color is greater than or equal to a predetermined threshold is obtained as the learning color component representative value. A region detection method in which, in step b) above, a representative value of the first color component of a pixel in at least one target image whose value of the third color component of interest is equal to or greater than a predetermined threshold is obtained as the representative value of the target color component.

3. A region detection method according to claim 1, A region detection method in which at least one of the target images is one of the groups obtained by dividing multiple target images into multiple groups.

4. A region detection method according to claim 1, A region detection method in which the number of the at least one target image is 1.

5. A region detection method according to claim 1, After step c) above, f) A step of displaying a selected target image selected from the at least one target image, g) A step of modifying the correction function, h) A step of correcting the selected target image using the modified correction function, i) A step of displaying the selected image, A region detection method that further enhances the capabilities of the system.

6. A region detection method according to any one of claims 1 to 5, A region detection method in which the target region is a region at the cellular level.

7. A computer-readable program that causes a computer to detect a target region to be detected based on staining from a biological staining image using a trained model, wherein the execution of the program by the computer is performed by the computer, a) A step of obtaining a representative value of the first color component of interest, which is the saturation of multiple training images used to generate the trained model, or a representative value of the first color component of interest that corresponds to the color corresponding to the dye or a color similar to said color, as the representative value of the trained color component. b) A step of obtaining a representative value of the first color component of at least one target image, which is a biological staining image in which the target region is to be detected by inputting it into the trained model, as the representative value of the target color component, c) A step of obtaining a correction function from the learned color component representative value and the target color component representative value, d) A step of correcting the second color component of at least one target image, which is saturation, or the second color component of the color corresponding to the dye or a color similar to said color, using the correction function; e) A step of detecting a target region from the at least one target image by inputting the corrected at least one target image into the trained model, A program that executes the command.

8. A region detection device for detecting a target region that should be detected based on staining from a biological staining image, A target image, which is a biologically stained image, is input to a trained model that detects a target region from the target image. A correction function acquisition unit acquires a first color component of interest, which is the saturation of a plurality of training images used to generate the aforementioned trained model, and a representative value of the first color component of interest that corresponds to the color corresponding to the dye or a color similar to said color, as a representative value of the first color component of the first color component of at least one target image that is to be input to the trained model and from which the target region is to be detected, as a representative value of the target color component, and acquires a correction function from the representative value of the trained color component and the representative value of the target color component, A correction unit that corrects the second color component of the at least one target image, which is saturation, or the second color component of the color corresponding to the dye or a color similar to said color, using the correction function, A region detection device equipped with the following features.

Images