Image processing device, image processing method, classification device, and program

Abstract

In order to improve the accuracy of classifying a sample cell as benign or malignant in pathological diagnosis, an image processing device (1) comprises: a first acquisition unit (1) for acquiring a target image including a sample cell as a subject; a second acquisition unit (12) for acquiring characteristic information representing the characteristics of a learning image used to train an estimation model trained so as to estimate, upon receiving input of an image including a cell as a subject, whether the cell is a benign cell or a malignant cell; and a generation unit (13) for referencing the characteristic information and generating an image in which the characteristics of the target image are changed.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program. [Background technology]

[0002] In pathological diagnosis, a technology for a trained model that performs image recognition on images containing a specimen as the subject has been disclosed.

[0003] Patent Document 1 discloses a technique for normalizing WSI (Whole Slide Image) data by pre-processing each of multiple WSIs with different dyes to reduce sample variability, and then applying the normalized WSI data to a machine learning model. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japan Special Publication No. 2021-524630 [Overview of the project] [Problems that the invention aims to solve]

[0005] The technology described in Patent Document 1 does not consider the relationship between the images used for training and the images input into the machine learning model for image recognition in pathological diagnosis. Therefore, the technology described in Patent Document 1 has the problem that the accuracy of image recognition of the images input into the machine learning model may be low.

[0006] Such problems can lead to false positives in the benign-malignant classification of sample cells, where benign cells are mistakenly classified as malignant. False positives necessitate retesting even when it's not required. Furthermore, false negatives, where malignant cells are mistakenly classified as benign, can lead to missed diagnoses of cancer. Therefore, high accuracy in image recognition of input images is desirable for benign-malignant classification.

[0007] One aspect of the present invention has been made in view of the above-mentioned problems, and one example of its objective is to provide a technique for improving the accuracy of classifying specimen cells as benign or malignant in pathological diagnosis. [Means for solving the problem]

[0008] An image processing apparatus according to one aspect of the present invention comprises: a first acquisition means for acquiring a target image that includes sample cells as the subject; a second acquisition means for receiving an image that includes cells as the subject as input and acquiring characteristic information that indicates the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and a generation means for generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0009] An image processing method according to one aspect of the present invention includes an image processing device acquiring a target image containing sample cells as the subject, acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant, using the image containing cells as the subject as input, and generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0010] A program according to one aspect of the present invention is a program that causes a computer to function as an image processing apparatus. The program causes the computer to function as: a first acquisition unit that acquires a target image including a specimen cell as a subject; a second acquisition unit that acquires characteristic information indicating characteristics of a learning image used for learning an estimation model that is learned to estimate whether a cell included in an input image as a subject is a benign cell or a malignant cell; and a generation unit that generates an image with characteristics of the target image changed by referring to the characteristic information.

Effect of the Invention

[0011] According to one embodiment of the present invention, in pathological diagnosis, the accuracy of classifying the benignity and malignancy of specimen cells can be improved.

Brief Description of the Drawings

[0012] [Figure 1] It is a block diagram showing the configuration of an image processing apparatus according to Exemplary Embodiment 1 of the present invention. [Figure 2] It is a flowchart showing the image processing method according to Exemplary Embodiment 1 of the present invention. [Figure 3] It is a block diagram showing the configuration of a classification apparatus according to Exemplary Embodiment 2 of the present invention. [Figure 4] It is a flowchart showing an example of the process flow in which the generation unit according to Exemplary Embodiment 2 of the present invention generates an image with the noise level of the target image changed. [Figure 5] It is a flowchart showing an example of the process flow in which the generation unit according to Exemplary Embodiment 2 of the present invention generates an image with the color tone of the target image changed. [Figure 6] It is a diagram showing an example of the histogram of hue in Exemplary Embodiment 2 of the present invention. [Figure 7] [[ID=3​​​​​This flowchart shows another example of the process flow in which the generation unit according to Exemplary Embodiment 2 of the present invention changes the resolution of a target image. [Figure 10] This is a block diagram showing an example of the hardware configuration of an image processing apparatus and a classification apparatus in each exemplary embodiment of the present invention. [Modes for carrying out the invention]

[0013] [Exemplary Embodiment 1] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of the exemplary embodiments described later.

[0014] (Overview of Image Processing Device 1) The image processing apparatus 1 according to this exemplary embodiment is an image processing apparatus that generates an image in which the characteristics of a target image, which includes sample cells as the subject, have been modified. For example, the image processing apparatus 1 takes an image containing cells as input and generates an image in which the characteristics of the target image have been modified to match the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant.

[0015] Image characteristics refer to the properties and features that an image possesses. Examples include noise level, which indicates the amount of noise contained in the image; hue, brightness, and saturation, which constitute the color of the image; resolution, which indicates the number of pixels contained in a given length within the image; and information indicating the size of a given object included as a subject in the image.

[0016] Furthermore, while the specific configuration of the estimation model is not limited to this exemplary embodiment, examples include the use of CNNs (Convolutional Neural Networks), RNNs (Recurrent Neural Networks), or combinations thereof. Alternatively, non-neural network models such as random forests or support vector machines may also be used.

[0017] (Configuration of Image Processing Device 1) The configuration of the image processing apparatus 1 according to this exemplary embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the image processing apparatus 1 according to this exemplary embodiment.

[0018] As shown in Figure 1, the image processing apparatus 1 comprises a first acquisition unit 11, a second acquisition unit 12, and a generation unit 13. In this exemplary embodiment, the first acquisition unit 11, the second acquisition unit 12, and the generation unit 13 are configured to implement the first acquisition means, the second acquisition means, and the generation means, respectively.

[0019] The first acquisition unit 11 acquires a target image that includes the sample cells as the subject. The first acquisition unit 11 supplies the acquired target image to the generation unit 13.

[0020] The second acquisition unit 12 takes an image containing cells as input and acquires characteristic information indicating the characteristics of the training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant. The second acquisition unit 12 supplies the acquired characteristic information to the generation unit 13.

[0021] The generation unit 13 refers to the characteristic information supplied from the second acquisition unit 12 and generates an image with modified characteristics of the target image supplied from the first acquisition unit 11.

[0022] As described above, the image processing apparatus 1 according to this exemplary embodiment employs a configuration comprising: a first acquisition unit 11 that acquires a target image including sample cells as the subject; a second acquisition unit 12 that takes an image including cells as the subject as input and acquires characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and a generation unit 13 that generates an image with modified characteristics of the target image by referring to the characteristic information.

[0023] In other words, the image processing device 1 according to this exemplary embodiment generates an image in which the characteristics of the target image have been modified to match the characteristics of the training image used to train the estimation model. Therefore, according to the image processing device 1 according to this exemplary embodiment, the accuracy of classifying specimen cells as benign or malignant can be improved in pathological diagnosis.

[0024] (Flow of image processing method S1) The flow of the image processing method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the image processing method S1 according to this exemplary embodiment.

[0025] (Step S11) In step S11, the first acquisition unit 11 acquires a target image that includes the sample cells as the subject. The first acquisition unit 11 supplies the acquired target image to the generation unit 13.

[0026] (Step S12) In step S12, the second acquisition unit 12 takes an image containing cells as input and acquires characteristic information indicating the characteristics of the training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant. The second acquisition unit 12 supplies the acquired characteristic information to the generation unit 13.

[0027] (Step S13) In step S13, the generation unit 13 refers to the characteristic information supplied from the second acquisition unit 12 and generates an image in which the characteristics of the target image supplied from the first acquisition unit 11 have been modified.

[0028] As described above, the image processing method S1 according to this exemplary embodiment employs a configuration that includes acquiring a target image containing sample cells as the subject, acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant, using the image containing cells as the subject as input, and generating an image in which the characteristics of the target image have been modified by referring to the characteristic information. For this reason, the image processing method S1 according to this exemplary embodiment provides the same effects as the image processing device 1 described above.

[0029] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0030] (Overview of Classification Device 2) The classification device 2 according to this exemplary embodiment is a device that classifies sample cells to determine whether they are benign or malignant cells.

[0031] For example, the classification device 2 takes an image containing cells as input and uses that image to input an estimation model that has been trained to estimate whether the cells are benign or malignant, thereby classifying whether the sample cells are benign or malignant.

[0032] More specifically, the classification device 2 acquires a target image containing the sample cells as the subject, and generates an image whose characteristics match those of the training image used to train the estimation model. Then, the classification device 2 inputs the generated image into the estimation model and, referring to the results of the estimation model, classifies whether the sample cells are benign or malignant. In other words, the classification device 2 includes the configuration of the image processing device 1 described above, and by inputting the image generated by the image processing device 1 into the estimation model, it classifies whether the sample cells are benign or malignant.

[0033] For example, the training images are digitized images containing sample cells as the subject, while the target images are images of sample cells captured under a microscope. When the training images are digitized, a large number of training images can be acquired, allowing the classification device 2 to improve the accuracy of its estimation model. Furthermore, when the target images are images captured under a microscope, the classification device 2 can be used for cytology in rapid on-site evaluation (ROSE).

[0034] The image characteristics and estimation model are as described above.

[0035] (Configuration of Classification Device 2) The configuration of the classification device 2 according to this exemplary embodiment will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the classification device 2 according to this exemplary embodiment.

[0036] As shown in Figure 3, the classification device 2 comprises a control unit 10, a storage unit 21, and an input / output unit 22.

[0037] The memory unit 21 stores data that the control unit 10, which will be described later, will refer to. Examples of data stored in the memory unit 21 include characteristic information PI that shows the characteristics of the training image TP used to train the estimation model, the target image SP that includes the sample cell as the subject, and the training image TP.

[0038] The input / output unit 22 is an interface that acquires data from other connected devices or outputs data to other connected devices. The input / output unit 22 supplies data acquired from other devices to the control unit 10, and outputs data supplied from the control unit 10 to other devices.

[0039] Furthermore, the input / output unit 22 may be a communication module that communicates with other devices via a network. The specific configuration of the network is not limited to this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), wired LAN, WAN (Wide Area Network), public telephone network, mobile data communication network, or a combination of these networks can be used.

[0040] (Control Unit 10) The control unit 10 controls each component of the classification device 2. Furthermore, as shown in Figure 3, the control unit 10 also functions as a first acquisition unit 11, a second acquisition unit 12, a generation unit 13, and a classification unit 14. In this exemplary embodiment, the first acquisition unit 11, the second acquisition unit 12, the generation unit 13, and the classification unit 14 are configured to implement the first acquisition means, the second acquisition means, the generation means, and the classification means, respectively.

[0041] The first acquisition unit 11 acquires a target image SP that includes the sample cells as the subject. The first acquisition unit 11 stores the acquired target image SP in the storage unit 21.

[0042] The second acquisition unit 12 takes an image containing cells as input and acquires characteristic information PI, which indicates the characteristics of the training image TP used to train an estimation model that has been trained to estimate whether the cells are benign or malignant. The second acquisition unit 12 may be configured to acquire the characteristic information PI itself, or it may acquire one or more training image TPs and derive the characteristic information PI from one or more training image TPs. The second acquisition unit 12 stores the acquired characteristic information PI (and training image TPs) in the storage unit 21.

[0043] Here, as examples of characteristic information PI, we can cite characteristic information PI that shows the average characteristics of each of the multiple training image TPs, and characteristic information PI that shows the median of the variation in characteristics of each of the multiple training image TPs.

[0044] Furthermore, as shown in Figure 3, the second acquisition unit 12 includes an output unit 121.

[0045] The derivation unit 121 derives characteristic information PI from the training image TP. As an example, the derivation unit 121 extracts the characteristics of the training image TP and derives characteristic information PI that represents the extracted characteristics. As another example, the derivation unit 121 extracts the characteristics of each of the multiple training image TPs and derives characteristic information PI that represents the characteristic obtained by averaging each of the extracted characteristics, or the characteristic that is the median of the variability of each of the extracted characteristics.

[0046] The generation unit 13 refers to the characteristic information PI and generates an image with modified characteristics of the target image SP. The generation unit 13 supplies the generated image to the classification unit 14.

[0047] As one example, the generation unit 13 generates an image with a modified noise level of the target image SP. As another example, the generation unit 13 generates an image with a modified hue of the target image SP. As yet another example, the generation unit 13 generates an image with a modified resolution of the target image SP. As yet another example, the generation unit 13 generates an image with a modified size of objects contained in the target image SP.

[0048] Furthermore, the generation unit 13 may generate images in which multiple characteristics of the target image SP have been modified. For example, the generation unit 13 may generate images in which the noise level, hue, and resolution of the target image SP have been modified.

[0049] Furthermore, when the generation unit 13 generates images with multiple characteristics of the target image SP changed, it may generate images with the characteristics changed in a predetermined order, or it may generate images with multiple characteristics of the target image SP changed regardless of the order.

[0050] As an example of a configuration that generates images with modified characteristics in a predetermined order, the generation unit 13 first generates an image with a modified noise level of the target image SP. Next, the generation unit 13 processes the image with the modified noise level to change the color tone and generates an image with the modified color tone. Finally, the generation unit 13 processes the image with the modified color tone to change the resolution and generates an image with the modified resolution.

[0051] The processing flow executed by the generation unit 13 will be described later.

[0052] The classification unit 14 inputs the image supplied from the generation unit 13 into an estimation model and classifies whether the sample cells included as subjects in the image supplied from the generation unit 13 are benign or malignant cells. The classification unit 14 outputs the classification result via the input / output unit 22.

[0053] For example, if the estimation result output from the estimation model indicates that the sample cells are benign cells, the classification unit 14 classifies the sample cells included as subjects in the image supplied from the generation unit 13 as benign cells. On the other hand, if the estimation result output from the estimation model indicates that the sample cells are malignant cells, the classification unit 14 classifies the sample cells included as subjects in the image supplied from the generation unit 13 as malignant cells.

[0054] (Process to generate an image with a modified noise level) An example of the process by which the generation unit 13 generates an image with a modified noise level of the target image SP will be explained with reference to Figure 4. Figure 4 is a flowchart showing an example of the flow of the process S2 in which the generation unit 13 generates an image with a modified noise level of the target image SP according to this exemplary embodiment. In the following, characteristic information PI is information indicating the noise level in the training image TP, and is information indicating the noise level in the background area where no objects are included as subjects, and the process by which the generation unit 13 calculates the noise level by applying a Gaussian filter will be explained.

[0055] (Step S21) In step S21, the generation unit 13 extracts a background region from the target image SP, which is an area that does not contain an object as a subject. The method by which the generation unit 13 extracts the background region is not particularly limited, but one example is to extract an area with a pixel value less than or equal to a predetermined value as the background region.

[0056] (Step S22) In step S22, the generation unit 13 applies a Gaussian filter to the extracted background region. The sigma value of the Gaussian filter applied by the generation unit 13 is not particularly limited, but examples include values ​​such as 1.0 and 1.5.

[0057] (Step S23) In step S23, the generation unit 13 calculates the noise level of the background region after applying the Gaussian filter. For example, if the characteristic information PI indicates the standard deviation of pixel values ​​as the noise level, the generation unit 13 calculates the standard deviation of pixel values ​​in the background region after applying the Gaussian filter as the noise level. Then, the generation unit 13 calculates the difference between the calculated standard deviation of pixel values ​​in the background region and the standard deviation indicated by the characteristic information PI.

[0058] (Step S24) In step S24, the generation unit 13 determines whether the calculated difference is within a predetermined range.

[0059] (Step S25) If, in step S24, it is determined that the calculated difference is not within a predetermined range (step S24: NO), then in step S25, the generation unit 13 changes the sigma value of the applied Gaussian filter.

[0060] For example, if the standard deviation of the pixel values ​​in the background region after applying the Gaussian filter is higher than the standard deviation indicated by the characteristic information PI, the generation unit 13 increases the sigma value by 10%. On the other hand, if the standard deviation of the pixel values ​​in the background region after applying the Gaussian filter is lower than the standard deviation indicated by the characteristic information PI, the generation unit 13 decreases the sigma value by 10%. In other words, the generation unit 13 derives a sigma value such that the difference between the noise level indicated by the characteristic information PI and the noise level in the background region of the target image SP falls within a predetermined range.

[0061] Then, the generation unit 13 returns to the process in step S22 in order to apply the Gaussian filter to the modified sigma value.

[0062] (Step S26) In step S24, if it is determined that the calculated difference is within a predetermined range (step S24: YES), in step S26, the generation unit 13 applies a sigma Gaussian filter to the target image SP such that the difference between the noise level indicated by the characteristic information PI and the noise level in the background region of the target image SP is within a predetermined range, thereby generating an image with a modified noise level. In other words, the generation unit 13 generates an image with a modified noise level of the target image SP such that the noise level in the background region of the extracted target image SP is brought closer to the noise level indicated by the characteristic information PI.

[0063] In this way, the generation unit 13 generates an image in which the noise level of the extracted target image SP has been modified so that the noise level in the background region of the target image SP approaches the noise level indicated by the characteristic information PI. Therefore, the generation unit 13 can input an image with a noise level equivalent to that of the training image TP into the estimation model, thereby improving the accuracy of the classification of benign and malignant specimen cells in pathological diagnosis.

[0064] (Process 1 for generating an image with altered color tones) An example of the process by which the generation unit 13 generates an image with altered hue SP will be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of the flow of the process S3 in which the generation unit 13 generates an image with altered hue SP according to this exemplary embodiment. In the following, characteristic information PI is information indicating the hue in the training image TP, and is information indicating the hue of the object region which contains an object as the subject. As an example, we will explain the case where characteristic information PI is information indicating the histograms of hue, saturation, and brightness in the object region of the training image TP.

[0065] (Step S31) In step S31, the generation unit 13 extracts object regions in the target image SP, which are areas that contain objects as subjects. The method by which the generation unit 13 extracts object regions is not particularly limited, but one example is to extract areas with a predetermined pixel value or higher as object regions.

[0066] (Step S32) In step S32, the generation unit 13 generates a histogram of the hue of the extracted object region as information indicating the hue of the extracted object region. As an example, the generation unit 13 generates histograms of the hue, saturation, and lightness of the extracted object region.

[0067] (Step S33) In step S33, the generation unit 13 generates an image in which the hue of the object region of the extracted target image SP is modified to approximate the hue indicated by the characteristic information PI. An example of the processing of the generation unit 13 in step S33 will be explained with reference to Figure 6. Figure 6 is a diagram showing an example of a hue histogram in this exemplary embodiment. The hue histogram will be explained below, but the same applies to the saturation and brightness histograms.

[0068] When the histogram of the hue indicated by the characteristic information PI is the histogram shown on the upper side of FIG. 6, the generation unit 13 calculates the average value mean in the histogram of the hue indicated by the characteristic information PI sp1 , the left end a of the 95% confidence interval pi and the right end b pi .

[0069] Next, when the histogram of the hue in the object region of the target image SP generated in step S32 is the histogram in the middle of FIG. 6, the generation unit 13 similarly calculates the average value mean in the histogram of the hue in the object region of the target image SP sp1 , the left end a of the 95% confidence interval sp1 and the right end b sp1 .

[0070] Then, the generation unit 13 calculates the changed value Out using the following formula (1).

[0071]

Equation

[0072] That is, the generation unit 13 uses formula (1) to calculate, in the 95% confidence interval of each histogram, for a certain hue value In in the target image SP, the value Var obtained by subtracting the lower limit value from the upper limit value of the 95% confidence interval at the certain hue value poi1 to be that of the target histogram Var tarBy changing it to match, it converts to the target hue value.

[0073] The histogram of hue in the object region of the converted image is shown at the bottom of Figure 6. As shown at the bottom of Figure 6, the mean of the hue histogram in the object region of the converted image is sp2 , the leftmost end a of the 95% confidence interval sp2 and rightmost b sp2 This is in the object region of the target image SP. Hue The mean of the histogram sp1 , the leftmost end a of the 95% confidence interval sp1 and rightmost b sp1 In contrast, the mean of the histogram shown by characteristic information PI pi , the leftmost end a of the 95% confidence interval pi and rightmost b pi It's getting closer.

[0074] In this way, the generation unit 13 generates an image in which the color tone in the object region of the target image SP is modified to approximate the color tone indicated by the characteristic information PI. Therefore, the generation unit 13 can input an image having a color tone equivalent to that of the training image TP into the estimation model, thereby improving the accuracy of the benign / malignant classification of specimen cells in pathological diagnosis.

[0075] (Process 2 for generating an image with altered color tones) In the process of changing the color tone of the target image SP, the generation unit 13 may change the color tone of the background area of ​​the target image SP in addition to the object area of ​​the target image SP.

[0076] As an example, the generation unit 13 may change the pixel values ​​RGB of the background region of the target image SP to predetermined values ​​(for example, if it is a 256-tone image, then (R,G,B)=(230,230,230)).

[0077] As another example, the generation unit 13 may be configured to refer to the background region of the training image TP. For example, the generation unit 13 may obtain the average value of the pixel values ​​in the background region of the training image TP and change the pixel values ​​of the background region of the target image SP to this average value. As yet another example, the generation unit 13 may extract the background regions of multiple training images TP and calculate the average value of the pixel values ​​in each extracted background region. Then, the generation unit 13 may change the pixel values ​​of the background region of the target image SP to the calculated average value. In this case, the generation unit 13 may use the median of the variation in pixel values ​​instead of the average value of the pixel values.

[0078] (Process 1 for generating images with changed resolution) An example of the process by which the generation unit 13 changes the resolution of the target image SP will be explained with reference to Figure 7. Figure 7 is a flowchart showing an example of the flow of the process S4 in which the generation unit 13 changes the resolution of the target image SP according to this exemplary embodiment. In the following, characteristic information PI is information indicating the resolution of the training image TP, and the process by which the generation unit 13 refers to an image that includes a micrometer as the subject will be explained.

[0079] (Step S41) In step S41, the generation unit 13 acquires an image of the micrometer taken at the same resolution as the target image SP. An example of a micrometer image will be described with reference to Figure 8. Figure 8 is a diagram showing an example of a micrometer image in this exemplary embodiment. The generation unit 13 acquires an image including the micrometer as the subject, as shown in image P1 of Figure 8.

[0080] (Step S42) In step S42, the generation unit 13 calculates the resolution of the micrometer image.

[0081] As an example, the generation unit 13 first generates a binarized image of image P1, as shown in image P2 in Figure 8. Next, the generation unit 13 performs a closing process by performing dilation and erosion on image P2, as shown in image P3 in Figure 8. Then, as shown in image P4 in Figure 8, the generation unit 13 uses the direction along the horizontal direction of image P4 as the X-axis and the direction perpendicular to the X-axis as the Y-axis, and the coordinates of the leftmost pixel of the micrometer (X L ,Y L ) and the coordinates (X) of the rightmost pixel of the micrometer R ,Y R The following equation (2) is used to detect the ) and calculate the resolution.

[0082]

number

[0083] In other words, the generation unit 13 calculates the resolution by using equation (2) to calculate the number of pixels of the target image SP contained within a predetermined length.

[0084] (Step S43) In step S43, the generation unit 13 generates an image in which the calculated resolution has been changed to the resolution indicated by the characteristic information PI.

[0085] In this way, the generation unit 13 generates an image in which the resolution of the target image SP is changed so that the resolution indicated by the characteristic information PI is the same as the resolution of the target image SP. Therefore, since an image in which the size of the objects included as subjects in the training image TP is the same as the size of the objects included as subjects in the target image SP can be input to the estimation model, the accuracy of classifying the benign and malignant nature of specimen cells in pathological diagnosis can be improved.

[0086] (Process 2 for generating images with changed resolution) Another example of the process by which the generation unit 13 changes the resolution of the target image SP will be described with reference to Figure 9. Figure 9 is a flowchart showing another example of the flow of the process S5 in which the generation unit 13 changes the resolution of the target image SP according to this exemplary embodiment. In the following, characteristic information PI is information indicating the size of a predetermined object included as a subject in the training image TP, and the generation unit 13 will be described in a configuration in which it extracts a predetermined object included as a subject in the target image SP.

[0087] Here, the predetermined object is not particularly limited, but an object with small individual differences in size is preferred, and red blood cells and lymphocytes are examples of this. With this configuration, the generation unit 13 uses red blood cells or lymphocytes with small individual differences as the standard for the size of objects included in the target image SP, so the size of the object can be calculated appropriately. The following describes the case where the predetermined object is a red blood cell.

[0088] (Step S51) In step S51, the generation unit 13 extracts red blood cells included as subjects in the target image SP as predetermined objects included as subjects in the target image SP. One example of a method by which the generation unit 13 extracts red blood cells included as subjects in the target image SP is a known method using deep learning segmentation (DL segmentation).

[0089] (Step S52) In step S52, the generation unit 13 calculates the size of the extracted red blood cells. The generation unit 13 may also approximate the extracted red blood cells as ellipses and calculate their major and minor axes.

[0090] Furthermore, if multiple red blood cells are extracted, for example, the generation unit 13 may calculate the major and minor axes of the red blood cell with the largest area. As another example, the generation unit 13 may calculate the major and minor axes of the red blood cell that represents the median of the area variation among the multiple red blood cells. As yet another example, the generation unit 13 may calculate the average values ​​of the major and minor axes of the multiple red blood cells.

[0091] (Step S53) In step S53, the generation unit 13 generates an image in which the resolution of the target image SP is changed so that the calculated size (major axis and minor axis) of the red blood cells matches the size (major axis and minor axis) indicated by the characteristic information PI.

[0092] In this way, the generation unit 13 generates an image in which the resolution of the target image SP is changed so that the size of a predetermined object indicated by the characteristic information PI is equivalent to the size of a predetermined object included as a subject in the target image SP. Therefore, since an image in which the size of an object included as a subject in the training image TP is equivalent to the size of an object included as a subject in the target image SP can be input to the estimation model, the accuracy of classifying specimen cells as benign or malignant in pathological diagnosis can be improved.

[0093] (Effect of Classification Device 2) As described above, in the classification device 2 according to this exemplary embodiment, an image containing cells as the subject is taken as input, and an image is generated by modifying the characteristics of a target image SP containing sample cells, by referring to characteristic information PI that indicates the characteristics of a training image TP used to train an estimation model that has been trained to estimate whether the cells are benign or malignant. Then, in the classification device 2 according to this exemplary embodiment, the generated image is input to the estimation model.

[0094] In other words, the classification device 2 according to this exemplary embodiment generates an image in which the characteristics of the target image SP are modified to match the characteristics of the training image TP used to train the estimation model, and then inputs this image into the estimation model. Therefore, the classification device 2 according to this exemplary embodiment has the effect of improving the accuracy of classifying specimen cells as benign or malignant in pathological diagnosis.

[0095] [Examples of implementation using software] Some or all of the functions of the image processing device 1 and the classification device 2 may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0096] In the latter case, the image processing device 1 and the classification device 2 are implemented by a computer that executes instructions for a program, which is software that implements each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Computer C comprises at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as the image processing device 1 and the classification device 2. In computer C, the processor C1 reads the program P from the memory C2 and executes it, thereby implementing the functions of the image processing device 1 and the classification device 2.

[0097] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0098] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0099] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0100] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0101] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0102] (Note 1) An image processing apparatus comprising: a first acquisition means for acquiring a target image containing sample cells as the subject; a second acquisition means for taking an image containing cells as the subject as input and acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and a generation means for generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0103] (Note 2) The characteristic information is information indicating the noise level in the training image, and more particularly, information indicating the noise level in a background region that does not contain an object as a subject, and the generation means extracts a background region from the target image and generates an image in which the noise level of the target image has been modified so that the noise level in the extracted background region of the target image approaches the noise level indicated by the characteristic information, as described in Appendix 1 of the image processing apparatus.

[0104] (Note 3) The image processing apparatus according to Appendix 2, wherein the generation means applies a Gaussian filter to the background region of the target image to calculate the noise level, and derives a sigma value such that the difference between the noise level indicated by the characteristic information and the noise level in the background region of the target image is within a predetermined range.

[0105] (Note 4) The characteristic information is information indicating the hue in the training image, and is information indicating the hue in the object region which includes an object as the subject, and the generation means extracts the object region from the target image and generates an image which is modified so that the hue in the object region of the extracted target image is close to the hue indicated by the characteristic information, as described in any one of the items 1 to 3 of the image processing apparatus.

[0106] (Note 5) The image processing apparatus described in Appendix 4, wherein the characteristic information is information showing histograms of hue, saturation, and brightness in the object region of the training image.

[0107] (Note 6) The image processing apparatus according to any one of the appendices 1 to 5, wherein the characteristic information is information indicating the resolution of the training image, and the generation means generates an image in which the resolution of the target image is changed to the resolution indicated by the characteristic information by referring to a micrometer image taken at the same resolution as the target image.

[0108] (Note 7) The characteristic information is information indicating the size of a predetermined object included as a subject in the training image, and the generation means extracts a predetermined object included as a subject in the target image, and generates an image in which the resolution of the target image is changed so that the size of the extracted object in the target image is the size indicated by the characteristic information, as described in any one of the items in Appendix 1 to 6.

[0109] (Note 8) The image processing apparatus described in Appendix 7, wherein the predetermined object is a red blood cell or a lymphocyte.

[0110] (Note 9) The image processing apparatus according to any one of the appendices 1 to 8, wherein the second acquisition means is an acquisition means for acquiring the training image and a derivation means for deriving characteristic information from the training image.

[0111] (Note 10) The image processing apparatus according to any one of the appendices 1 to 9, wherein the second acquisition means acquires characteristic information showing the average of the characteristics in each of the multiple training images, or characteristic information showing the median of the variation in the characteristics in each of the multiple training images.

[0112] (Note 11) The image processing apparatus according to any one of the appendices 1 to 10, wherein the learning image is a digitized image containing sample cells as the subject, and the target image is an image captured from a microscope showing the sample cells.

[0113] (Note 12) A classification device comprising a classification means for inputting an image generated by the image processing device described in Appendix 1 into the estimation model, and classifying whether the sample cells included as subjects in the image generated by the image processing device described in Appendix 1 are benign or malignant cells.

[0114] (Note 13) An image processing method comprising: an image processing device acquiring a target image containing sample cells as the subject; taking an image containing cells as the subject as input and acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0115] (Note 14) A program that causes a computer to function as an image processing device, the program causing the computer to function as: a first acquisition means for acquiring a target image containing sample cells as the subject; a second acquisition means for taking an image containing cells as the subject as input and acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and a generation means for generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0116] [Additional Note 3] Some or all of the embodiments described above can also be expressed as follows:

[0117] An image processing device comprising at least one processor, the processor performing a first acquisition process for acquiring a target image containing sample cells as the subject; a second acquisition process for taking an image containing cells as the subject as input and acquiring characteristic information indicating the characteristics of a training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant; and a generation process for generating an image in which the characteristics of the target image have been modified by referring to the characteristic information.

[0118] Furthermore, this image processing device may also be equipped with memory, and this memory may store a program that causes the processor to execute the first acquisition process, the second acquisition process, and the generation process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of symbols]

[0119] 1 Image Processing Device 2 Classification device 10 Control Unit 11. First acquisition section 12 Second acquisition section 13 Generation part 14 Classification section 121 Derivation part

Claims

1. A first acquisition means for acquiring a target image that includes sample cells as the subject, A second acquisition means that takes an image containing cells as input and acquires characteristic information indicating the characteristics of the training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant, A generation means that generates an image in which the characteristics of the target image have been modified by referring to the characteristic information, Equipped with, The characteristic information is information indicating the noise level in the training image, and is information indicating the noise level in the background area where no object is included as the subject. The generating means is The background region is extracted from the aforementioned target image. An image is generated in which the noise level of the target image is modified so that the noise level in the background region of the extracted target image approaches the noise level indicated by the characteristic information. The characteristic information is information indicating the hue in the training image, and is information indicating the hue in the object region which includes an object as the subject. The generating means is Extract the object region from the aforementioned target image, An image processing device that generates an image in which the color tone of the object region of the extracted target image is modified to approximate the color tone indicated by the characteristic information.

2. The characteristic information is information indicating the resolution of the training image, The generation means refers to a micrometer image taken at the same resolution as the target image and generates an image in which the resolution of the target image has been changed to the resolution indicated by the characteristic information. The image processing apparatus according to claim 1.

3. The characteristic information is information indicating the size of a predetermined object included as a subject in the training image, The generating means is A predetermined object included as a subject in the aforementioned target image is extracted, An image is generated in which the resolution of the target image is changed so that the size of the objects in the extracted target image matches the size indicated by the characteristic information. The image processing apparatus according to claim 1.

4. The second acquisition means described above is: The aforementioned training images are acquired, The system includes a derivation means for deriving characteristic information from the aforementioned training images. The image processing apparatus according to claim 1.

5. The second acquisition means acquires characteristic information that shows the average characteristics of each of the multiple training images, or characteristic information that shows the median of the variation in the characteristics of each of the multiple training images. The image processing apparatus according to claim 1.

6. A classification means that inputs an image generated by the image processing apparatus described in claim 1 into the estimation model, and classifies whether the sample cells included as subjects in the image generated by the image processing apparatus described in claim 1 are benign or malignant cells. A classification device equipped with the following features.

7. The image processing device To acquire target images that include sample cells as subjects, This involves taking an image containing cells as input and obtaining characteristic information that indicates the characteristics of the training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant, The process involves generating an image with modified characteristics of the target image by referring to the aforementioned characteristic information, Includes, The characteristic information is information indicating the noise level in the training image, and is information indicating the noise level in the background area where no object is included as the subject. The above generation is, The background region is extracted from the aforementioned target image. An image is generated in which the noise level of the target image is modified so that the noise level in the background region of the extracted target image approaches the noise level indicated by the characteristic information. The characteristic information is information indicating the hue in the training image, and is information indicating the hue in the object region which includes an object as the subject. The above generation is, Extract the object region from the aforementioned target image, An image processing method that generates an image in which the color tone of the object region of the extracted target image is modified to approximate the color tone indicated by the characteristic information.

8. The characteristic information is information indicating the resolution of the learning image, The above generation involves referencing a micrometer image taken at the same resolution as the target image, and generating an image in which the resolution of the target image is changed to the resolution indicated by the characteristic information. The image processing method according to claim 7.

9. The characteristic information is information indicating the size of a predetermined object included as a subject in the learning image, The above generation is, A predetermined object included as a subject in the aforementioned target image is extracted, An image is generated in which the resolution of the target image is changed so that the size of the objects in the extracted target image matches the size indicated by the characteristic information. The image processing method according to claim 7.

10. A program that makes a computer function as an image processing device, The aforementioned program, the computer, A first acquisition means for acquiring a target image that includes sample cells as the subject, A second acquisition means that takes an image containing cells as input and acquires characteristic information indicating the characteristics of the training image used to train an estimation model that has been trained to estimate whether the cells are benign or malignant, A generation means that generates an image in which the characteristics of the target image have been modified by referring to the characteristic information, To make it function as, The characteristic information is information indicating the noise level in the training image, and is information indicating the noise level in the background area where no object is included as the subject. The generating means is The background region is extracted from the aforementioned target image. An image is generated in which the noise level of the target image is modified so that the noise level in the background region of the extracted target image approaches the noise level indicated by the characteristic information. The characteristic information is information indicating the hue in the training image, and is information indicating the hue in the object region which includes an object as the subject. The generating means is Extract the object region from the aforementioned target image, A program that generates an image in which the color tone of the object region of the extracted target image is modified to approximate the color tone indicated by the characteristic information.

Images