Information processing device, information processing method, and computer program

Abstract

To provide an information processing device, an information processing method, a computer program, and a method for generating a learning model.SOLUTION: An information processing device includes an acquisition part for acquiring an observation image of a sample, an extraction part for setting a virtual frame on the acquired observation image, and extracting a plurality of partial images while sliding the virtual frame so as to partially overlap each other, a classification part for inputting each of the plurality of partial images extracted by the extraction part to a learned model configured so as to output information of a classification class showing classification of the image in accordance with the input of the image, and acquiring information of a classification class of each of the plurality of partial images, a totalization part for totalizing information of classification classes acquired about the plurality of partial images in each overlapping parts at which a plurality of the partial images overlap, a generation part for determining a classification class which should classify each overlapping parts on the basis of a totalization result, and generating area division images by dividing the observation image into a plurality of kinds of areas in accordance with the classified overlapping parts, and a display part for displaying the generated area division images.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and Computer Program Mu Regarding. [Background technology]

[0002] In the field of pathological diagnosis, an image analysis device has been proposed that inputs images of tissues or cells into a machine learning learning model and determines the presence or absence of a lesion such as a tumor by referring to the calculation results of the learning model (see, for example, Patent Documents 1-3).

[0003] Furthermore, in the field of pathological diagnosis, a microscope device has been proposed that sequentially captures images of small regions of a specimen and seamlessly stitches together the resulting images to create a virtual slide image (see, for example, Patent Document 4). Such a microscope device can achieve both high resolution and a wide field of view. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-95212 [Patent Document 2] Patent No. 6801653 [Patent Document 3] Patent No. 6222584 [Patent Document 4] Japanese Patent Application Publication No. 11-264937 Summary of the Invention [Problem to be solved by the invention]

[0005] In the field of machine learning, the size of an image input to a learning model is generally several hundred pixels square (for example, 512 × 512 pixels). Therefore, when analyzing an image having hundreds of millions to billions of pixels (for example, a virtual slide image) using a machine learning learning model, a partial image of an appropriate size is cut out from the image to be analyzed, and the cut-out partial image is input into the learning model to perform calculations using the learning model.

[0006] However, there is no guarantee that a partial image arbitrarily extracted from an image to be analyzed will contain the tissue or lesion to be detected in a distinguishable state. There are cases where the extracted partial image does not contain the target at all, or where the target is included in the partial image with only a portion missing. Increasing the size of the partial image increases the likelihood that the target will be included, but because the presence or absence of the target is determined using partial images as a unit, structures smaller than the partial image cannot be detected, making it impossible to precisely analyze the observed image. On the other hand, reducing the size of a partial image makes it difficult to visually identify the tissue or lesion contained therein, making it difficult to annotate the partial image in the first place, and it is therefore impossible to obtain sufficient training data to generate a learning model.

[0007] In one aspect, an information processing device and information processing method are provided that extract multiple partial images from an observed image while sliding a virtual frame so that the partial images to be extracted partially overlap each other, aggregate classification class information obtained from a trained model for the overlapping parts of the partial images to determine the classification of the overlapping parts, and generate a region segmentation image corresponding to the classified overlapping parts. This makes it possible to detect structures such as tissues and lesions that appear in areas smaller than the partial images, even when using partial images that are large enough to recognize structural features in the observed image, and to precisely analyze the observed image. and Computer Program M The purpose is to provide. [Means for solving the problem]

[0008] An information processing device according to one aspect includes an acquisition unit that acquires an observation image of a specimen, an extraction unit that sets a virtual frame on the acquired observation image and extracts a plurality of partial images by sliding the virtual frame so that the partial images to be extracted partially overlap each other, a classification unit that inputs each of the plurality of partial images extracted by the extraction unit into a trained model configured to output classification class information indicating the classification of the image in response to the input image, and acquires classification class information for each of the plurality of partial images, an aggregation unit that aggregates the classification class information acquired for the plurality of partial images for each overlapping portion where the plurality of partial images overlap, a generation unit that determines the classification class into which each overlapping portion should be classified based on the aggregation result, and generates a region-segmented image by dividing the observation image into a plurality of types of regions corresponding to the classified overlapping portions, and a display unit that displays the generated region-segmented image. [Effects of the Invention]

[0009] In one aspect, even when a partial image large enough to recognize structural features within the observed image is used, structures such as tissues and lesions that appear in areas smaller than the partial image can be detected, allowing for detailed analysis of the observed image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a diagnosis support system. [Figure 2] FIG. 2 is a block diagram showing the inside of an information processing device. [Figure 3] FIG. 10 is a schematic diagram showing an example of a kidney biopsy image. [Figure 4] FIG. 10 is an explanatory diagram illustrating a patch image generation method. [Figure 5] FIG. 10 is a diagram illustrating an example of classification classes. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of training data storage. [Figure 7] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. [Figure 8]10 is a flowchart showing a procedure of a process executed by the information processing device in a learning phase. [Figure 9] 10 is a flowchart showing the procedure of a model generation process. [Figure 10] FIG. 1 is a schematic diagram illustrating an example of a sliding window. [Figure 11] FIG. 10 is a schematic diagram showing an example of patch image extraction. [Figure 12] FIG. 10 is a conceptual diagram illustrating an example of a tally table. [Figure 13] FIG. 10 is a schematic diagram showing an example of a region segmentation image of a glomerulus. [Figure 14] FIG. 10 is a schematic diagram showing an example of a region-divided image of an inflamed portion and a lesion portion. [Figure 15] FIG. 10 is a schematic diagram showing another example of the region division image. [Figure 16] 10 is a flowchart showing a procedure of a process executed by an information processing device in an operation phase. DETAILED DESCRIPTION OF THE INVENTION

[0011] An information processing device according to an embodiment includes an acquisition unit that acquires an observation image of a specimen, an extraction unit that sets a virtual frame on the acquired observation image and extracts a plurality of partial images by sliding the virtual frame so that the partial images to be extracted partially overlap each other, a classification unit that inputs each of the plurality of partial images extracted by the extraction unit into a trained model configured to output classification class information indicating the classification of the image in response to the input image, and acquires classification class information for each of the plurality of partial images, an aggregation unit that aggregates the classification class information acquired for the plurality of partial images for each overlapping portion where the plurality of partial images overlap, a generation unit that determines the classification class into which each overlapping portion should be classified based on the aggregation result, and generates a region-segmented image by dividing the observation image into a plurality of types of regions corresponding to the classified overlapping portions, and a display unit that displays the generated region-segmented image.

[0012] As used herein, a specimen includes tissues, cells, body fluids, excrement, and processed products thereof collected from a subject. Alternatively, a specimen may be tissues, cells, body fluids, excrement, and processed products thereof collected from a non-human animal.

[0013] Tissues and cells collected from subjects include, but are not limited to, tissues surgically removed from organs such as the esophagus, stomach, colon, liver, kidneys, pancreas, spleen, and gallbladder, as well as tissues and cells collected as biopsies from non-organs such as fat, blood vessels, muscles, and membranous structures. Body fluids include blood, bone marrow, cerebrospinal fluid, lymph, saliva, and tears. Excretions include urine and feces. Processed materials include tissues collected from subjects that have been fixed with a specified fixative, embedded in paraffin, sliced ​​for easy observation, or stained. Examples of stains that can be used include periodic acid Schiff stain (PAS stain), periodic acid methenamine silver stain (PAM stain), and hematoxylin and eosin stain.

[0014] An observation image is a digital image obtained by capturing an image of a specimen using a microscope such as an optical microscope, a transmission electron microscope (TEM), or a scanning electron microscope (SEM), and is composed of a plurality of pixels. The observation image includes a virtual slide image generated by seamlessly stitching together a plurality of captured images captured by a microscope. The observation image may have a pixel count of hundreds of millions to billions of pixels or more.

[0015] The virtual frame is a sliding window set to extract a partial image from the observed image. The size of the virtual frame is set to match the size of the partial image to be input to the trained model. In one example, the size of the virtual frame is 512 x 512 pixels.

[0016] The partial images are obtained by extracting the images within the virtual frame at each position from the observed image while sliding the virtual frame on the observed image. The sliding width of the virtual frame is set so that the partial images to be extracted partially overlap each other. If the size of the virtual frame is 512 x 512 pixels, the sliding width can be set to, for example, 64 pixels in each of the vertical and horizontal directions. By sequentially sliding the virtual frame in 64 x 64 pixel block units in the vertical and horizontal directions, a maximum of 64 partial images will overlap.

[0017] A trained model is a model trained using an appropriate machine learning or deep learning algorithm and is configured to output classification class information indicating the classification of an image in response to an input image. Examples of trained models include models for image segmentation and models for object detection. Models for image segmentation include EfficientNet, SegNet (Segmentation Network), FCN (Fully Convolutional Network), U-Net (U-Shaped Network), and PSPNet (Pyramid Scene Parsing Network). Models for object detection include YOLO (You Only Look Once) and SSD (Single Shot Multi-Box Detector). A trained model is generated by training the model using a dataset containing multiple partial images extracted from an observed image and label information indicating the classification class of each partial image as training data. The label information for each partial image included in the training data is assigned by a doctor or other expert.

[0018] A classification class represents a subset having the same attribute. In this specification, classification classes are set to classify partial images. For example, if the observed image is a renal biopsy image, the classification classes may be set as glomerulus, Bowman's capsule, medulla, cortex, tubule, blood vessel, artery, muscle, fat, inflammation, Kimmelstiel-Wilson lesion, etc. The classification classes may be set appropriately depending on the type of observed image.

[0019] When a partial image is input, the classification class information output from the learning model is tallied for each overlapping portion of the partial image. In the above example, the overlapping portions are formed in 64x64 pixel block units, so the classification class information is tallied for each block. Based on the classification class tallied results, the classification class into which the overlapping portions should be classified is determined.

[0020] A segmented image is an image obtained by dividing an observed image into multiple types of regions. A segmented image is generated by extracting overlapping portions classified into the same classification class as a series of regions. For example, if overlapping portions classified into the glomerular classification class are extracted as a series of regions, a segmented image can be generated in which the image is divided into regions belonging to glomeruli and regions not belonging to glomeruli. The number of divided regions is not limited to two, and may be three or more. In other words, a segmented image may be generated in which regions belonging to classification classes such as glomeruli, Bowman's capsule, medulla, cortex, tubules, blood vessels, arteries, muscles, fat, inflammation, and Kimmelstiel-Wilson lesions are individually divided.

[0021] The information processing device may include a reception unit that receives a selection regarding a type of region to be displayed, the generation unit generates a region segmentation image showing only the region corresponding to the selected type, and the display unit displays the generated region segmentation image superimposed on the observation image. The reception unit is a user interface that receives operations by a user such as a doctor. The user interface includes a keyboard, a mouse, a touch panel, etc. The type selected by the reception unit is the classification class described above. For example, if the classification class of glomerulus is selected as the type, a region segmentation image belonging to glomerulus is displayed superimposed on the observation image.

[0022] The display unit may display information about the proportion of each region in the observation image together with the region-divided image. The proportion of each region in the observation image may be calculated by dividing the area of ​​each region by the area of ​​the observation image. The proportion may be calculated by dividing the number of pixels in each region by the number of pixels in the entire observation image.

[0023] The trained model may be generated using, as training data, a dataset including a plurality of partial images extracted from an observed image and label information indicating the classification class of each partial image. The label information is information for specifying the classification class, and a value (class value) assigned to each classification class, such as 01, 02, ..., is used.

[0024] The training data may include training data in which the partial images are sorted into a first image group having a relatively large variation in pixel values ​​and a second image group having a relatively small variation in pixel values, and the label information is assigned to the partial images sorted into the first image group. The variation in pixel values ​​is an index indicating how much pixel values ​​vary within a certain area. Standard deviation or variance is used as the variation in pixel values. Partial images in which the difference in shading within the image is relatively clear and structural features are clearly visible are selected for the first image group having a relatively large variation in pixel values. On the other hand, partial images in which areas with little difference in shading occupy a large portion of the image are selected for the second image group having a relatively small variation in pixel values. The partial images sorted into the first image group can be used as the partial images used in the training data.

[0025] The training data may further include training data in which the partial images selected to the second image group are sorted into a background image group and a non-background image group based on the variation in pixel values ​​across the entire region of the partial image and the variation in pixel values ​​in the four corner regions of the partial image, and the label information is assigned to the partial images selected to the background image group and the partial images selected to the non-background image group. The partial images selected to the second image group are sorted into a background image group and a non-background image group. The background image group includes images with relatively small variation in pixel values ​​across the entire region of the partial image, and the non-background image group includes images with relatively large variation in pixel values ​​in the four corner regions of the partial image. The partial images used in the training data may be partial images selected into the background image group and the non-background image group.

[0026] The information processing device may include a storage unit that stores identification information of each partial image and classification class information of each partial image obtained by the classification unit in association with each other. The identification information of the partial image is information for identifying each partial image. The identification information of the partial image extracted from the observed image is stored in the storage unit such as a memory in association with classification class information assigned by the annotation.

[0027] The observation image may be a composite image generated by stitching together a plurality of captured images obtained by capturing images of a specimen, and the extraction unit may extract the partial images by sliding the virtual frame in two directions orthogonal to each other of the observation image. The composite image is, for example, a virtual slide image. The composite image is not limited to a virtual slide image, and may be an image stitched together from a plurality of captured images.

[0028] The observed image may be a renal biopsy image, and the trained model may be configured to output information on classification classes including glomeruli, tubules, blood vessels, cortex, medulla, inflammatory areas, and lesions in response to an input of the renal biopsy image. The renal biopsy image is an observed image obtained by renal biopsy, and is an image including tissues such as glomeruli, tubules, blood vessels, cortex, and medulla. The renal biopsy image may include inflammatory areas and lesions as detection targets.

[0029] The size of the partial image may be determined so as to include a glomerulus. When a renal biopsy image is used as the observation image, the size of the partial image is determined so as to include a glomerulus.

[0030] As an application example of the present invention, a diagnosis support system for supporting the diagnosis of a renal biopsy image will be specifically described below with reference to the drawings. (Embodiment 1) 1 is a schematic diagram showing an example of the configuration of a diagnosis support system according to an embodiment. The diagnosis support system according to an embodiment includes an information processing device 1 and a microscope device 2.

[0031] The information processing device 1 is a processing device such as a server computer or personal computer that analyzes images of renal biopsy tissue and outputs information related to diagnostic support. The information processing device 1 is installed, for example, in a medical facility where renal biopsy tissue is diagnosed. Alternatively, the information processing device 1 may be installed outside the medical facility and transmit and receive various information via communication. The information processing device 1 acquires renal biopsy images from the microscope device 2 and outputs information related to diagnostic support by analyzing the acquired renal biopsy images.

[0032] In this embodiment, the renal biopsy image is acquired from the microscope device 2, but the renal biopsy image may be acquired by communication or other means from an external computer.

[0033] 2 is a block diagram showing the inside of the information processing device 1. The information processing device 1 is a dedicated or general-purpose computer equipped with a control unit 11, a storage unit 12, an input unit 13, a communication unit 14, an operation unit 15, a display unit 16, and the like.

[0034] The control unit 11 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM included in the control unit 11 stores control programs and the like that control the operation of each hardware unit included in the information processing device 1. The CPU in the control unit 11 executes the control programs stored in the ROM and various computer programs stored in the storage unit 12 (described later) and controls the operation of each hardware unit, thereby causing the entire device to function as the information processing device of the present application. The RAM included in the control unit 11 temporarily stores data generated during execution of calculations.

[0035] The control unit 11 is configured to include a CPU, a ROM, and a RAM, but may also include one or more arithmetic circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, volatile or non-volatile memory, etc. The control unit 11 may also include functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a start instruction is given until when an end instruction is given, and a counter that counts numbers.

[0036] The storage unit 12 includes a storage device such as a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 12 stores various computer programs executed by the control unit 11 and various data used by the control unit 11.

[0037] The computer programs stored in the storage unit 12 include a diagnostic assistance program PG1. The diagnostic assistance program PG1 is a computer program for causing a computer to execute a process of extracting multiple partial images from a renal biopsy image, inputting each of the partial images into a trained model MD to obtain classification class information for each partial image, aggregating the classification class information for overlapping portions of the partial images, and generating and displaying a region segmentation image based on the aggregation result.

[0038] The computer programs stored in the storage unit 12 may include a model generation program PG2. The model generation program PG2 is a computer program that causes a computer to execute a process of extracting a plurality of partial images from a renal biopsy image, sorting the extracted partial images into a first image group having a relatively large variation in pixel values ​​and a second image group having a relatively small variation in pixel values, and generating a trained model using, as training data, a dataset including the partial images sorted into the first image group and label information for the partial images.

[0039] The computer program stored in storage unit 12 is provided by a non-transitory recording medium RM on which the computer program is readably recorded. The recording medium RM is, for example, a portable memory such as a CD-ROM, a USB memory, an SD (Secure Digital) card, or a CompactFlash (registered trademark). Control unit 11 reads the computer program recorded on recording medium RM using a reading device (not shown) and stores the read computer program in storage unit 12. Alternatively, the computer program stored in storage unit 12 may be provided by communication. In this case, control unit 11 simply acquires the computer program via communication unit 14 and stores the acquired computer program in storage unit 12.

[0040] The storage unit 12 may include a trained model MD. When a partial image of a renal biopsy image is input, the trained model MD is configured to output information on the classification class to which the partial image belongs. The trained model MD is a machine learning learning model constructed, for example, by EfficientNet. The trained model MD is not limited to EfficientNet, and may be constructed using any neural network capable of image segmentation, such as SegNet, FCN, U-Net, or PSPNet. Alternatively, the trained model MD may be constructed using a neural network for object detection, such as YOLO or SSD. The storage unit 12 stores information defining the trained model MD, such as information on the layers constituting the neural network, information on the nodes constituting each layer, and information on weights and biases set between nodes.

[0041] The input unit 13 includes an interface for connecting an external device. One example of an external device connected to the input unit 13 is a microscope device 2 that captures images of renal biopsy tissue and generates an observation image. The microscope device 2 sequentially captures images of an observation target at a predetermined magnification (e.g., 20x or 40x) while moving an objective lens in a direction perpendicular to the optical axis, and seamlessly stitches together the captured images to generate an observation image (also called a virtual slide image or WSI: Whole Slide Image). The information processing device 1 acquires an observation image of renal biopsy tissue (renal biopsy image) from the microscope device 2 connected to the input unit 13. Alternatively, the information processing device 1 may acquire, via communication, a renal biopsy image that has been imported into an external computer and subjected to image processing by the computer.

[0042] The communication unit 14 includes a communication interface for transmitting and receiving various types of data. The communication interface included in the communication unit 14 is, for example, a communication interface conforming to a LAN (Local Area Network) communication standard used in WiFi (registered trademark) or Ethernet (registered trademark). When data to be transmitted is input from the control unit 11, the communication unit 14 transmits the data to be transmitted to a specified destination. Furthermore, when the communication unit 14 receives data transmitted from an external device, it outputs the received data to the control unit 11.

[0043] The operation unit 15 is equipped with operation devices such as a keyboard, a mouse, a touch panel, etc., and accepts various operations and settings by medical personnel, etc. The control unit 11 performs appropriate control based on various operation information provided by the operation unit 15, and stores setting information in the memory unit 12 as necessary.

[0044] The display unit 16 includes a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) display, and displays information to be notified to a doctor or the like in response to an instruction from the control unit 11.

[0045] The information processing device 1 is not limited to a single computer, but may be a computer system consisting of multiple computers and peripheral devices. The information processing device 1 may also be a virtual machine virtually constructed by software. The information processing device 1 according to the embodiment is configured to include an operation unit 15 and a display unit 16, but these are not essential. The information processing device 1 may also be configured to receive operations from an external computer communicably connected to the information processing device 1 and display various information on an external monitor.

[0046] In the learning phase, the information processing device 1 generates a learned model MD by performing machine learning using training data, and in the operation phase, it analyzes renal biopsy images using the learned model MD and outputs information related to diagnostic support based on the analysis results.

[0047] The learning phase will now be described. In the learning phase, the information processing device 1 (A1) acquires a training renal biopsy image, (A2) generates a large number of patch images by dividing the renal biopsy image into predetermined sizes, (A3) generates training data in which labels are assigned to the patch images, and (A4) generates a trained model MD by performing machine learning using the generated training data.

[0048] (A1) Acquisition of renal biopsy images Renal biopsy images are images of renal biopsy tissue (specimens). To observe renal biopsy tissue, the tissue is stained with PAS. PAS staining stains glycoproteins red and is an excellent staining method for observing renal biopsy tissues such as glomeruli and tubules. Alternatively, stains such as PAM staining or hematoxylin and eosin staining may be used.

[0049] A microscope device 2 is used to observe the renal biopsy tissue. The microscope device 2 is an existing optical microscope, and is equipped with a stage on which the observation object (renal biopsy tissue) is placed, an actuator that drives the stage in two axial directions in a horizontal plane, a light source that irradiates the observation object with illumination light, an image sensor that captures the observation object, and an optical system that projects the optical image of the observation object onto the image sensor. The optical system includes an objective lens. The microscope device 2 sequentially captures images of the observation object while driving the stage, and seamlessly stitches together the multiple captured images to generate a virtual slide image (digital image) that achieves high resolution and a wide field of view.

[0050] Figure 3 is a schematic diagram showing an example of a renal biopsy image. Figure 3 is a virtual slide image of renal biopsy tissue. The virtual slide image includes tissues such as glomeruli, Bowman's capsule, basement membrane, epithelial cells, mesangial cells, renal tubules, and blood vessels.

[0051] The information processing device 1 acquires renal biopsy images generated by the microscope device 2 through the input unit 13. The information processing device 1 stores the acquired renal biopsy images in the storage unit 12. The information processing device 1 may acquire renal biopsy images imported into an external computer by means of communication or the like.

[0052] (A2) Patch image generation The control unit 11 of the information processing device 1 generates patch images (partial images) to be used as training data for the trained model MD from the acquired renal biopsy image. FIG. 4 is an explanatory diagram illustrating a patch image generation method. The information processing device 1 generates patch images by dividing the renal biopsy image into multiple regions. In the example of FIG. 4, the renal biopsy image is divided into 7 × 26 rectangular regions, but the number of divisions is determined depending on the size of the original renal biopsy image, the magnification ratio (magnification of the objective lens) during imaging, and the like. Each patch image has approximately 512 × 512 pixels, and the size is adjusted so that it is large enough to recognize the structure of the glomerulus (approximately 90 × 90 μm in actual dimensions).

[0053] The control unit 11 selects patch images to be used for training data from the generated patch images. The selection of patch images is based on the variation in pixel values. The control unit 11 calculates standard deviation (SD) as the variation in pixel values. Alternatively, the control unit 11 may calculate the variance of pixel values.

[0054] For each patch image, the control unit 11 calculates the standard deviation SD_all of the entire patch image area and the standard deviation SD_four corners of the four corner areas. The four corner areas represent the four corner areas when the patch image is divided into, for example, 16 small areas (4 × 4). The control unit 11 compares the calculated standard deviations SD_all and SD_four corners with a threshold. If both are equal to or greater than the threshold, the control unit 11 determines that the pixel value variation is relatively large and selects the patch image as part of the first image group. An example of the threshold is 8, but it may be set appropriately depending on the format of the renal biopsy image, the staining method of the renal biopsy tissue, etc. On the other hand, if any one of the standard deviations is less than the threshold, the pixel variation is determined to be relatively small and the patch image is selected as part of the second image group. Note that in Figure 4, when all five calculated standard deviations are equal to or greater than 8, it is abbreviated as "SD≧8," and when any one of them is less than 8, it is abbreviated as "SD<8."

[0055] The control unit 11 further sorts the patch images sorted into the second image group into a background image group and a non-background image group. The background image group is a group of images in which the standard deviation SD_all of the entire patch image region is less than a threshold, and the non-background image group is a group of images in which the standard deviation SD_all of the entire patch image region is equal to or greater than a threshold and one of the standard deviations SD_four corners of the four corner regions is less than a threshold.

[0056] The control unit 11 selects the patch images selected into the first image group as patch images to be used for training data. The control unit 11 may also select a predetermined number of patch images randomly selected from the patch images selected into the background image group as patch images to be used for training data. Furthermore, the control unit 11 may also select the patch images selected into the non-background image group as patch images to be used for training data.

[0057] (A3) Generating training data The control unit 11 receives labels (classification classes) for the selected patch images, and generates a dataset including a plurality of patch images and label information indicating the classification class of each patch image as training data. Labeling (annotation) of each patch image is performed manually by an expert such as a doctor. FIG. 5 is a diagram showing an example of classification classes. Among the classification classes shown in FIG. 5, class values ​​01 to 13 represent glomerulus-related classification classes, 14 to 26 represent renal tubule-related classification classes, 27 to 32 represent cast-related classification classes, 33 to 35 represent blood vessel-related classification classes, 36 to 40 represent matrix-related classification classes, 41 to 42 represent other classification classes, 43 to 48 represent classification classes unsuitable for analysis, and 49 to 56 represent background-related classification classes. The classification classes of the patch images are not limited to those shown in FIG. 5 and may be set as appropriate.

[0058] When annotation is performed based on the classification classes shown in Fig. 5, each patch image in the first image group is assigned a class value of 01 to 50 (excluding 45). Furthermore, each patch image in the background image group is assigned a class value of 51 to 56, and each patch image in the non-background image group is assigned a class value of 45.

[0059] The control unit 11 stores a data set of a plurality of patch images generated from renal biopsy images and class values ​​(classification classes) assigned to each patch image as training data in the storage unit 12. Fig. 6 is a conceptual diagram showing an example of how training data is stored. The example in Fig. 6 shows a state in which classification class information (class values) is stored in the storage unit 12 in association with patch image IDs that identify patch images.

[0060] (A4) Generation of trained model MD The control unit 11 generates a trained model MD based on the training data stored in the storage unit 12. FIG. 7 is a schematic diagram showing an example of the configuration of the trained model MD. The trained model MD includes an input layer, an intermediate layer (hidden layer), and an output layer. The input layer, intermediate layer, and output layer each include a plurality of nodes. The pixel values ​​of each pixel that constitutes a patch image are input to each node of the input layer. Data indicating the pixel values ​​of each pixel of the patch image input to each node of the input layer is sent to the intermediate layer.

[0061] The intermediate layers are composed of convolutional layers, pooling layers, fully connected layers, etc. For simplicity, the example in Figure 7 shows only two intermediate layers, but the number of intermediate layers included in the trained model MD is not limited to two and may be three or more. The types of intermediate layers are designed appropriately depending on the neural network used. Individual nodes included in each intermediate layer are connected to nodes in the layers provided before and after. Parameters of the trained model MD, including the internal activation level of each node and the strength of connections between nodes (weight coefficients), are determined during the training process. Each node in the intermediate layer outputs a value calculated based on parameters such as internal activation level and weight coefficient to the subsequent node.

[0062] The output layer calculates probabilities using a softmax function based on values ​​input from the intermediate layer. The probabilities output by each node in the output layer represent the probability that the patch image to be classified belongs to each classification class. The trained model MD according to this embodiment is configured to output, for a patch image, a probability P1 that the patch image corresponds to the classification class with class value 01 (glomerulus), a probability P2 that the patch image corresponds to the classification class with class value 02 (glomerulus with mesangial proliferation), ..., and a probability P56 that the patch image corresponds to the classification class with class value 56 (black). When the control unit 11 provides the patch image to be classified to the trained model MD, it references the probabilities output from the output layer of the trained model MD and estimates the classification class with the highest probability as the classification class for that patch image.

[0063] The information processing device 1 generates a trained model MD using training data prepared in advance. That is, the information processing device 1 can generate the trained model MD by learning with a predetermined learning algorithm using patch images included in the training data and labels (correct answer data) assigned to the patch images.

[0064] 8 is a flowchart showing the procedure of processing executed by the information processing device 1 in the learning phase. The control unit 11 of the information processing device 1 acquires a renal biopsy image through the input unit 13 or the communication unit 14 (step S101). The control unit 11 acquires the renal biopsy image, for example, from the microscope device 2 connected to the input unit 13. Alternatively, the control unit 11 acquires the renal biopsy image from an external computer communicatively connected via the communication unit 14. The renal biopsy image acquired by the information processing device 1 is, for example, a virtual slide image formed by seamlessly pasting together captured images of renal biopsy tissue.

[0065] Next, the control unit 11 extracts patch images from the acquired renal biopsy image (step S102). The control unit 11 extracts patch images by dividing the renal biopsy image into multiple regions. The size of the patch images is such that the structure of the glomeruli can be recognized. The control unit 11 stores the patch images extracted from the renal biopsy image in the storage unit 12.

[0066] Next, the control unit 11 selects patch images to be used in the training data (step S103). The control unit 11 calculates the variation in pixel values ​​(e.g., standard deviation SD) for each patch image, and selects images with relatively large variations in pixel values ​​as a first image group and images with relatively small variations in pixel values ​​as a second image group. The control unit 11 also sorts the patch images selected as the second image group into a background image group with small variations in pixel values ​​across the entire region and a non-background image group with small variations in pixel values ​​only in one of the four corner regions.

[0067] Next, the control unit 11 accepts labels for the selected patch images (step S104). The control unit 11 accepts a label (class value) for each patch image by accepting an operation from a specialist such as a doctor via the operation unit 15. When 56 types of classification classes are set as shown in FIG. 5, a class value of 01 to 50 (excluding 45) is assigned to the patch images selected into the first image group. Furthermore, a class value of 51 to 56 is assigned to the patch images selected into the background image group, and a class value of 45 is assigned to the patch images selected into the non-background image group.

[0068] Next, the control unit 11 associates each patch image with the label (class value) assigned to each patch image, and stores the data set in the storage unit 12 as training data for learning (step S105).

[0069] Next, the control unit 11 determines whether to start learning (step S106). When a predetermined number of training data have been obtained, the control unit 11 determines to start learning. Alternatively, the control unit 11 may determine to start learning when an instruction to start learning is received through the operation unit 15. When it is determined not to start learning (S106: NO), the control unit 11 returns the process to step S101.

[0070] If it is determined that learning should be started (S106: YES), the control unit 11 executes a model generation process, which will be described later (step S107).

[0071] 9 is a flowchart showing the steps of the model generation process. The control unit 11 selects a pair of a patch image and a class value assigned to the patch image from the training data stored in the storage unit 12 (step S121), and inputs the selected patch image into a pre-prepared learning model to execute a calculation using the learning model (step S122). Note that, before the start of learning, it is assumed that initial values ​​are set for the parameters characterizing the learning model (internal activity of each node and weights between nodes).

[0072] The control unit 11 evaluates the calculation results of the learning model (step S123) and determines whether learning is complete (step S124). The control unit 11 can evaluate the calculation results of the learning model based on an error function (also referred to as an objective function, loss function, or cost function) based on an estimated value (class value) of the classification class estimated from the calculation results and the class value included as ground truth data. For example, the control unit 11 determines that learning is complete when the error function becomes equal to or less than a threshold (or equal to or greater than a threshold) during the process of optimizing (minimizing or maximizing) the error function using a gradient descent method such as steepest descent.

[0073] If it is determined that the learning is not complete (S124: NO), the control unit 11 updates the parameters including the internal activity of each node and the weights between nodes (step S125), and returns the process to step S121. The control unit 11 can update the parameters of each node using the backpropagation method, which sequentially updates the learning parameters including the internal activity of each node and the weights between nodes from the output layer to the input layer of the learning model.

[0074] When it is determined that the learning is completed (S124: YES), the control unit 11 obtains a learned learning model, and stores the learned model in the storage unit 12 as a learned model MD (step S125).

[0075] In the present embodiment, a configuration has been described in which the trained model MD is generated in the information processing device 1, but it may also be generated in an external server device. The information processing device 1 may acquire the trained model MD generated in the external server device via communication or a recording medium, and install the acquired trained model MD in the storage unit 12.

[0076] In addition, in this embodiment, annotation is performed by having an expert such as a doctor manually label the images, but as learning progresses, the patch images input into the trained model MD and the classification results based on the calculation results of the trained model MD may be used as part of the training data.

[0077] Next, the operation phase will be described. The information processing device 1 transitions to the operation phase after acquiring the trained model MD. In the operation phase, the information processing device 1 (B1) acquires a renal biopsy image of a diagnostic target, (B2) extracts multiple patch images from the renal biopsy image, (B3) inputs each patch image into the trained model MD to classify them into classification classes, (B5) aggregates the classification results, and (B6) generates a region segmentation image based on the aggregated results and displays it on the display unit 16.

[0078] (B1) Acquisition of renal biopsy images The information processing device 1 acquires a renal biopsy image of a diagnostic target. The renal biopsy image of a diagnostic target is an observation image of renal biopsy tissue taken from a subject. An example of a renal biopsy image is a virtual slide image generated by seamlessly stitching together multiple captured images obtained from the microscope device 2. The information processing device 1 acquires the renal biopsy image from the microscope device 2 connected to the input unit 13. Alternatively, the information processing device 1 acquires, via communication, a renal biopsy image that has been imported into an external computer and subjected to image processing by the computer.

[0079] (B2) Extraction of patch images The information processing device 1 sets a sliding window on the acquired renal biopsy image, and extracts patch images (partial images) while sliding the sliding window in two orthogonal directions.

[0080] FIG. 10 is a schematic diagram showing an example of a sliding window SW. The sliding window SW is a virtual frame set for extracting a patch image from a renal biopsy image, and has the same size as the patch image to be extracted. The control unit 11 of the information processing device 1 extracts a patch image by reading pixel information within the sliding window SW from the renal biopsy image at the position where the sliding window SW is set. The control unit 11 stores the extracted patch image in the memory unit 12. The control unit 11 slides the sliding window SW block by block in two orthogonal directions (X direction and Y direction shown in FIG. 10) on the renal biopsy image, sequentially extracting patch images at each position. One block has dimensions 1 / n of the sliding window SW in both the X direction and the Y direction, where n is an integer equal to or greater than 2. FIG. 10 shows an example where n=8.

[0081] FIG. 11 is a schematic diagram showing an example of patch image extraction. The control unit 11 sequentially extracts patch images by sliding the sliding window SW on the renal biopsy image one block at a time. FIG. 11A shows an example in which the sliding window SW is set in the upper left corner of the renal biopsy image and a patch image is extracted at that position. FIG. 11B shows an example in which the sliding window SW is slid one block in the X direction from the state of FIG. 11A and a patch image is extracted at that position. FIG. 11C shows an example in which the sliding window SW and patch image extraction are further advanced, and a patch image is extracted from the sliding window SW that has been slid 10 blocks in the X direction and 6 blocks in the Y direction. In this way, the control unit 11 extracts patch images from the entire renal biopsy image by sliding the sliding window SW one block at a time on the renal biopsy image.

[0082] (B3) Classification into classification classes The control unit 11 inputs multiple patch images extracted from a renal biopsy image of a diagnostic target into the trained model MD and performs calculations using the trained model MD. When a patch image is input, the trained model MD is trained to output the probability that the patch image falls into each classification class from each node in the output layer. The control unit 11 can classify the patch image into one of the classification classes by identifying the classification class with the highest probability based on the calculation results obtained from the trained model MD.

[0083] (B4) Aggregation of classification results In this embodiment, patch images are extracted by sliding the sliding window SW in units of blocks divided into n in the X and Y directions. Therefore, when each patch image is classified, a maximum of n×n classification results can be obtained for each block. Each time a patch image is classified, the control unit 11 records the classification results in an aggregation table. The aggregation table is prepared in, for example, the storage unit 12.

[0084] FIG. 12 is a conceptual diagram illustrating an example of a counting table. Each block is represented by coordinates (x, y), such that the block in the upper left corner of the renal biopsy image is represented by (1,1), the block adjacent to it to the right is represented by (2,1), the block adjacent to that to the right is represented by (3,1), and so on. In this case, a patch image extracted from a sliding window SW set in the upper left corner of the renal biopsy image includes a total of 64 blocks, from (1,1) to (8,8). If classification of this patch image results in a classification of glomeruli (class value = 01), the control unit 11 increments the counters corresponding to the class value 01 by 1 for the 64 blocks from (1,1) to (8,8). As a result, within the range shown in FIG. 12, the counters corresponding to the class value 01 for the blocks from (1,1) to (8,1) are incremented by 1.

[0085] Similarly, when the sliding window SW is slid by one block in the X direction and a patch image is extracted, this patch image includes a total of 64 blocks from (2,1) to (9,8). As a result of classifying this patch image, if the classification result is again a glomerulus (class value = 01), the control unit 11 increments the counters corresponding to the class value 01 by 1 for the 64 blocks from (2,1) to (9,8). As a result, in the range shown in FIG. 12, the counters corresponding to the class value 01 for the blocks from (2,1) to (8,1) are incremented by 1.

[0086] In this way, the control unit 11 extracts patch images while sliding the sliding window SW in block units, classifies each patch image, and tallies the classification results for each block.

[0087] (B5) Generation and display of region-segmented images The control unit 11 generates a segmentation image by referencing the count values ​​recorded in the tallying table. For example, when generating a segmentation image that extracts only the glomerulus region, the control unit 11 may generate an image (segmentation image) of the region where the glomerulus is presumed to be present by referencing the count values ​​recorded in the column for class value 01 in the tallying table. FIG. 13 is a schematic diagram showing an example of a segmentation image of a glomerulus. FIG. 13A shows an example of a kidney biopsy image, and FIG. 13B shows an example of a segmentation image of a glomerulus generated based on the calculation results of the trained model MD. The control unit 11 can generate a contour map as shown in FIG. 13B by separately depicting regions where the count value corresponding to class value 01 falls within the ranges of, for example, 0 to 14, 15 to 24, 25 to 34, 35 to 44, 45 to 54, and 55 to 64. Note that the contour map shown in FIG. 13B indicates that the region is more likely to be a glomerulus as the color changes from white to black.

[0088] Similarly, when generating a region segmentation image in which a region including inflammation has been extracted, the control unit 11 reads out count values ​​recorded in association with class values ​​corresponding to inflammation (class values ​​19, 20, 30, etc.) from the tally table, and generates an image of the region estimated to be inflammation based on the read count values. When generating a region segmentation image in which a region including a lesion has been extracted, the control unit 11 reads out count values ​​recorded in association with class values ​​corresponding to specific lesions (class values ​​02, 03, 04, 06, 08, etc.) from the tally table, and generates an image of the region estimated to be a lesion based on the read count values.

[0089] FIG. 14 is a schematic diagram showing an example of a segmented image of an inflamed area and a lesion. FIG. 14A shows an example of a renal biopsy image, FIG. 14B shows an example of a segmented image of an inflamed area generated based on the calculation results of the trained model MD, and FIG. 14C shows an example of a segmented image of a lesion (Kimmelstiel-Wilson lesion) generated based on the calculation results of the trained model MD. The control unit 11 can generate a contour map such as that shown in FIG. 14B by separately depicting the areas where the count value corresponding to the class value 20 belongs to each of the above ranges. Similarly, the control unit 11 can generate a contour map such as that shown in FIG. 14C by separately depicting the areas where the count value corresponding to the class value 04 belongs to each of the above ranges. Note that in the contour map shown in FIG. 14B, the area indicates a higher probability of inflammation as the color changes from white to black, and in the contour map shown in FIG. 14C, the area indicates a higher probability of a Kimmelstiel-Wilson lesion as the color changes from white to black.

[0090] In this embodiment, as examples, a segmentation image showing a glomerular region (Figure 13B), a segmentation image showing an inflammatory region (Figure 14B), and a segmentation image showing a Kimmelstiel-Wilson lesion region (Figure 14C) are shown, but the control unit 11 can generate segmentation images for each of 56 classification classes.

[0091] The control unit 11 displays the generated region segmentation image on the display unit 16. The region segmentation image displayed on the display unit 16 can serve as diagnostic support information for a doctor. The control unit 11 may display the renal biopsy image of the diagnostic target and the region segmentation image generated from this renal biopsy image side by side, or may display the region segmentation image superimposed on the renal biopsy image.

[0092] 13 shows a region segmentation image of a glomerulus, and FIG. 14 shows region segmentation images of an inflamed portion and a lesion portion, but the control unit 11 is not limited to region segmentation images of a glomerulus, an inflamed portion, and a lesion portion, and can individually generate region segmentation images corresponding to each classification class. The control unit 11 may accept a selection of a classification class via the operation unit 15, and generate a region segmentation image corresponding to the selected classification class.

[0093] In addition, although the present embodiment is configured to generate a segmentation image for each classification class, multiple classification classes may be combined to generate a single segmentation image. For example, since classification classes 22 and 23 both indicate classification classes corresponding to the medulla, control unit 11 may add up the count values ​​of these classification classes and generate a segmentation image based on the total value.

[0094] The control unit 11 may also compare the count value with a threshold value and extract only areas where the count value is equal to or greater than the threshold value to generate a classified image. The threshold value may be a value set for each classification class, or may be a uniform value regardless of the classification class. The threshold value may also be set by a doctor's input operation and may be changed as appropriate after the start of operation.

[0095] Figure 15 is a schematic diagram showing another example of a segmented image. Figure 15A shows an example of a renal biopsy image, and Figure 15B shows another example of a segmented image generated based on the calculation results of the trained model MD. In the example of Figure 15B, the following regions are individually extracted: healthy cortical tubules (class value 14), cortical medulla (class value 21), matrix dilation (class values ​​24, 25), glomerulus (class value 01), and artery (class values ​​33, 34), and a segmented image including these five regions is generated.

[0096] When generating a segmented image including multiple regions, the control unit 11 may calculate the percentage of the entire image that each region occupies and display it together with the segmented image. The area of ​​the entire image is calculated as the number of all blocks included in the renal biopsy image. The area of ​​each segmented image is calculated as the number of blocks included in each region. The control unit 11 can calculate the percentage of the entire image that each region occupies by dividing the area of ​​each segmented image by the area of ​​the entire image.

[0097] The control unit 11 displays the generated region segmentation image on the display unit 16. The region segmentation image displayed on the display unit 16 can serve as diagnostic support information for a doctor. The control unit 11 may display the renal biopsy image of the diagnostic target and the region segmentation image generated from this renal biopsy image side by side, or may display the region segmentation image superimposed on the renal biopsy image.

[0098] 16 is a flowchart showing the procedure of processing executed by the information processing device 1 in the operation phase. The control unit 11 of the information processing device 1 acquires a renal biopsy image through the input unit 13 or the communication unit 14 (step S201). The control unit 11 acquires the renal biopsy image, for example, from the microscope device 2 connected to the input unit 13. Alternatively, the control unit 11 acquires the renal biopsy image from an external computer communicatively connected via the communication unit 14. The renal biopsy image acquired by the information processing device 1 is, for example, a virtual slide image formed by seamlessly pasting together captured images of renal biopsy tissue.

[0099] Next, the control unit 11 extracts a patch image from the acquired renal biopsy image (step S202). The control unit 11 sets a sliding window SW on the acquired renal biopsy image, and extracts the patch image by reading pixel information within the window from the renal biopsy image at the position where the sliding window SW is set.

[0100] Next, the control unit 11 inputs the extracted patch image into the trained model MD and executes a calculation using the trained model MD (step S203). Each node in the input layer of the trained model MD is provided with pixel value data for each pixel constituting the patch image. The data provided to each node in the input layer is output to an adjacent node in the intermediate layer. In the intermediate layer, calculations are performed based on the internal activity of each node and the weights between nodes. The output layer of the trained model MD calculates probabilities using a softmax function based on the values ​​input from the intermediate layer, and outputs the results as calculation results from each node. The probabilities output by each node in the output layer represent the probability that the patch image to be classified belongs to each classification class.

[0101] Next, the control unit 11 classifies the patch image based on the calculation result by the trained model MD (step S204). The control unit 11 can classify the patch image into one of the classification classes by referring to the calculation result obtained from the trained model MD and identifying the classification class with the highest probability.

[0102] Next, the control unit 11 tallies the classification results (step S205). The control unit 11 tallies the classification results by recording them in a tally table. Specifically, the control unit 11 tallies the classification results by incrementing by 1 a counter corresponding to the classified class value for each block constituting the patch image.

[0103] Next, the control unit 11 determines whether classification has been completed for all patch images (step S206). If there are patch images for which classification has not been completed (S206: NO), the control unit 11 slides the sliding window SW (virtual frame) by one block in the X or Y direction (step S207), and returns the process to step S202.

[0104] If it is determined that classification has been completed for all patch images (S206: YES), the control unit 11 generates a region division image based on the classification aggregation result (step S208). The control unit 11 may generate a region division image including one or more predetermined regions. Alternatively, the control unit 11 may identify a classification class with a high occurrence rate and generate a region classification image including regions of, for example, up to the top five classification classes. The control unit 11 may also accept a selection of a classification class via the operation unit 15 and generate a region division image of a region corresponding to the selected classification class. Furthermore, the control unit 11 may generate a region division image including a region corresponding to the selected classification class and a region with a high occurrence rate.

[0105] Next, the control unit 11 displays the generated region division image on the display unit 16 (step S209). The control unit 11 may display the renal biopsy image and the region division image side by side, or may superimpose the region division image on the renal biopsy image. The control unit 11 may calculate the proportion of the entire image that each region occupies, and display it together with the region division image.

[0106] The inventors of the present application collected renal biopsy virtual slide data from 5,002 patients who underwent renal biopsy at 24 institutions in Japan and created a database linked to clinical data. Patch images were extracted from PAS-stained images and used as training data.

[0107] The inventors divided the data into a training and validation dataset and a test set and confirmed that there were no differences in patient backgrounds. The training and validation dataset was further divided into five subsets, and approximately 260,000 patch images were selected from two of these subsets and classified into 56 classes to create training data. A trained model (AI1) was generated based on this training data. AI1 was used to predict patch images from the remaining three subsets, and approximately 1.39 million images that showed high reliability were created as new training data. A new trained model (AI2) was generated using the newly created training data. When AI2 was used to classify the approximately 260,000 training data used in training AI1, the average precision was 0.916 and the average recall was 0.947. Furthermore, a trained model (AI3) was generated using all patch images (approximately 2.4 million images) included in the training and validation datasets as training data. When lesions were visualized using region segmentation images, lesions were successfully detected.

[0108] As described above, in this embodiment, a large observation image, such as a virtual slide image, is divided into patch images and machine learning is performed to construct a trained model MD configured to output classification class information in response to the input patch images. Furthermore, during inference using the trained model MD, multiple patch images are extracted while sliding the sliding window SW so that the patch images overlap, and the classification class information obtained for the overlapping portions is aggregated to generate a segmented image. In this embodiment, an image of 512 x 512 pixels is used as the patch image. However, classification classes can be determined in units of 64 x 64 pixel blocks formed as overlapping portions. By generating a segmented image using these blocks, the structure of tissues and lesions in the observation image can be depicted more precisely. This embodiment is expected to enable standardization of diagnosis of renal biopsy images and provide better medical care to patients.

[0109] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0110] For example, although the information processing device 1 according to the embodiment is configured to output information related to diagnostic support based on a renal biopsy image, the subject of diagnosis is not limited to a renal biopsy image. For example, the method disclosed in the present embodiment can be applied to any observation image obtained by capturing an image of a specimen, such as a cytological image in which cells are diagnosed, or a cancer biopsy image in which cancer is diagnosed.

[0111] Furthermore, although the present embodiment is configured to transition to the operation phase after the completion of the learning phase, the trained model MD may be retrained at an appropriate timing after the start of operation. That is, the control unit 11 may accumulate a data set of patch images input to the trained model MD after the start of operation and classification information estimated based on the calculation results of the trained model MD as training data, and retrain the trained model MD periodically or at a timing instructed by a system administrator or the like. Furthermore, if there is an error in the classification result by the trained model MD, a correction of the classification class may be accepted through the operation unit 15. The control unit 11 may accumulate the patch images input to the trained model MD and the corrected classification class information, and use these data sets as training data for retraining. Note that the retraining procedure is similar to the procedure shown in the flowchart of FIG. 8, and therefore description thereof will be omitted. [Explanation of symbols]

[0112] 1. Information processing equipment 2. Microscope equipment 11 Control section 12 Storage section 13 Input section 14 Communications Department 15 Control section 16 Display PG1 Diagnostic Support Program PG2 Model Generation Program MD pre-trained model

Claims

1. an acquisition unit for acquiring an observation image of the specimen; an extracting unit that sets a virtual frame on the acquired observation image and extracts a plurality of partial images by sliding the virtual frame so that the partial images to be extracted partially overlap each other; a classification unit that inputs each of the plurality of partial images extracted by the extraction unit into a trained model configured to output classification class information indicating the classification of the image in response to input of the image, and acquires classification class information for each of the plurality of partial images; a counting unit that counts classification class information acquired for each overlapping portion where a plurality of partial images overlap; a generation unit that determines a classification class into which each overlapping portion should be classified based on the aggregation result, and generates a region-divided image by dividing the observation image into a plurality of types of regions corresponding to the classified overlapping portions; a display unit that displays the generated region-divided image; An information processing device comprising:

2. a reception unit that receives a selection regarding a type of area to be displayed; the generation unit generates a region segmentation image showing only the region corresponding to the selected type; The display unit displays the generated region division image by superimposing it on the observation image. The information processing device according to claim 1 .

3. The display unit displays information about the proportion of each region in the observation image together with the region-divided image.

3. The information processing device according to claim 1.

4. The trained model is generated using a dataset including a plurality of partial images extracted from an observed image and label information indicating the classification class of each partial image as training data.

4. The information processing device according to claim 1.

5. The training data includes training data obtained by sorting the partial images into a first image group having a relatively large variation in pixel values ​​and a second image group having a relatively small variation in pixel values, and assigning the label information to the partial images sorted into the first image group. The information processing device according to claim 4 .

6. The training data further includes training data in which the partial images selected into the second image group are sorted into a background image group and a non-background image group based on a variation in pixel values ​​in the entire region of the partial image and a variation in pixel values ​​in the four corner regions of the partial image, and the label information is assigned to the partial images selected into the background image group and the partial images selected into the non-background image group. The information processing device according to claim 5 .

7. a storage unit that stores identification information of each partial image and information on the classification class of each partial image obtained from the classification unit in association with each other; 7. The information processing device according to claim 1, further comprising:

8. the observed image is a composite image generated by stitching together a plurality of captured images obtained by capturing images of a specimen, The extraction unit extracts the partial image by sliding the virtual frame in two directions perpendicular to the observation image.

8. The information processing device according to claim 1.

9. the observed image is a kidney biopsy image, The trained model is configured to output information on classification classes including glomeruli, tubules, blood vessels, cortex, medulla, inflammatory areas, and lesions in response to input of a renal biopsy image.

9. The information processing device according to claim 1.

10. The size of the partial image is determined so as to include the glomerulus. The information processing device according to claim 9 .

11. Obtaining an observation image of the specimen; setting a virtual frame on the acquired observation image, and extracting a plurality of partial images by sliding the virtual frame so that the partial images to be extracted partially overlap each other; Inputting each of the extracted partial images into a trained model configured to output classification class information indicating the classification of the image in response to input of the image, and acquiring classification class information for each of the plurality of partial images; aggregating classification class information acquired for the plurality of partial images for each overlapping portion where the plurality of partial images overlap; determining a classification class into which each overlapping portion should be classified based on the aggregation result, and dividing the observed image into a plurality of types of regions corresponding to the classified overlapping portions to generate a region-divided image; Output the generated segmented image An information processing method in which processing is carried out by a computer.

12. On the computer, Obtaining an observation image of the specimen; setting a virtual frame on the acquired observation image, and extracting a plurality of partial images by sliding the virtual frame so that the partial images to be extracted partially overlap each other; Inputting each of the extracted partial images into a trained model configured to output classification class information indicating the classification of the image in response to input of the image, and acquiring classification class information for each of the plurality of partial images; aggregating classification class information acquired for the plurality of partial images for each overlapping portion where the plurality of partial images overlap; determining a classification class into which each overlapping portion should be classified based on the aggregation result, and dividing the observed image into a plurality of types of regions corresponding to the classified overlapping portions to generate a region-divided image; Output the generated segmented image A computer program for executing a process.

Images