IMAGE IDENTIFICATION PROGRAM, IMAGE IDENTIFICATION METHOD, IMAGE IDENTIFICATION DEVICE, AND INFORMATION PROCESSING SYSTEM

Abstract

To accurately identify a state on a lesion from a medical image.SOLUTION: An image identification device 1 executes first identification processing for identifying in which state an imaged area is of states (1)-(5) on a lesion on the basis of an image 3 taken of the inside of a human body. If an identification result by the first identification processing is state (1), the image identification device 1 executes second identification processing for identifying in which state of states (1) and (2) the imaged area is on the basis of the image 3. It is sometimes difficult to discriminate states (1) and (2) from the image taken, for example. When there is a possibility of state (2) being identified as state (1) by mistake in the first identification processing, the image identification device can accurately identify in which state of states (1) and (2) the imaged area is.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image recognition program, an image recognition method, an image recognition apparatus, and an information processing system.

Background Art

[0002] Medical images such as CT (Computed Tomography) images are widely used for diagnosing various diseases such as lung diseases. In image diagnosis using medical images, doctors have to read a large number of images, which places a heavy burden on them. Therefore, there is a need for a technology to assist doctors' diagnosis work in some way by a computer.

[0003] As diagnostic support technologies using medical images, there are the following proposals. For example, a learning device has been proposed that generates a first model for performing image recognition on medical images of a first image quality by learning using a group of medical images of the first image quality, and generates a second model for performing image recognition on medical images of a second image quality by learning using the group of medical images of the second image quality based on the first model. Also, a medical image classification device has been proposed that outputs a first classification result in which each pixel of a medical image is classified into a plurality of types of case regions, and converts the first classification result based on conversion definition information selected from a plurality of pieces of conversion definition information to output a second classification result.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, in the identification process of identifying which of a plurality of states related to a lesion the photographed region is in from a medical image, it may be difficult to distinguish between specific two states depending on the disease. For example, even if it is the state of the same type of lesion, there are cases where the density or brightness on the image is different, making it difficult to distinguish from the state of another lesion. Alternatively, even for the state of a certain lesion, if a specific object such as a blood vessel appears in the image, there are cases where it is misidentified as the state of another lesion.

[0006] On one aspect, an object of the present invention is to provide an image identification program, an image identification method, an image identification apparatus, and an information processing system that enable highly accurate identification of the state related to a lesion from a medical image.

Means for Solving the Problem

[0007] In one proposal, a computer is caused to execute a first identification process of identifying which of a plurality of states related to a lesion the photographed region is in based on an image of the interior of the human body. When the identification result by the first identification process is the first state among the plurality of states, a second identification process of identifying whether the region is in the first state or the second state among the plurality of states based on the image is executed, and an image identification program for executing the process is provided.

[0008] Also, in one proposal, an image identification method is provided in which a computer executes a process similar to the process based on the above image identification program. Furthermore, in one proposal, an image identification apparatus that executes a process similar to the process based on the above image identification program is provided.

[0009] Also, in one aspect, an information processing system is provided that includes the following learning processing unit and image identification unit. In this information processing system, the learning processing unit generates a first learning model for executing a first identification process of identifying which state among a plurality of states related to a lesion is a region of an image obtained by photographing the inside of a human body through learning using a neural network, and generates a second learning model for executing a second identification process of identifying whether the region is in a first state or a second state among the plurality of states by fine-tuning and re-learning the neural network. The image identification unit acquires a photographed image obtained by photographing the inside of the human body to be identified, executes a first identification process of identifying which state among the plurality of states the region of the photographed image is in based on the first learning model, and when the identification result by the first identification process is the first state among the plurality of states, executes a second identification process of identifying whether the region of the photographed image is in the first state or the second state based on the second learning model.

Advantages of the Invention

[0010] In one aspect, it is possible to accurately identify the state related to a lesion from a medical image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] Fig. 1 is a diagram showing an example of the configuration and processing of an image recognition device according to a first embodiment. The image recognition device 1 shown in Fig. 1 is an information processing device that acquires an image 3 of the inside of a human body and recognizes, based on this image 3, which of a plurality of states related to a lesion the imaged region is in. For example, when a cross section of a lung field is imaged, the type of shadow is recognized as one of a plurality of states.

[0013] The image recognition device 1 includes a processing unit 2. The processing unit 2 is realized as, for example, a processor. The processing unit 2 executes the following processes. The processing unit 2 executes a first classification process based on the acquired image 3 (step S1). In this first classification process, it is determined which of the above-mentioned multiple states the photographed area is in. In this embodiment, it is assumed that it is determined which of states (1) to (5) the photographed area is in, as an example.

[0014] The processing unit 2 determines whether the classification result of the first classification process is a predetermined first state among states (1) to (5) (step S2). In the present embodiment, it is assumed that the processing unit 2 determines whether the classification result is state (1) as an example.

[0015] If the identification result by the first identification process is not in state (1), the processing unit 2 outputs the identification result by the first identification process as the final identification result (step S3). As this output, any one of states (2) to (5) will be output.

[0016] On the other hand, if the identification result by the first identification process is in state (1), the processing unit 2 executes a second identification process based on image 3 (step S4). In this second identification process, it is identified whether the photographed area is in state (1) or a second state different from state (1) among the above plurality of states. In the present embodiment, it is assumed that it is identified whether it is in state (1) or state (2). Then, the identification result by the second identification process is output as the final identification result.

[0017] According to the above processing, the accuracy of identifying the state regarding the lesion from the photographed image can be improved. In particular, when it is difficult to distinguish between state (1) and state (2) from the photographed image, and there is a possibility that state (2) is erroneously identified as state (1) in the first identification process, it becomes possible to accurately identify whether it is in state (1) or state (2).

[0018] 〔Second Embodiment〕 Next, a system that can identify a plurality of shadow types in the lung field as a plurality of states regarding a lesion will be described.

[0019] FIG. 2 is a diagram showing a configuration example of a diagnostic support processing system according to the second embodiment. The diagnostic support system shown in FIG. 2 is a system that supports image diagnosis by CT imaging, and includes a learning processing device 100, an image identification device 200, and a CT device 300. Note that the image identification device 200 is an example of the image identification device 1 shown in FIG. 1.

[0020] The CT device 300 captures X-ray CT images of a human body. In this embodiment, the CT device 300 captures a predetermined number of CT slice images of an axial cross section of the chest region while changing the position in the height direction of the human body (direction perpendicular to the axial cross section) at predetermined intervals.

[0021] The image recognition device 200 identifies the lung field region from each slice image captured by the CT device 300 and divides the image of the lung field region into "image blocks" of a certain size. The image recognition device 200 identifies the type of shadow for each image block. This recognition process is performed using a pre-trained learning model.

[0022] The learning processing device 100 generates, by machine learning, a learning model to be used in the shadow type classification process by the image classification device 200. Data of the learning model generated by the learning processing device 100 is read into the image classification device 200, for example, via a network or via a portable recording medium.

[0023] Fig. 3 is a diagram showing an example of the hardware configuration of an image recognition device. The image recognition device 200 is realized, for example, as a computer having the hardware configuration shown in Fig. 3. As shown in Fig. 3, the image recognition device 200 has a processor 201, a RAM (Random Access Memory) 202, an HDD (Hard Disk Drive) 203, a GPU (Graphics Processing Unit) 204, an input interface (I / F) 205, a reader 206, and a communication interface (I / F) 207.

[0024] The processor 201 comprehensively controls the entire image recognition device 200. The processor 201 is, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or PLD (Programmable Logic Device). Further, the processor 201 may be a combination of two or more elements among the CPU, MPU, DSP, ASIC, and PLD.

[0025] The RAM 202 is used as the main memory device of the image recognition device 200. At least a part of the OS (Operating System) program and application programs to be executed by the processor 201 are temporarily stored in the RAM 202. Also, various data necessary for the processing by the processor 201 are stored in the RAM 202.

[0026] The HDD 203 is used as the auxiliary storage device of the image recognition device 200. The OS program, application programs, and various data are stored in the HDD 203. Note that as the auxiliary storage device, other types of non-volatile storage devices such as an SSD (Solid State Drive) can also be used.

[0027] A display device 204a is connected to the GPU 204. The GPU 204 causes an image to be displayed on the display device 204a in accordance with an instruction from the processor 201. Examples of the display device include a liquid crystal display and an organic EL (Electroluminescence) display.

[0028] An input device 205a is connected to the input interface 205. The input interface 205 transmits the signal output from the input device 205a to the processor 201. Examples of the input device 205a include a keyboard and a pointing device. Examples of the pointing device include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

[0029] A portable recording medium 206a is detachably attached to the reading device 206. The reading device 206 reads the data recorded on the portable recording medium 206a and transmits it to the processor 201. Examples of the portable recording medium 206a include an optical disk and a semiconductor memory.

[0030] The communication interface 207 transmits and receives data to and from other devices such as the CT device 300 via a network. With the above hardware configuration, the processing functions of the image identification device 200 can be realized. Note that the learning processing device 100 can also be realized as a computer having a hardware configuration as shown in FIG. 3.

[0031] Next, an overview of the shadow identification process executed by the image identification device 200 will be described. First, FIG. 4 is a diagram showing the basic flow of the shadow identification process. When the chest region of the subject to be photographed is photographed by the CT device 300, the image identification device 200 acquires a plurality of slice images obtained by the photographing. The image identification device 200 extracts the slice images in which the lung field region appears from the acquired slice images (step S11). Hereinafter, the set of the extracted slice images will be referred to as the "lung field image set".

[0032] Next, the image identification device 200 divides each slice image in the lung field image set into image blocks of a certain size (step S12). In this embodiment, as an example, the slice image is divided into image blocks of 16 × 16 pixels. At the same time, the image identification device 200 identifies a lung field region from each slice image in the lung field image set. This identification process is performed using, for example, a learning model for identifying a lung field region that has been trained in advance by machine learning. Through this process, the image blocks of each slice image are classified into image blocks within the lung field region and image blocks outside the lung field region.

[0033] Next, the image identification device 200 identifies the opacity type for each image block in the lung field region (step S13). The opacity type indicates the type of opacity, and in this embodiment, indicates one of ground-glass opacity, infiltrate, honeycomb lung, emphysema, and other opacities. Furthermore, normal is also included as one of the opacity types. Such opacity type identification is performed by a classifier using a learning model for opacity identification that has been trained in advance by machine learning.

[0034] However, very faint ground-glass opacities can occur in viral pneumonia caused by COVID-19 (coronavirus disease 2019). For this reason, when using a single classifier that can distinguish between the six types of opacities mentioned above, areas with ground-glass opacities may be mistakenly classified as normal.

[0035] Furthermore, in the case of viral pneumonia, if the size of the image block is set to a size equivalent to an area of about 1 square centimeter in real space, it is difficult to distinguish between normal areas including blood vessels and infiltrative shadow areas. For this reason, if a single classifier that can distinguish between the six types of shadows mentioned above is used, normal areas may be mistakenly classified as infiltrative shadows.

[0036] Thus, when using a single discriminator, there is a problem that it is difficult to accurately discriminate between ground glass shadows, infiltration shadows, and normal conditions. Therefore, as shown in FIG. 5 below, the image discrimination device 200 of the present embodiment performs two-stage shadow discrimination.

[0037] FIG. 5 is a diagram showing two-stage discrimination processing executed in the second embodiment. As shown in FIG. 5, the image discrimination device 200 executes shadow discrimination processing using a first-stage discriminator (main discriminator 211) and a second-stage discriminator (first sub-discriminator 212 and second sub-discriminator 213).

[0038] The main discriminator 211 discriminates which of the six types of shadow types, namely ground glass shadow, infiltration shadow, honeycomb lung, pulmonary emphysema, other shadows, and normal, the state of the lesion in the input image block is. Here, when the discrimination result by the main discriminator 211 is any of ground glass shadow, honeycomb lung, pulmonary emphysema, and other shadows, the discrimination result is output as the final result as it is.

[0039] On the other hand, when the discrimination result by the main discriminator 211 is normal, the image block is input to the first sub-discriminator 212. The first sub-discriminator 212 discriminates whether the state of the lesion in the input image block is normal or ground glass shadow. By using the first sub-discriminator 212 specialized for discriminating between normal and ground glass shadow in this way, the possibility of misidentifying the area of the ground glass shadow as normal is reduced, and the discrimination accuracy can be improved.

[0040] Also, when the discrimination result by the main discriminator 211 is an infiltration shadow, the image block is input to the second sub-discriminator 213. The second sub-discriminator 213 discriminates whether the state of the lesion in the input image block is an infiltration shadow or normal. By using the second sub-discriminator 213 specialized for discriminating between infiltration shadow and normal in this way, the possibility of misidentifying a normal area (especially a normal area including blood vessels) as an infiltration shadow is reduced, and the discrimination accuracy can be improved.

[0041] In this embodiment, the main classifier 211, the first sub-classifier 212, and the second sub-classifier 213 are all executed using a learning model that has been trained in advance by machine learning. In this embodiment, it is assumed that the learning model is generated by deep learning using a CNN (Convolutional Neural Network).

[0042] In this case, a large number of image blocks to which one of the six shadow types described above is added as a correct label are prepared for each shadow type, and these image blocks are used as training images to train the main classifier 211. Also, a large number of image blocks to which one of two shadow types, normal or ground-glass opacity, is added as a correct label are prepared for each shadow type, and these image blocks are used as training images to train the first sub-classifier 212. Furthermore, a large number of image blocks to which one of two shadow types, infiltrate or normal, is added as a correct label are prepared for each shadow type, and these image blocks are used as training images to train the second sub-classifier 213.

[0043] Here, in order to improve the classification accuracy of the first sub-classifier 212 and the second sub-classifier 213, it is desirable that the training images for these are prepared separately from the training images for the main classifier 211. In particular, it is desirable that the training images labeled with ground-glass shadows among the training images for the first sub-classifier 212 include many images in which faint ground-glass shadows appear. Furthermore, it is desirable that the training images labeled with "normal" among the training images for the second sub-classifier 213 include many images in which blood vessels appear.

[0044] Thus, when attempting to generate a two-stage discriminator as shown in FIG. 5 through learning, teacher images for learning the second-stage discriminator must be prepared separately from the learning of the first-stage discriminator. For this reason, there is a problem that a great deal of effort is required for the task (annotation) of collecting such a large number of teacher images and adding correct labels. For example, when using CT images taken from actual patients in a hospital as teacher images, it is generally difficult to share such teacher images among hospitals, so it becomes necessary to collect and annotate teacher images for each hospital.

[0045] Also, as another method for accurately discriminating between normal and thin ground-glass opacity, and between normal including infiltrative opacity and blood vessels, for example, a method of using one discriminator that discriminates eight classes including "thin ground-glass opacity" and "normal including blood vessels" in addition to the above six types of shadow types can be considered. However, even in this case, for the learning of the discriminator, a large number of teacher images labeled with thin ground-glass opacity and a large number of teacher images labeled with normal including blood vessels are still required.

[0046] Therefore, the learning processing device 100 performs learning of the first sub-discriminator 212 and the second sub-discriminator 213 by finely tuning (fine-tuning) the network obtained by the learning of the main discriminator 211. Thereby, even when using a relatively small number of teacher images for the learning of the first sub-discriminator 212 and the second sub-discriminator 213, sufficient discrimination accuracy can be obtained in these discriminators.

[0047] FIG. 6 is a diagram showing a configuration example of the processing functions provided in the learning processing device. As shown in FIG. 6, the learning processing device 100 includes a storage unit 110, a main discriminator learning unit 121, a first sub-discriminator learning unit 122, and a second sub-discriminator learning unit 123. Note that the storage unit 110 is realized as a storage area of a storage device provided in the learning processing device 100. The processing of the main discriminator learning unit 121, the first sub-discriminator learning unit 122, and the second sub-discriminator learning unit 123 is realized, for example, by a processor provided in the learning processing device 100 executing a predetermined program.

[0048] In the memory unit 110, each data of the learning models 111 to 113 is stored. The learning model 111 is a model of the main discriminator 211 and is generated by the main discriminator learning unit 121. The learning model 112 is a model of the first sub-discriminator 212 and is generated by the first sub-discriminator learning unit 122. The learning model 113 is a model of the second sub-discriminator 213 and is generated by the second sub-discriminator learning unit 123. Note that as each data of the learning models 111 to 113, data indicating the network structure of each model, weight coefficients set between nodes on the network, and the like are stored.

[0049] The main discriminator learning unit 121 uses the teacher image set 131 as learning data and generates a learning model 111 indicating the main discriminator 211 by learning. The teacher image set 131 includes a plurality of image blocks (teacher images) labeled with any one of the shadow types of ground glass shadow, infiltrative shadow, honeycomb lung, emphysema, other shadows, and normal, for each shadow type.

[0050] The first sub-discriminator learning unit 122 uses the teacher image set 132 as learning data and, using a part of the data of the learning model 111, generates a learning model 112 indicating the first sub-discriminator 212 by learning. The teacher image set 132 includes a plurality of image blocks (teacher images) labeled with the shadow types of normal or ground glass shadow, for each shadow type. The image blocks labeled with the ground glass shadow as the shadow type include image blocks in which a thin ground glass shadow appears.

[0051] The second sub-discriminator learning unit 123 uses the teacher image set 133 as learning data and, using a part of the data of the learning model 111, generates a learning model 113 indicating the second sub-discriminator 213 by learning. The teacher image set 133 includes a plurality of image blocks (teacher images) labeled with the shadow types of infiltrative shadow or normal, for each shadow type. The image blocks labeled with the normal shadow type include image blocks in which blood vessels appear.

[0052] FIG. 7 is a diagram showing a method for generating a learning model that shows a sub-identifier. As described above, the learning of the main identifier 211, the first sub-identifier 212, and the second sub-identifier 213 is performed by deep learning using a CNN. The network 211a shown in FIG. 7 is an example of a network (CNN) that realizes the main identifier 211. The network 211a includes an input layer 211a1, an intermediate layer (hidden layer) 211a2, and an output layer 211a3.

[0053] The input layer 211a1 includes, for example, nodes corresponding to the number of pixels (16×16) of the image block. On the other hand, the output layer 211a3 includes nodes corresponding to the above-described six types of shadow types. Each pixel value of each image block included in the teacher image set 131 is input to the input layer 211a1, and the weight coefficients between the nodes are optimized so that the error of the output value (probability) corresponding to the correct label of each image block is minimized by the error backpropagation method or the like. Thereby, the learning model 111 corresponding to the main identifier 211 is generated.

[0054] On the other hand, the learning of the first sub-identifier 212 is performed by fine-tuning the network 211a of the main identifier 211. As shown in FIG. 7, the network 212a that realizes the first sub-identifier 212 includes an input layer 212a1, an intermediate layer (hidden layer) 212a2, and an output layer 212a3.

[0055] The output layer 212a3 includes two nodes corresponding to each of the shadow types of normal and ground glass shadow, and random initial values are set as the weight coefficients between these nodes and the nodes in the previous layer. On the other hand, as the network structures of the input layer 212a1 and the intermediate layer 212a2, the network structures of the input layer 211a1 and the intermediate layer 211a2 in the network 211a of the main identifier 211 are used as they are. Then, as the initial values of the weight coefficients between the nodes in the input layer 212a1 and the intermediate layer 212a2, the weight coefficients at the same positions in the input layer 211a1 and the intermediate layer 211a2 are set, and learning using the teacher image set 132 is started.

[0056] By learning the first sub-identifier 212 in this way, it is possible to generate a learning model 112 of the first sub-identifier 212 that has substantially the same identification accuracy using a smaller number of teacher images as compared to the case of learning without using the learned data of the main identifier 211.

[0057] Although not shown, the learning of the second sub-identifier 213 is also executed in the same manner as above by using the network structures of the input layer 211a1 and the intermediate layer 211a2 in the main identifier 211 as they are and using the weight coefficients between the nodes in these layers as initial values. As a result, it is possible to generate a learning model 113 of the second sub-identifier 213 that has substantially the same identification accuracy using a smaller number of teacher images as compared to the case of learning without using the learned data of the main identifier 211.

[0058] FIG. 8 is an example of a flowchart showing the learning processing procedure by the learning processing apparatus. [Step S21] The main identifier learning unit 121 sets the network (CNN) of the main identifier 211.

[0059] [Step S22] The main identifier learning unit 121 executes learning of the main identifier 211 using the teacher image set 131. As a result, a learning model 111 indicating the main identifier 211 is generated.

[0060] [Step S23] The first sub-identifier learning unit 122 sets the network (CNN) of the first sub-identifier 212 using a part of the data of the generated learning model 111. At this time, based on the data of the learning model 111, the configurations of the input layer and the intermediate layer in the network of the main identifier 211 are used as they are as the configurations of the input layer and the intermediate layer in the network of the first sub-identifier 212. And the weight coefficients between the nodes included in the input layer and the intermediate layer for the main identifier 211 are set as the initial values of the weight coefficients between the nodes included in the input layer and the intermediate layer in the network of the first sub-identifier 212.

[0061] [Step S24] The first sub-identifier learning unit 122 executes learning of the first sub-identifier 212 using the teacher image set 132. As a result, a learning model 112 indicating the first sub-identifier 212 is generated.

[0062] [Step S25] The second sub-identifier learning unit 123 sets the network (CNN) of the second sub-identifier 213 using a part of the data of the generated learning model 111. At this time, based on the data of the learning model 111, the configurations of the input layer and the intermediate layer in the network of the main identifier 211 are directly used as the configurations of the input layer and the intermediate layer in the network of the second sub-identifier 213. And the weight coefficients between the nodes included in the input layer and the intermediate layer for the main identifier 211 are set as the initial values of the weight coefficients between the nodes included in the input layer and the intermediate layer in the network of the second sub-identifier 213.

[0063] [Step S26] The second sub-identifier learning unit 123 executes learning of the second sub-identifier 213 using the teacher image set 133. As a result, a learning model 113 indicating the second sub-identifier 213 is generated.

[0064] FIG. 9 is a diagram showing a configuration example of the processing functions provided in the image identification device. As shown in FIG. 9, the image identification device 200 includes a storage unit 220, a lung field region specifying unit 230, and a shadow identification processing unit 240. Note that the storage unit 220 is realized as a storage area of the storage device included in the image identification device 200. The processing of the lung field region specifying unit 230 and the shadow identification processing unit 240 is realized, for example, by the processor 201 executing a predetermined program.

[0065] The data of the learning models 111 to 113 generated by the learning processing device 100 is stored in the storage unit 220. The data of the learning models 111 to 113 is input from the learning processing device 100 to the image identification device 200 via a network or a portable recording medium and stored in the storage unit 220. Also, the data of the learning model 221 for specifying the lung field region is stored in the storage unit 220.

[0066] The lung field region identifying unit 230 acquires a plurality of slice images obtained by imaging the chest region of the imaging subject using the CT device 300. The lung field region identifying unit 230 extracts a set of slice images (lung field image set) that show a lung field region from the acquired plurality of slice images. Based on the extracted lung field image set, the lung field region identifying unit 230 identifies a lung field region from each slice image in the lung field image set using the learning model 221 for lung field region identification.

[0067] The nodule classification processing unit 240 classifies the nodule type for each image block from each slice image in the lung image set using the nodule classification learning models 111 to 113. The nodule classification processing unit 240 includes a main classification unit 241, a first sub-classification unit 242, a second sub-classification unit 243, and a classification control unit 244.

[0068] The main classification unit 241 reads data from the learning model 111 and executes the processing of the main classifier 211. The first sub-classification unit 242 reads data from the learning model 112 and executes the processing of the first sub-classification unit 212. The second sub-classification unit 243 reads data from the learning model 113 and executes the processing of the second sub-classification unit 213. The classification control unit 244 controls the shadow classification processing using the main classification unit 241, the first sub-classification unit 242, and the second sub-classification unit 243.

[0069] FIG. 10 is an example of a flowchart showing a procedure for a shadow identification process performed by an image identification device. [Step S31] The chest region of the subject is imaged using the CT device 300. The lung field region identification unit 230 acquires a set of slice images (input image set) obtained by imaging, and extracts a set of slice images (lung field image set) that show the lung field region.

[0070] For example, the lung field region specifying unit 230 generates a frontal projection image by the minimum value projection method based on the slice images included in the input image set, and binarizes the generated frontal projection image using a predetermined threshold value. As a result, the projection region of the lung field region appears, so the lung field region specifying unit 230 specifies the slice images indicating the upper end and the lower end of the lung field region from among the input image sets. Among the slice images included in the input image set, those from the specified upper end slice image to the specified lower end slice image are extracted as the slice images in which the lung field region is shown.

[0071] [Step S32] The lung field region specifying unit 230 specifies the lung field region from each slice image in the lung field image set using the learning model 221 for specifying the lung field region based on the extracted lung field image set. This learning model 221 is generated, for example, by deep learning using a large number of slice images as teacher images and teacher labels indicating whether each pixel in the slice image is in the region within the lung field or outside the lung field. In this case, in step S32, by inputting each slice image included in the lung field image set into the discriminator based on such a learning model 221, it is determined for each pixel of each slice image whether it is in the region within the lung field or outside the lung field.

[0072] Next, the lung field region specifying unit 230 divides each slice image included in the lung field image set into image blocks of a certain size (16×16 pixels). Each of the divided image blocks is classified into the region within the lung field and the region outside the lung field by collating with the determination result of the lung field region as described above.

[0073] [Step S33] The discrimination processing loop up to step S40 is executed by the shadow discrimination processing unit 240 with all the image blocks within the lung field among the image blocks of each slice image included in the lung field image set as the discrimination processing target.

[0074] [Step S34] The classification control unit 244 inputs the image block to the main classification unit 241, which then executes a shadow classification process. The main classification unit 241 processes the input image block and classifies the shadow type of the image block as ground-glass opacity, infiltrative opacity, honeycomb lung, emphysema, or other shadow.

[0075] [Step S35] The classification control unit 244 determines whether the classification result by the main classification unit 241 is normal. If the classification result is normal, the process proceeds to step S38. If the classification result is not normal, the process proceeds to step S36.

[0076] [Step S36] The classification control unit 244 determines whether the classification result by the main classification unit 241 is an infiltrative shadow. If the classification result is an infiltrative shadow, the process proceeds to step S39. If the classification result is not an infiltrative shadow, the process proceeds to step S37.

[0077] [Step S37] The classification control unit 244 outputs the classification result obtained by the main classification unit 241 as the final shadow classification result for the image block. [Step S38] The classification control unit 244 inputs the image block to the first sub-classification unit 242, which then executes a shadow classification process. The first sub-classification unit 242 processes the input image block and classifies the shadow type of the image block as either normal or ground-glass opacity. The classification control unit 244 outputs the classification result by the first sub-classification unit 242 as the final shadow classification result for the image block.

[0078] [Step S39] The classification control unit 244 inputs the image block to the second sub-classification unit 243, which then executes a shadow classification process. The second sub-classification unit 243 processes the input image block and classifies the shadow type of the image block as either an infiltrate or normal. The classification control unit 244 outputs the classification result by the second sub-classification unit 243 as the final shadow classification result for the image block.

[0079] When the execution of the identification processing loop for all the image blocks in the lung field is completed in [Step S40], the shadow identification processing ends. Through the above processing, the shadow type of each image block can be accurately identified. In particular, a normal area and a thin reticular shadow can be identified with high precision. Also, a normal area including blood vessels and an infiltration shadow can be identified with high precision.

[0080] Note that in the above second embodiment, the learning processing function of the identifier and the identification processing function by the identifier using the learning result were respectively mounted on different information processing apparatuses, but these may be mounted on the same information processing apparatus. For example, the main identifier learning unit 121, the first sub-identifier learning unit 122, and the second sub-identifier learning unit 123 shown in FIG. 6 may be mounted on the image identification apparatus 200.

[0081] Note that the processing functions of the apparatuses (for example, the image identification apparatuses 1 and 200, the learning processing apparatus 100) shown in the above embodiments can be realized by a computer. In that case, a program describing the processing contents of the functions that each apparatus should have is provided, and by executing that program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical disks, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0082] When distributing the program, for example, portable recording media such as DVDs and CDs on which the program is recorded are sold. Also, the program can be stored in the storage device of a server computer and transferred from the server computer to other computers via a network.

[0083] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. Note that the computer can also directly read the program from the portable recording medium and execute processing according to the program. Further, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

Explanation of Signs

[0084] 1 Image identification device 2 Processing unit 3 Image S1 to S4 Steps

Claims

1. Cause a computer to execute a first identification process for identifying, based on an image of the interior of a human body, which state among three or more states regarding a lesion including a first state and a second state the captured region is in, and when the identification result by the first identification process is the first state, execute a second identification process for identifying, based on the image, whether the region is in the first state or the second state, An image identification program for causing the execution of the process.

2. The image is acquired by photographing a cross-section of a lung field, and the three or more states each indicate a type of shadow, The image identification program according to Claim 1.

3. The first state indicates that the region is normal, and the second state indicates that the region is a ground-glass opacity, The image identification program according to Claim 2.

4. The first state indicates that the region is an infiltration shadow, and the second state indicates that the region is normal, The image identification program according to Claim 2.

5. The first identification process is executed based on a first learning model generated by learning using a neural network, and the second identification process is executed based on a second learning model generated by fine-tuning and re-learning the neural network, The image identification program according to any one of Claims 1 to 4.

6. Cause a computer to execute a first identification process for identifying, based on an image of the interior of a human body, which state among three or more states regarding a lesion including a first state and a second state the captured region is in, and when the identification result by the first identification process is the first state, execute a second identification process for identifying, based on the image, whether the region is in the first state or the second state, An image identification method.

7. Execute a first identification process for identifying, based on an image of the interior of a human body, which state among three or more states regarding a lesion including a first state and a second state the captured region is in, and when the identification result by the first identification process is the first state, execute a second identification process for identifying, based on the image, whether the region is in the first state or the second state, a processing unit An image identification device having the same.

8. Generate a first learning model for executing a first identification process that identifies, by learning using a neural network, which state among three or more states related to a lesion including a first state and a second state the region of an image obtained by photographing the inside of a human body is, and generate a second learning model for executing a second identification process that identifies whether the region is in the first state or the second state by fine-tuning and re-learning the neural network, a learning processing unit; An image identification unit that acquires a photographed image obtained by photographing the inside of a human body to be identified, executes the first identification process that identifies which state among the three or more states including the first state and the second state the region of the photographed image is based on the first learning model, and when the identification result by the first identification process is the first state, executes the second identification process that identifies whether the region of the photographed image is in the first state or the second state based on the second learning model; An information processing system having the above.

Images