Image processing apparatus, image processing method, and image processing program

Abstract

To provide an image processing device, image processing method and image processing program which can accurately perform determination about diagnosis of a lesion of a breast.SOLUTION: An image processing device includes at least one processor. The processor detects a specific structure pattern expressing a lesion candidate structure for a breast in a plurality of projection images obtained with tomosynthesis imaging of the breast or a plurality of tomographic images obtained from the plurality of projection images, generates a composite two-dimensional image by combining the plurality of tomographic images, specifies an important object region where the specific structure pattern exists in the composite two-dimensional image, and performs determination about diagnosis of the lesion on the basis of the composite two-dimensional image and the important object region.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Using a radiation image obtained by irradiating the breast with radiation, a doctor or the like diagnoses a lesion in the breast, and techniques for assisting the diagnosis by a doctor or the like are known. For example, Patent Document 1 discloses a technique for extracting a candidate for disorder in the structure of the mammary gland from a mammogram by CAD (Computer-Aided Diagnosis), and determining whether or not the candidate for disorder in the structure of the mammary gland is a lesion based on a feature amount in a region of interest by a lesion determination means.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, the accuracy of determining whether or not a lesion is present may not be sufficient. For example, in the technique described in Patent Document 1, it may be difficult to determine whether or not a lesion is present when the structures of the mammary glands overlap.

[0005] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of accurately performing a determination related to the diagnosis of a breast lesion.

Means for Solving the Problems

[0006] To achieve the above object, the image processing apparatus according to the first aspect of the present disclosure includes at least one processor. The processor is configured to process a series of a plurality of projection images obtained by tomosynthesis imaging of the breast, or a specific structural pattern representing a lesion candidate structure of the breast in a plurality of tomographic images obtained from the plurality of projection images. Detect and generate a plurality of first mask images in which the positions of specific structural patterns appear synthesize a plurality of tomographic images to generate a synthesized two-dimensional image, From the plurality of first mask images based on the synthesized two-dimensional image and a specific structural pattern existing in a target region of interest in the synthesized two-dimensional image, Identify and generate a second mask image representing the area of focus make a determination regarding the diagnosis of a lesion. The second mask image

[0007] In the image processing apparatus according to the second aspect of the present disclosure, in the image processing apparatus according to the first aspect, the processor focuses on making a determination regarding the diagnosis of a lesion for the target region of interest in the synthesized two-dimensional image more than other regions.

[0008] In the image processing apparatus according to the third aspect of the present disclosure, in the image processing apparatus according to the first or second aspect, the processor extracts the target region of interest from the synthesized two-dimensional image and makes a determination regarding the diagnosis of a lesion for the extracted target region of interest.

[0009] In the image processing apparatus according to the fourth aspect of the present disclosure, in the image processing apparatus according to the third aspect, the processor extracts the target region of interest based on conditions corresponding to the type of the specific structural pattern.

[0010] In the image processing apparatus according to the fifth aspect of the present disclosure, in the image processing apparatus according to any one of the first to fourth aspects, the processor detects the specific structural pattern for each type of the specific structural pattern.

[0011] In the image processing apparatus according to the sixth aspect of the present disclosure, in the image processing apparatus according to any one of the first to fifth aspects, the processor identifies the type of the specific structural pattern and specifies the target region of interest for each identified type.

[0012] The image processing apparatus according to the seventh aspect of the present disclosure is the image processing apparatus according to any one of the first to sixth aspects, wherein the processor identifies the type of a specific structural pattern and makes a determination regarding the diagnosis of a lesion based on the identified type and the area of focus.

[0013] The image processing apparatus according to the eighth aspect of the present disclosure is the image processing apparatus according to the seventh aspect, wherein the processor makes a determination as to whether the lesion is a benign lesion or a malignant lesion as a determination regarding the diagnosis of the lesion.

[0014] The image processing apparatus according to the ninth aspect of the present disclosure is the image processing apparatus according to the seventh aspect, wherein the processor makes a determination as to whether or not there is a lesion as a determination regarding the diagnosis of the lesion.

[0015] The image processing apparatus according to the tenth aspect of the present disclosure is the image processing apparatus according to the seventh aspect, wherein the processor makes a determination as to whether or not the lesion is a malignant lesion as a determination regarding the diagnosis of the lesion.

[0016] The image processing apparatus according to the eleventh aspect of the present disclosure is the image processing apparatus according to the seventh aspect, wherein the processor makes a determination as to whether the lesion is a benign lesion, a malignant lesion, or other than a lesion as a determination regarding the diagnosis of the lesion.

[0017] The image processing apparatus according to the twelfth aspect of the present disclosure is the image processing apparatus according to the seventh aspect, wherein the processor makes a determination regarding the degree of malignancy as a determination regarding the diagnosis of the lesion.

[0018] The image processing apparatus according to the thirteenth aspect of the present disclosure is the image processing apparatus according to any one of the first to twelfth aspects, wherein the processor is a detector that outputs information representing a specific structural pattern as a detection result from a plurality of input projection images or a plurality of tomographic images, and identifies the type of the specific structural pattern using a plurality of detectors provided for each type.

[0019] The image processing apparatus according to the 14th aspect of the present disclosure is the image processing apparatus according to any one of the 1st to 13th aspects. The processor uses a detector generated by machine - learning a geometric structure pattern for a specific structure pattern, a detector generated by machine - learning simulation image data by a mathematical model, or a detector generated by machine - learning a machine - learning model using a radiation image of a breast as learning data to detect the specific structure pattern.

[0020] The image processing apparatus according to the 15th aspect of the present disclosure is the image processing apparatus according to any one of the 1st to 14th aspects. When the size of the breast included in a plurality of tomographic images is different from the size of the breast included in the synthesized two - dimensional image, the processor performs processing to make the sizes equal, and specifies a key target area where a specific structure pattern exists in the synthesized two - dimensional image.

[0021] The image processing apparatus according to the 16th aspect of the present disclosure is the image processing apparatus according to any one of the 1st to 15th aspects. In the process of synthesizing a plurality of tomographic images to generate a synthesized two - dimensional image, a process of specifying a key target area where a specific structure pattern exists in the synthesized two - dimensional image is incorporated.

[0022] Also, to achieve the above object, the image processing method according to the 17th aspect of the present disclosure is a specific structure pattern representing a lesion candidate structure for a breast in a series of a plurality of projection images obtained by tomosynthesis imaging of the breast, or a plurality of tomographic images obtained from the plurality of projection images. Detect and generate a plurality of first mask images in which the positions of specific structural patterns appear synthesize a plurality of tomographic images to generate a synthesized two - dimensional image, From the plurality of first mask images specify a key target area where a specific structure pattern exists in the synthesized two - dimensional image Identify and generate a second mask image representing the area of focus using the synthesized two - dimensional image, The second mask image and based on the above, it is an image processing method in which a computer executes a process of making a determination regarding the diagnosis of a lesion.

[0023] Also, in order to achieve the above object, the image processing program according to the 18th aspect of the present disclosure is a specific structural pattern representing a lesion candidate structure for the breast in a series of a plurality of projection images obtained by breast tomosynthesis imaging or a plurality of tomographic images obtained from the plurality of projection images Detect and generate a plurality of first mask images in which the positions of specific structural patterns appear , synthesizes a plurality of tomographic images to generate a synthesized two-dimensional image, From the plurality of first mask images a key target region where a specific structural pattern exists in the synthesized two-dimensional image Identify and generate a second mask image representing the area of focus , the synthesized two-dimensional image, The second mask image and based on the above, it is for causing a computer to execute a process of making a determination regarding the diagnosis of a lesion .

Advantages of the Invention

[0024] According to the present disclosure, a determination regarding the diagnosis of a breast lesion can be made with high accuracy.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that these embodiments do not limit the present invention.

[0027] First, an example of the overall configuration of the radiation image imaging system according to this embodiment will be described. FIG. 1 shows a configuration diagram representing an example of the overall configuration of the radiation image imaging system 1 according to this embodiment. As shown in FIG. 1, the radiation image imaging system 1 according to this embodiment includes a mammography apparatus 10, a console 12, a PACS (Picture Archiving and Communication Systems) 14, and an image processing apparatus 16. The console 12, the PACS 14, and the image processing apparatus 16 are connected by wired communication or wireless communication via a network 17.

[0028] First, the mammography apparatus 10 according to this embodiment will be described. FIG. 1 shows a side view representing an example of the appearance of the mammography apparatus 10 according to this embodiment. Note that FIG. 1 shows an example of the appearance when the mammography apparatus 10 is viewed from the left side of the subject.

[0029] The mammography apparatus 10 according to this embodiment operates according to the control of the console 12, and is an apparatus that irradiates the breast of a subject with radiation R (for example, X-rays) from a radiation source 29 to capture a radiation image of the breast. Further, the mammography apparatus 10 according to this embodiment has a function of performing normal imaging in which imaging is performed with the radiation source 29 at an irradiation position along the normal direction of the detection surface 20A of the radiation detector 20, and so-called tomosynthesis imaging (details will be described later) in which imaging is performed by moving the radiation source 29 to each of a plurality of irradiation positions.

[0030] As shown in FIG. 1, the mammography apparatus 10 includes an imaging table 24, a base 26, an arm portion 28, and a compression unit 32.

[0031] Inside the imaging table 24, a radiation detector 20 is arranged. As shown in FIG. 2, in the mammography apparatus 10 of the present embodiment, when performing imaging, the breast of the subject U is positioned by the user on the imaging surface 24A of the imaging table 24.

[0032] The radiation detector 20 detects the radiation R that has passed through the breast U, which is the subject. Specifically, the radiation detector 20 detects the radiation R that has entered the breast U of the subject and the inside of the imaging table 24 and reached the detection surface 20A of the radiation detector 20, generates a radiation image based on the detected radiation R, and outputs image data representing the generated radiation image. Hereinafter, a series of operations of irradiating the radiation R from the radiation source 29 and generating a radiation image by the radiation detector 20 may be referred to as "imaging". The type of the radiation detector 20 of the present embodiment is not particularly limited. For example, it may be an indirect conversion type radiation detector that converts the radiation R into light and then converts the converted light into charge, or a direct conversion type radiation detector that directly converts the radiation R into charge.

[0033] The compression plate 30 used for compressing the breast during imaging is attached to a compression unit 32 provided on the imaging table 24, and is moved in a direction approaching or away from the imaging table 24 (hereinafter referred to as the "vertical direction") by a compression plate drive unit (not shown) provided in the compression unit 32. The compression plate 30 compresses the breast of the subject between itself and the imaging table 24 by moving in the vertical direction.

[0034] The arm portion 28 can rotate relative to the base 26 by the shaft portion 27. The shaft portion 27 is fixed to the base 26, and the shaft portion 27 and the arm portion 28 rotate integrally. Gears are respectively provided on the shaft portion 27 and the pressing unit 32 of the imaging table 24. By switching between the meshing state and the non-meshing state of these gears, the pressing unit 32 of the imaging table 24 and the shaft portion 27 are connected and rotate integrally, and the shaft portion 27 can be switched to a state of idling separately from the imaging table 24. Note that the switching of the transmission and non-transmission of power of the shaft portion 27 is not limited to the above gears, and various mechanical elements can be used. The arm portion 28 and the imaging table 24 can rotate relative to the base 26 separately with the shaft portion 27 as the rotation axis.

[0035] When performing tomosynthesis imaging in the mammography apparatus 10, the radiation source 29 is sequentially moved to each of a plurality of irradiation positions with different irradiation angles by the rotation of the arm portion 28. The radiation source 29 has a radiation tube (not shown) that generates radiation R, and the radiation tube is moved to each of the plurality of irradiation positions according to the movement of the radiation source 29. FIG. 2 shows a diagram for explaining an example of tomosynthesis imaging. In FIG. 2, the illustration of the compression plate 38 is omitted. In the present embodiment, as shown in FIG. 2, the radiation source 29 is at irradiation positions 19 with different irradiation angles by a predetermined angle β t (t = 1, 2, ···, the maximum value is 7 in FIG. 2), in other words, it is moved to a position where the irradiation angle of the radiation R with respect to the detection surface 20A of the radiation detector 20 is different. At each irradiation position 19 t , according to the instruction of the console 12, the radiation R is irradiated from the radiation source 29 toward the breast U, and a radiation image is taken by the radiation detector 20. In the radiation image imaging system 1, the radiation source 29 is moved to each of the irradiation positions 19 t , and at each irradiation position 19 tWhen tomosynthesis imaging for taking a radiation image is performed, in the example of FIG. 2, seven radiation images are obtained. In the following, in tomosynthesis imaging, when a radiation image taken at each irradiation position 19 is described separately from other radiation images, it is referred to as a "projection image", and a plurality of projection images taken in one tomosynthesis imaging are referred to as "a series of plurality of projection images". Also, when collectively referring to radiation images regardless of types such as tomographic images and synthesized two-dimensional images described later in addition to projection images, it is simply referred to as a "radiation image". Also, in the following, each irradiation position 19 t For the projection image taken at, etc., the irradiation position 19 t And corresponding images, etc., for each image representing symbol, the irradiation position 19 t The symbol "t" representing is attached and described.

[0036] Note that as shown in FIG. 2, the irradiation angle of the radiation R refers to the angle α formed by the normal line CL of the detection surface 20A of the radiation detector 20 and the radiation axis RC. The radiation axis RC refers to an axis connecting the focal point of the radiation source 29 at each irradiation position 19 and a preset position such as the center of the detection surface 20A. Also, here, the detection surface 20A of the radiation detector 20 is a surface substantially parallel to the imaging surface 24A.

[0037] On the other hand, in the mammography apparatus 10, when normal imaging is performed, the radiation source 29 is at the irradiation position 19 where the irradiation angle α is 0 degrees t (The irradiation position 19 along the normal direction t , in FIG. 2, the irradiation position 194) and remains as it is. The radiation R is irradiated from the radiation source 29 according to the instruction of the console 12, and a radiation image is taken by the radiation detector 20.

[0038] The mammography apparatus 10 and the console 12 are connected by wired communication or wireless communication. The radiation image taken by the radiation detector 20 in the mammography apparatus 10 is output to the console 12 by wired communication or wireless communication via a communication I / F (Interface) unit (not shown).

[0039] As shown in FIG. 1, the console 12 of the present embodiment includes a control unit 40, a storage unit 42, a user I / F unit 44, and a communication I / F unit 46.

[0040] As described above, the control unit 40 of the console 12 has a function of controlling the imaging of breast radiation images by the mammography apparatus 10. Examples of the control unit 40 include a computer system including a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0041] The storage unit 42 has a function of storing information related to the imaging of radiation images and radiation images acquired from the mammography apparatus 10. The storage unit 42 is a non-volatile storage unit, and examples thereof include an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0042] The user I / F unit 44 includes input devices such as various buttons and switches operated by a user such as a technician for the imaging of radiation images, and display devices such as lamps and displays for displaying information related to the imaging and radiation images obtained by the imaging.

[0043] The communication I / F unit 46 communicates various data such as information related to the imaging of radiation images and radiation images with the mammography apparatus 10 by wired communication or wireless communication. Further, the communication I / F unit 46 communicates various data such as radiation images with the PACS 14 and the image processing apparatus 16 by wired communication or wireless communication via the network 17.

[0044] Also, as shown in FIG. 1, the PACS 14 of the present embodiment includes a storage unit 50 that stores a group of radiation images 52 and a communication I / F unit (not shown). The group of radiation images 52 includes radiation images taken by the mammography apparatus 10 acquired from the console 12 via a communication I / F unit (not shown).

[0045] When a doctor or the like (hereinafter simply referred to as "doctor") diagnoses a breast lesion using a radiation image, the image processing apparatus 16 has a function of assisting the doctor's diagnosis by making a determination regarding the diagnosis of the lesion.

[0046] FIG. 3 shows a block diagram illustrating an example of the configuration of the image processing apparatus 16 according to the present embodiment. As shown in FIG. 3, the image processing apparatus 16 according to the present embodiment includes a control unit 60, a storage unit 62, a display unit 70, an operation unit 72, and a communication I / F unit 74. The control unit 60, the storage unit 62, the display unit 70, the operation unit 72, and the communication I / F unit 74 are connected to each other via a bus 79 such as a system bus or a control bus so as to be able to exchange various information.

[0047] The control unit 60 controls the overall operation of the image processing apparatus 16. The control unit 60 includes a CPU 60A, a ROM 60B, and a RAM 60C. Various programs for controlling by the CPU 60A and the like are stored in advance in the ROM 60B. The RAM 60C temporarily stores various data.

[0048] The storage unit 62 is a non-volatile storage unit, and specific examples include an HDD, an SSD, and the like. Various information such as a learning program 63A, an image processing program 63B, learning data 64, a lesion diagnosis model 66, and a structure pattern detector 68, which will all be described in detail later, are stored in the storage unit 62.

[0049] The display unit 70 displays a radiation image and various information. The display unit 70 is not particularly limited and includes various displays and the like. Also, the operation unit 72 is used for a user to input instructions and various information for diagnosing a breast lesion using a radiation image by a doctor. The operation unit 72 is not particularly limited, and examples include various switches, a touch panel, a touch pen, and a mouse. Note that the display unit 70 and the operation unit 72 may be integrated into a touch panel display.

[0050] The communication I / F unit 74 communicates various types of information with the console 12 and the PACS 14 via the network 17 by wireless communication or wired communication.

[0051] Regarding the function of assisting the doctor's diagnosis by making a determination related to the diagnosis of a lesion in the image processing apparatus 16 of the present embodiment, an explanation will be given. First, an outline of the determination related to the diagnosis of a lesion in the image processing apparatus 16 of the embodiment will be described. FIG. 4 shows a schematic diagram for explaining the outline of the flow of determination related to the diagnosis of a lesion in the image processing apparatus 16 of the present embodiment.

[0052] In the image processing apparatus 16 of the present embodiment, a specific structure pattern P representing a lesion candidate structure in each of the plurality of tomographic images 100 is detected. When the breast to be imaged contains a lesion, at least a part of the plurality of tomographic images 100 obtained from a series of projection images obtained by tomosynthesis imaging includes a specific structure pattern P representing a lesion candidate structure. The image processing apparatus 16 detects a specific structure pattern representing a lesion candidate structure from each of the plurality of tomographic images 100 using the structure pattern detector 68.

[0053] As an example, in the present embodiment, as the structure pattern detector 68, a detector using a known CAD (Computer-Aided Diagnosis) algorithm is used. In the algorithm by CAD, the probability (likelihood) that a pixel in the tomographic image 100 represents a specific structure pattern P is derived, and a pixel whose probability is equal to or higher than a predetermined threshold is detected as a specific structure pattern P.

[0054] Further, the detection result by the structure pattern detector 68 of the present embodiment is output as a mask image 102 in which the position of a specific structure pattern P appears. In other words, the mask image 102 obtained as the detection result of the structure pattern detector 68 is an image representing the position of the specific structure pattern P in each tomographic image 100. As an example, the mask image 102 of the present embodiment is a binary image representing a specific structure pattern P as "1" and others as "0". Note that the mask image 102 is obtained for each of the plurality of tomographic images 100. That is, the same number of mask images 102 as the plurality of tomographic images 100 are obtained.

[0055] Also, in the image processing apparatus 16, a synthesized two-dimensional image 106 obtained by synthesizing a plurality of tomographic images 100 is generated. The synthesized two-dimensional image 106 includes the breast M and a specific structure pattern P.

[0056] Furthermore, the image processing apparatus 16 specifies, by a target region specifying unit 86, a target region 104 in which a specific structure pattern P exists in the synthesized two-dimensional image 106 from the mask image 102. The target region 104 is a region in which a specific structure pattern P exists in the synthesized two-dimensional image 106 and in which determination regarding the diagnosis of a lesion is mainly made. The target region specifying unit 86 outputs, as a specification result, a mask image 103 representing the target region 104 in which a specific structure pattern P exists.

[0057] Then, the image processing apparatus 16 makes a determination regarding the diagnosis of a lesion using the lesion diagnosis model 66 based on the synthesized two-dimensional image 106 and the target region 104 specified by the target region specifying unit 86, and obtains a determination result 110 indicating whether the lesion is malignant or benign.

[0058] As an example, in the present embodiment, as the lesion diagnosis model 66, a convolutional neural network (CNN) in which machine learning is performed by deep learning using learning data 64 is used. FIG. 5 shows an example of the lesion diagnosis model 66 of the present embodiment.

[0059] The lesion diagnosis model 66 shown in FIG. 5 includes an input layer 200, an intermediate layer 201, a flattening layer 210, and an output layer 212. An image to be processed (the composite two-dimensional image 106 in this embodiment) is input to the input layer 200. The input layer 200 propagates the information of each pixel (per 1 pixel) of the input image to be processed as it is to the intermediate layer 201. For example, when the size of the image to be processed is 28 pixels × 28 pixels and the data is grayscale, the size of the data propagated from the input layer 200 to the intermediate layer 201 is 28×28×1 = 784.

[0060] The intermediate layer 201 includes a convolutional layer 202 and a convolutional layer 206 that perform convolutional processing (conv), and a pooling layer 204 and a pooling layer 208 that perform pooling processing (pool).

[0061] With reference to FIG. 6, the convolutional processing performed in the convolutional layer 202 and the convolutional layer 206 will be described. As shown in FIG. 6, in the convolutional processing, when the pixel value Ip(x,y) of the target pixel Ip of the input data DI is "e", the pixel values of the surrounding adjacent pixels are "a" to "d", "f" to "i", and the coefficients of the 3×3 filter F are "r" to "z", the pixel value Icp(x,y) of the pixel Icp of the output data DIc, which is the result of the convolutional operation regarding the target pixel Ip, is obtained, for example, according to the following formula (1). Note that the coefficients of this filter F correspond to the weights indicating the strength of the connection between the nodes of the previous and subsequent layers. Icp(x,y)=a×z+b×y+c×x+d×w+e×v+f×u+g×t+h×s+i×r ··· (1)

[0062] In the convolution process, for each pixel, the above-described convolution operation is performed to output the pixel value Icp(x, y) corresponding to each target pixel Ip. In this way, output data DIc having pixel values Icp(x, y) arranged in a two-dimensional manner is output. The output data DIc is output one for each filter F. When a plurality of filters F of different types are used, the output data DIc is output for each filter F. The filter F means a neuron (node) in the convolutional layer, and since the features that can be extracted for each filter F are determined, the number of features that can be extracted from one input data DI in the convolutional layer is equal to the number of filters F.

[0063] Also, in the pooling layer 204 and the pooling layer 208, a pooling process is performed to reduce the original image while retaining features. In other words, in the pooling layer 204 and the pooling layer 208, a pooling process is performed to select a local representative value and reduce the resolution of the input image to reduce the image size. For example, if a pooling process of selecting a representative value from a 2×2 pixel block is performed with a stride of "1", that is, shifted by one pixel at a time, a reduced image reduced to half the size of the input image is output.

[0064] In the present embodiment, as shown in FIG. 5, the above-described convolutional layer 202 and convolutional layer 206, and the pooling layer 204 and pooling layer 208 are arranged in the order of convolutional layer 202, pooling layer 204, convolutional layer 206, and pooling layer 208 from the side closer to the input layer 200.

[0065] As shown in FIG. 5, the convolutional layer 202 applies a 3×3 filter F1 to the input (propagated) image and performs the above-described convolution operation, thereby extracting the features of the input image and outputting an image feature map cmp1 in which pixel values are arranged in a two-dimensional manner. As described above, the number of the image feature maps cmp1 is equal to the number corresponding to the type of the filter F1.

[0066] The pooling layer 204 performs a pooling process of selecting a representative value from a 2×2 pixel block for the image feature map cmp1, thereby outputting a plurality of image feature maps cmp2 with the size of the image feature map cmp1 reduced to 1 / 4 (the vertical and horizontal sizes are 1 / 2).

[0067] Similar to the convolutional layer 202, the convolutional layer 206 applies a 3×3 filter F2 and performs the above-described convolutional operation to output a plurality of image feature maps cmp3 in a two-dimensional pixel value array with the features of the input image feature map cmp2 extracted.

[0068] Similar to the pooling layer 204, the pooling layer 208 performs a pooling process of selecting a representative value from a 2×2 pixel block for the image feature map cmp3, thereby outputting a plurality of image feature maps cmp4 with the size of the image feature map cmp3 reduced to 1 / 4 (the vertical and horizontal sizes are 1 / 2).

[0069] The flattening layer 210 after the intermediate layer 201 rearranges the data in the state where the numerical values of the data remain as the image feature map cmp4. As shown in FIG. 5, the three-dimensional data represented by the plurality of image feature maps cmp4 is rearranged into one-dimensional data. As shown in FIG. 5, the value of each node 211 included in the flattening layer 210 corresponds to the pixel value of each pixel of the plurality of image feature maps cmp4.

[0070] The output layer 212 is a fully connected layer in which all nodes 211 are connected, and includes a node 213A corresponding to a determination that the lesion is malignant and a node 213B corresponding to a determination that the lesion is benign. The output layer 212 uses the softmax function, which is an example of an activation function, to output the probability corresponding to the determination that the lesion corresponding to the node 213A is malignant and the probability corresponding to the determination that the lesion corresponding to the node 213B is benign.

[0071] When the probability of node 213A in the output layer 212 is greater than or equal to the probability of node 213B, the lesion diagnosis model 66 outputs a determination result indicating that the lesion is malignant. On the other hand, when the probability of node 213A in the output layer 212 is lower than the probability of node 213B, the lesion diagnosis model 66 outputs a determination result indicating that the lesion is benign. Note that the lesion diagnosis model 66 may output information representing the probabilities of nodes 213A and 213B as the determination result instead of outputting labels such as "malignant" and "benign" as the determination result.

[0072] The lesion diagnosis model 66 of the present embodiment is generated by the image processing device 16 performing machine learning on a machine learning model using the learning data 64. Referring to FIG. 7, an example of a learning phase for the image processing device 16 to perform machine learning on the lesion diagnosis model 66 will be described.

[0073] The learning data 64 is composed of a set of a synthetic two-dimensional image 107 and correct answer data 111. The correct answer data 111 is information indicating whether the lesion indicated by a specific structural pattern P included in the synthetic two-dimensional image 107 is malignant or benign. In the present embodiment, machine learning of the lesion diagnosis model 66 is performed using the error backpropagation method.

[0074] In the learning phase, the synthetic two-dimensional image 107 of the learning data 64 is input to the lesion diagnosis model 66. Note that in the present embodiment, the synthetic two-dimensional image 107 may be divided into a plurality of batches (images), and the divided batches may be sequentially input to the lesion diagnosis model 66 for learning.

[0075] In addition, the learning data 64 used for the learning of the lesion diagnosis model 66 is not limited to this embodiment. For example, synthetic two-dimensional images 107 including a specific structural pattern P for benign lesions and synthetic two-dimensional images 107 including a specific structural pattern P for malignant lesions may be used. Also, for example, the learning data 64 may use synthetic two-dimensional images 107 including a specific structural pattern P for benign lesions, 107 including a structural pattern of a normal structure (tissue) having a structure approximating the lesion, and synthetic two-dimensional images 107 including a specific structural pattern P for malignant lesions.

[0076] The lesion diagnosis model 66 outputs determination results corresponding to the specific structural pattern P included in the synthetic two-dimensional image 107, specifically, the values of nodes 213A and 213B included in the output layer 212 of the lesion diagnosis model 66.

[0077] When the correct answer data 111 for the synthetic two-dimensional image 107 input to the lesion diagnosis model 66 is "malignant", the value of node 213A should be "1" and the value of node 213B should be "0". Also, when the correct answer data 111 for the synthetic two-dimensional image 107 input to the lesion diagnosis model 66 is "benign", the value of node 213A should be "0" and the value of node 213B should be "1".

[0078] Therefore, an operation of the difference (error) between the values of nodes 213A and 213B output from the lesion diagnosis model 66 and the values that nodes 213A and 213B corresponding to the correct answer data 111 should take is performed. Then, according to the error, using the error propagation method, the weight of each neuron is updated so as to reduce the error from the output layer 212 toward the input layer 200, and the lesion diagnosis model 66 is updated according to the update setting.

[0079] In the learning phase, a series of processes including inputting the synthetic two-dimensional image 107 of the learning data 64 into the lesion diagnosis model 66, outputting the values of the nodes 213A and 213B included in the output layer 212 from the lesion diagnosis model 66, performing an error calculation based on the values of the nodes 213A and 213B and the correct answer data 111, setting the update of the weights, and updating the lesion diagnosis model 66 are repeatedly performed.

[0080] FIG. 8 shows a functional block diagram of an example of the configuration related to the function of generating the lesion diagnosis model 66 in the image processing apparatus 16 of the present embodiment. As shown in FIG. 8, the image processing apparatus 16 includes a learning data acquisition unit 90 and a lesion diagnosis model generation unit 92. As an example, in the image processing apparatus 16 of the present embodiment, the CPU 60A of the control unit 60 executes the learning program 63A stored in the storage unit 62, and the CPU 60A functions as the learning data acquisition unit 90 and the lesion diagnosis model generation unit 92.

[0081] The learning data acquisition unit 90 has a function of acquiring the learning data 64 from the storage unit 62. Although one piece of learning data 64 is illustrated in FIG. 3, actually, a sufficient amount of learning data 64 for learning the lesion diagnosis model 66 is stored in the storage unit 62. Also, the correct answer data 111, in other words, whether the specific structural pattern P included in the synthetic two-dimensional image 107 is malignant or benign, is set based on the judgment by an expert such as a doctor or the result of a biopsy or cytodiagnosis. The learning data acquisition unit 90 outputs the acquired learning data 64 to the lesion diagnosis model generation unit 92.

[0082] The lesion diagnosis model generation unit 92 has a function of generating a lesion diagnosis model 66 that inputs the synthetic two-dimensional image 106, uses the learning data 64 as described above to perform machine learning on a machine learning model, and outputs a determination result as to whether the specific structural pattern P included in the synthetic two-dimensional image 106 is malignant or benign. The lesion diagnosis model generation unit 92 stores the generated lesion diagnosis model 66 in the storage unit 62.

[0083] Next, with reference to FIG. 9, the operation of the image processing apparatus 16 in the learning phase of the present embodiment will be described. By the CPU 60A executing the learning program 63A stored in the storage unit 62, the learning process shown in FIG. 9 is executed.

[0084] In step S100 of FIG. 9, the learning data acquisition unit 90 acquires the learning data 64 from the storage unit 62 as described above.

[0085] In the next step S102, the lesion diagnosis model generation unit 92 performs learning of the lesion diagnosis model 66 using the learning data 64 acquired in step S100 above. As described above, the lesion diagnosis model generation unit 92 repeats a series of processes including inputting the synthetic two-dimensional image 107 included in the learning data 64 into the lesion diagnosis model 66, outputting the values of the nodes 213A and 213B included in the output layer 212 of the lesion diagnosis model 66, calculating the error between the values of the nodes 213A and 213B and the correct data 111, updating the weights, and updating the lesion diagnosis model 66 to perform learning of the lesion diagnosis model 66. The lesion diagnosis model generation unit 92 stores the learned lesion diagnosis model 66 in the storage unit 62. When the process of step S102 ends, the learning process shown in FIG. 9 ends.

[0086] The lesion diagnosis model 66 generated by the learning phase in the image processing apparatus 16 as described above is used as described above in the operation phase when making a determination regarding the diagnosis of a lesion in the image processing apparatus 16. The function of making a determination regarding the diagnosis of a lesion in the image processing apparatus 16 of the present embodiment will be described in detail with reference also to FIG. 4 described above.

[0087] FIG. 10 shows a functional block diagram of an example of a configuration related to a function for making a determination regarding the diagnosis of a lesion in the image processing apparatus 16. As shown in FIG. 10, the image processing apparatus 16 includes a tomographic image generation unit 80, a composite two-dimensional image generation unit 82, a structure pattern detection unit 84, a key target region specification unit 86, a lesion diagnosis determination unit 88, and a display control unit 89. As an example, in the image processing apparatus 16 of the present embodiment, the CPU 60A of the control unit 60 executes an image processing program 63B stored in the storage unit 62, whereby the CPU 60A functions as the tomographic image generation unit 80, the composite two-dimensional image generation unit 82, the structure pattern detection unit 84, the key target region specification unit 86, the lesion diagnosis determination unit 88, and the display control unit 89.

[0088] The tomographic image generation unit 80 has a function of generating a plurality of tomographic images from a series of a plurality of projection images. The tomographic image generation unit 80 acquires a desired series of a plurality of projection images from the console 12 of the mammography apparatus 10 or from the PACS 14 based on an instruction to perform a diagnosis of a lesion. Then, the tomographic image generation unit 80 generates a plurality of tomographic images 100 having different heights from the imaging plane 24A from the acquired series of a plurality of projection images. Note that the method by which the tomographic image generation unit 80 generates the plurality of tomographic images 100 is not particularly limited. For example, the tomographic image generation unit 80 can generate the plurality of tomographic images 100 by reconstructing a series of a plurality of projection images by an inverse projection method such as the FBP (Filter Back Projection) method or the successive approximation reconstruction method. The tomographic image generation unit 80 outputs the generated plurality of tomographic images 100 to the composite two-dimensional image generation unit 82 and the structure pattern detection unit 84.

[0089] As described with reference to FIG. 4, the structure pattern detection unit 84 has a function of detecting a specific structure pattern P from each of the plurality of tomographic images 100 using the structure pattern detector 68. The structure pattern detection unit 84 of the present embodiment sequentially inputs the plurality of tomographic images 100 to the structure pattern detector 68 and outputs a mask image 102 representing the position of the specific structure pattern P in each tomographic image 100.

[0090] As described with reference to FIG. 4, the target area specifying unit 86 specifies a target area 104, which is an area where a specific structural pattern P exists in the composite two-dimensional image 106, based on a plurality of mask images 102. As described above, the mask image 102 is an image representing the position of the specific structural pattern P in each tomographic image 100. Since the height of each tomographic plane, in other words, the distance from the radiation source 37R to the tomographic plane, is different for each tomographic image 100, even for the same specific structural pattern P, the position, size, etc. shown in the tomographic image 100 may be different. Therefore, even for the same specific structural pattern P, depending on the corresponding tomographic image 100, the position, size, etc. of the specific structural pattern P represented by the mask image 102 may be different. The target area specifying unit 86 specifies the target area 104, which is an area where the specific structural pattern P exists in the composite two-dimensional image 106, based on these plurality of mask images 102.

[0091] Note that the method by which the target area specifying unit 86 specifies the target area 104 in the composite two-dimensional image 106 is not particularly limited. For example, in the same way as the composite two-dimensional image generation unit 82 generates the composite two-dimensional image 106 from a plurality of tomographic images 100, the target area specifying unit 86 generates a composite mask image by synthesizing a plurality of mask images 102. Further, the target area specifying unit 86 specifies, as the target area 104, the area including the specific structural pattern P in the generated composite mask image. The specification result by the target area specifying unit 86 is output as a mask image 103 representing the target area 104. In other words, the mask image 103 obtained as the specification result of the target area specifying unit 86 is an image representing the target area 104 in the composite two-dimensional image 106. The mask image 103 is an image of the same size as the composite two-dimensional image 106. As an example, the mask image 103 of the present embodiment is a binary image representing the target area 104 as "1" and other areas as "0". The target area specifying unit 86 outputs the mask image 103 to the lesion diagnosis determination unit 88.

[0092] On the one hand, the composite two-dimensional image generation unit 82 has a function of generating a composite two-dimensional image 106 by synthesizing a plurality of tomographic images 100. Note that there is no particular limitation on the method of generating the composite two-dimensional image 106, and the methods described in Japanese Patent Application Laid-Open No. 2014-128716 or U.S. Patent No. 8,983,156 can be used. For example, the composite two-dimensional image 106 can be generated by synthesizing a plurality of tomographic images 100 by an addition method, an averaging method, a maximum intensity projection method, a minimum intensity projection method, or the like. The composite two-dimensional image generation unit 82 outputs the generated composite two-dimensional image 106 to the lesion diagnosis determination unit 88. Note that the composite two-dimensional image generation unit 82 may generate the composite two-dimensional image 106 using the information of the plurality of mask images 102 that is the detection result of the structure pattern detector 68. Further, the process of specifying the key target region 104 by the key target region specifying unit 86 may be incorporated into the reconstruction process when the composite two-dimensional image generation unit 82 reconstructs a plurality of tomographic images 100 to generate the composite two-dimensional image 106.

[0093] As described with reference to FIG. 4 and the like, the lesion diagnosis determination unit 88 has a function of determining whether a lesion in the breast is malignant or benign by using the lesion diagnosis model 66 for the diagnosis of breast lesions. Further, the lesion diagnosis determination unit 88 of the present embodiment has a function of mainly determining the diagnosis of a lesion for the key target region 104 in the composite two-dimensional image 106 as compared with other regions. In other words, the lesion diagnosis determination unit 88 has a function of focusing on the key target region 104 and making a determination regarding the diagnosis of a lesion for the composite two-dimensional image 106.

[0094] An example in which the lesion diagnosis determination unit 88 focuses on making a determination regarding the diagnosis of a lesion in the key target region 104 will be described with reference to FIG. 11. The lesion diagnosis determination unit 88 makes a determination regarding the diagnosis of a lesion by making the weight of the key target region 104 in the synthesized two-dimensional image 106 larger than the weights of other regions. Specifically, as shown in FIG. 11, the lesion diagnosis determination unit 88 divides the synthesized two-dimensional image 106 into a plurality (15 in FIG. 11) of regions 108, inputs each region 108 into the lesion diagnosis model 66, and outputs, for each region 108, the probability of node 213A as the first difference image 110 from the lesion diagnosis model 66 as the probability that the region 108 is malignant. Thereby, a likelihood map 109 representing the probability of malignancy for the entire synthesized two-dimensional image 106 is generated. The lesion diagnosis determination unit 88 generates a weight map in which the weight of the key target region 104 is made larger than the weights of other regions based on the mask image 103. The lesion diagnosis determination unit 88 makes a determination regarding the diagnosis of a lesion using the above likelihood map and weight map. For example, when the value obtained by multiplying the likelihood represented by the likelihood map 109 by the weight of the weight map exceeds a predetermined threshold for each corresponding region or pixel, the lesion diagnosis determination unit 88 determines that the region (pixel) is malignant. Also, when it is not malignant, the lesion diagnosis determination unit 88 determines that it is benign. The lesion diagnosis determination unit 88 outputs, as the determination result 110, information indicating whether the lesion is malignant or benign to the display control unit 89.

[0095] The display control unit 89 has a function of performing control to cause the display unit 70 to display the information representing the determination result 110 obtained by the lesion diagnosis determination unit 88.

[0096] Next, with reference to FIG. 12, the operation of making a determination regarding the diagnosis of a lesion in the image processing apparatus 16 of the present embodiment will be described. By the CPU 60A executing the image processing program 63B stored in the storage unit 62, the lesion diagnosis determination process shown in FIG. 12 is executed.

[0097] In step S200 of FIG. 12, as described above, the tomographic image generation unit 80 acquires a series of a plurality of projection images from the console 12 or the PACS 14 of the mammography apparatus 10.

[0098] In the next step S202, as described above, the tomographic image generation unit 80 generates a plurality of tomographic images 100 from the series of a plurality of projection images acquired in the above step S200.

[0099] In the next step S204, as described above, the structure pattern detection unit 84 uses the structure pattern detector 68 to detect a specific structure pattern P from each of the plurality of tomographic images 100, and obtains a plurality of mask images 102 as detection results.

[0100] In the next step S206, as described above, the composite two-dimensional image generation unit 82 generates a composite two-dimensional image 106 from the plurality of tomographic images 100 generated in the above step S202.

[0101] In the next step S208, as described above, the key target area identification unit 86 uses the plurality of mask images 102 to identify a key target area 104, which is an area corresponding to the position of the specific structure pattern P in the composite two-dimensional image 106 generated in the above step S206. The key target area identification unit 86 outputs a mask image 103 representing the key target area 104 as an identification result.

[0102] In the next step S210, as described above, the lesion diagnosis determination unit 88 uses the lesion diagnosis model 66 to perform a determination regarding the diagnosis of a lesion. Specifically, the composite two-dimensional image 106 is input to the lesion diagnosis model 66, and the determination result output from the lesion diagnosis model 66 is acquired as a likelihood map 109. As described above, the lesion diagnosis model 66 uses the likelihood map 109 and the mask image 103 representing the key target area 104 to perform a determination regarding the diagnosis of a lesion by focusing on the key target area 104, and outputs a determination result 110.

[0103] In the next step S212, the display control unit 89 performs control to cause the display unit 70 to display the determination result 110 obtained from the determination regarding the diagnosis of the lesion in the above step S210. Note that the display form for causing the display unit 70 to display the determination result 110 is not particularly limited. FIG. 13 shows an example of a form in which the determination result 110 is displayed on the display unit 70 together with the synthesized two-dimensional image 106. In the example shown in FIG. 13, the determination result 110 is shown as a frame representing the key target region 104 including the specific structure pattern P in the synthesized two-dimensional image 106. By varying the color of this frame, the type of line representing the frame, etc. according to the determination result 110, it may also be used as a form for displaying the determination result 110. As a specific example, when the determination result 110 is "malignant", the determination result 110 is displayed as a red line, and when the determination result 110 is "benign", the determination result 110 is displayed as a blue line. Further, it is not limited to the form shown in FIG. 13, and the determination result 110 may be displayed on the display unit 70 in the form of characters, symbols, etc. representing the determination result 110.

[0104] Note that the forms of detecting the specific structure pattern P and making determinations regarding the diagnosis of lesions using the lesion diagnosis model 66 in the learning phase and the operation phase described above are examples, and various modifications are possible. Some of the modifications will be described below.

[0105] (Modification Example 1: Modification Example of the Operation Phase) FIG. 14 shows a schematic diagram for explaining the outline of the determination flow of the lesion diagnosis model 66 in the image processing apparatus 16 of this modification example.

[0106] In each of the above forms, a form has been described in which the entire synthesized two-dimensional image 106 is input to the lesion diagnosis model 66, and the lesion diagnosis model 66 makes a determination regarding the diagnosis of lesions for the entire synthesized two-dimensional image 106. In contrast, in this modification example, as shown in FIG. 14, the key target region 104 is extracted from the synthesized two-dimensional image 106 and cut out as a patch 120, the patch 104 is input to the lesion diagnosis model 66, and the lesion diagnosis model 66 makes a determination regarding the diagnosis of lesions for the patch 120.

[0107] FIG. 15 shows a functional block diagram of an example of a configuration related to a function for making a determination regarding the diagnosis of a lesion in the image processing apparatus 16 of this modified example. As shown in FIG. 15, the image processing apparatus 16 of this modified example is different from the image processing apparatus 16 (see FIG. 10) of each of the above-described embodiments in that it includes a target area extraction unit 87.

[0108] The image processing apparatus 16 causes the CPU 60A of the control unit 60 to execute an image processing program 63B stored in the storage unit 62, whereby the CPU 60A further functions as the target area extraction unit 87.

[0109] The target area extraction unit 87 receives the synthesized two-dimensional image 106 generated by the synthesis two-dimensional image generation unit 82 and the mask image 103 representing the target area 104 specified by the target area specifying unit 86. The target area extraction unit 87 has a function of extracting the target area 104 from the synthesized two-dimensional image 106 based on the mask image 103 representing the target area 104. In other words, the target area extraction unit 87 cuts out a patch 120 including the target area 104 from the synthesized two-dimensional image 106 based on the mask image 103. The target area extraction unit 87 outputs the cut-out patch 120 to the lesion diagnosis determination unit 88.

[0110] The lesion diagnosis determination unit 88 inputs the cut-out target area 104 to the lesion diagnosis model 66 and obtains the output determination result 110.

[0111] FIG. 16 shows a flowchart representing an example of the flow of the lesion diagnosis determination process by the image processing apparatus 16 of this modified example. The lesion diagnosis determination process shown in FIG. 16 is different from the lesion diagnosis determination (see FIG. 12) of each of the above-described embodiments in that it includes step S209 between step S208 and step S210.

[0112] As shown in FIG. 16, as described above, in step S209, the target area extraction unit 87 extracts the target area 104 from the composite two-dimensional image 106 and cuts out the patch 120 based on the mask image 103 representing the target area 104 specified in step S208.

[0113] Thereby, in step S210, as described above, the lesion diagnosis determination unit 88 inputs the patch 120 cut out in step S209 into the lesion diagnosis model 66, obtains the determination result output from the lesion diagnosis model 66, and outputs the determination result 110 in the same manner as the above-described lesion diagnosis determination process (see FIG. 12).

[0114] Thus, in this modified example, instead of the entire composite two-dimensional image 106, the patch 120 including the target area 104 is cut out from the composite two-dimensional image 106, and the cut-out patch 120 is input into the lesion diagnosis model 66 to make a determination regarding the diagnosis of the lesion. Therefore, compared with the case of inputting the entire composite two-dimensional image 106 into the lesion diagnosis model 66 to make a determination regarding the diagnosis of the lesion, the processing amount in the lesion diagnosis model 66 can be reduced, and it is possible to suppress the so-called increase in calculation cost.

[0115] (Modified Example 2: Modified Example of Detection of Specific Structural Pattern) A modified example regarding the detection of a specific structural pattern P from a plurality of tomographic images 100 in the image processing apparatus 16 of this modified example will be described.

[0116] There are multiple types of specific structural patterns P related to lesions. For example, since the shape varies according to the type of lesion, there are types corresponding to the shape of the lesion. Specifically, when the lesion is a tumor, the specific structural pattern P tends to show a spherical shape, and when the lesion is spiculate, the specific structural pattern P tends to show a radial shape. Therefore, in this modified example, a form of detecting the specific structural pattern P according to the type of the specific structural pattern P will be described.

[0117] FIG. 17 shows a schematic diagram for explaining the flow of detecting a specific structure pattern P in the image processing apparatus 16 of this modified example. In the example shown in FIG. 17, a specific structure pattern P1 of a tumor and a specific structure pattern P2 of a spicule are detected. Specifically, the structure pattern detection unit 84 detects, from each of the plurality of tomographic images 100, a specific structure pattern P1 of a tumor by the spherical structure pattern detector 681, and generates a mask image 1021 corresponding to the specific structure pattern P1 of the tumor. Further, the structure pattern detection unit 84 detects, from each of the plurality of tomographic images 100, a specific structure pattern P2 of a spicule by the radial structure pattern detector 682, and generates a mask image 1022 corresponding to the specific structure pattern P2 of the spicule.

[0118] Also, in the example shown in FIG. 17, a mask image 1031 representing a target region of interest 1041 including a spherical specific structure pattern P specified by the target region of interest specifying unit 86 is generated from the mask image 1021. Further, a mask image 1032 representing a target region of interest 1042 specified by the target region of interest specifying unit 86 including a radial specific structure pattern P is generated from the mask image 1022.

[0119] Note that the present invention is not limited to the example shown in FIG. 17. For example, a mask image 102 integrating the mask images 1021 and 1022 may be generated, and the target region of interest specifying unit 86 may specify a target region of interest 104 in which a specific structure pattern P exists in the composite two-dimensional image 106 based on the integrated mask image 102.

[0120] As described above, in this modified example, by using the structure pattern detector 68 corresponding to the type of the specific structure pattern P, the detection accuracy of the specific structure pattern P can be increased, and the type of the specific structure pattern P included in the breast M can be determined.

[0121] In addition, when the type of the specific structural pattern P, that is, the type of the lesion, is specified in this way, the size of the key target area 104 including the specific structural pattern P can be extracted under conditions according to the type, or the key target area 104 can be detected according to the type. For example, depending on the type of the lesion, the influence exerted on the periphery of the lesion may be different. Therefore, based on these conditions, the range to be extracted as the key target area 104 may be varied according to the type of the specific structural pattern P, etc.

[0122] Also, it may be in a form in which a determination regarding the diagnosis of the lesion is made using the lesion diagnosis model 66 for each type of the specific structural pattern P, that is, for each type of the lesion.

[0123] (Modification Example 3: Modification Example of Determination Regarding Lesion Diagnosis) A modification example of the determination regarding the diagnosis of the lesion in the image processing apparatus 16 of this modification example will be described.

[0124] As described above in Modification Example 2, there are a plurality of types of the specific structural pattern P according to the shape of the lesion. Therefore, in this modification example, a form in which a diagnosis regarding the lesion is made using the lesion diagnosis model 66 according to the type of the specific structural pattern P will be described.

[0125] FIG. 18 shows a schematic diagram for explaining an outline of the flow of determination regarding the diagnosis of the lesion in the image processing apparatus 16 of this modification example. In the example shown in FIG. 18, the lesion diagnosis determination unit 88 uses the spherical lesion diagnosis model 661 in which machine learning has been performed for the determination regarding the diagnosis of a spherical lesion such as a tumor, and based on the mask image 1031 representing the key target area 1041 in which the specific spherical structural pattern P1 exists, outputs a determination result 1101 regarding the spherical lesion included in the synthesized two-dimensional image 106. Further, the lesion diagnosis determination unit 88 uses the radial lesion diagnosis model 662 in which machine learning has been performed for the determination regarding the diagnosis of a radial lesion such as a spicule, and based on the mask image 1032 representing the key target area 1042 in which the specific radial structural pattern P2 exists, outputs a determination result 1102 regarding the radial lesion included in the synthesized two-dimensional image 106.

[0126] Thus, in this modified example, in the determination regarding the diagnosis of a lesion, since the determination is made according to the type of the specific structure pattern P, in other words, the type of the lesion, the determination accuracy can be made higher.

[0127] Note that, in this modified example, the form in which the determination result 110 is output for each type of the lesion, that is, each type of the specific structure pattern P has been described. However, the present invention is not limited to this modified example, and it may also be a form in which the determination result 110 obtained by comprehensively diagnosing all the lesions is output.

[0128] (Modified Example 4: Modified Example of Determination Result) In each of the above forms, the form in which the determination result 110 output by the lesion diagnosis determination unit 88 is either "malignant" or "benign" has been described. However, the determination result 110 output by the lesion diagnosis determination unit 88 is not limited thereto. In other words, in each of the above forms, the form in which the determination result output by the lesion diagnosis model 66 is either "malignant" or "benign" has been described. However, the determination result output by the lesion diagnosis model 66 is not limited thereto.

[0129] For example, since the mammary glands overlap with each other and appear to have a radial structure, there may be a case where it is misrecognized as a spicule and detected by the structure pattern detector 68. In such a case, it may be determined as "normal" rather than either "malignant" or "benign". In the lesion diagnosis model 66 in such a case, for example, as shown in FIG. 19, the output layer 212 of the lesion diagnosis model 66 includes a node 213A corresponding to the determination that the lesion is malignant, a node 213B corresponding to the determination that the lesion is benign, and a node 213C corresponding to the determination that the lesion is not (normal). Then, the probabilities of each of the nodes 213A to 213C are derived, and the determination associated with the node having the highest probability may be output as the determination result, or the probabilities of each of the nodes 213A to 213C may be output as the determination result.

[0130] Note that there are no restrictions on how to handle the determination result 110 output by the lesion diagnosis determination unit 88 and the determination result output from the lesion diagnosis model 66. For example, the shape of the lesion or the like may change depending on the degree of malignancy, for example, the progress of cancer. In such a case, the lesion diagnosis determination unit 88 may output a determination result 110 representing the degree of malignancy using the lesion diagnosis model 66 that can obtain a determination result regarding the degree of malignancy in the lesion.

[0131] Also, for example, using the lesion diagnosis model 66 that can obtain a determination result as to whether a specific structure pattern P detected as a lesion candidate structure is a lesion, the lesion diagnosis determination unit 88 may output a determination result 110 representing whether it is a lesion. In this case, for example, the node 213A in the output layer 212 (see FIG. 5) of the lesion diagnosis model 66 may be set as the node corresponding to the determination that it is a lesion, and the node 213B may be set as the node corresponding to the determination that it is not a lesion (normal).

[0132] Also, for example, using the lesion diagnosis model 66 that can obtain a determination result as to whether a specific structure pattern P detected as a lesion candidate structure is a malignant lesion, the lesion diagnosis determination unit 88 may output a determination result 110 representing whether it is a lesion. In this case, for example, the node 213A in the output layer 212 (see FIG. 5) of the lesion diagnosis model 66 may be set as the node corresponding to the determination that it is a malignant lesion, and the node 213B may be set as the node corresponding to the determination that it is not a malignant lesion or not a lesion (normal).

[0133] (Modification Example 5: Specific Modification Example of the Key Target Region) In this modification example, a modification example regarding the specification of the key target region 104 will be described.

[0134] The size of the breast M in the plurality of tomographic images 100 may be different from that of the breast M in the synthesized two-dimensional image 106. As shown in FIG. 20, the size of the breast M in the plurality of tomographic images 100 and the synthesized two-dimensional image 106 is determined according to the positional relationship among the radiation source 29, the breast U, and the detection surface 20A of the radiation detector 20. For example, in the example shown in FIG. 20, the size of the breast M included in each of the plurality of tomographic images 100 is a length L1 corresponding to the distance from end to end of the breast U in a plane parallel to the detection surface 20A of the radiation detector 20. On the other hand, the size of the breast M included in the synthesized two-dimensional image 106 is a length L2 corresponding to the distance between the contact points of the straight line connecting the radiation source 29 and the end of the breast U and the detection surface 20A of the radiation detector 20. Thus, in the example shown in FIG. 20, the size of the breast M included in the synthesized two-dimensional image 106 is larger than that of the breast M included in each of the plurality of tomographic images 100.

[0135] Thus, since the size of the breast M included in each of the plurality of tomographic images 100 is different from the size of the breast M included in the synthesized two-dimensional image 106, correction to make the size of the breast M the same (hereinafter referred to as "size correction") may be performed.

[0136] Therefore, when size correction is performed on at least one of the plurality of tomographic images 100 and the synthesized two-dimensional image 106, the key target area specifying unit 86 specifies the key target area 104 in which the specific structure pattern P exists in the synthesized two-dimensional image 106 by using the plurality of tomographic images 100 and the synthesized two-dimensional image 106 as they are.

[0137] On the other hand, when size correction is not performed, the target area specifying unit 86 corrects the target area 104 where a specific structure pattern P exists in the synthesized two-dimensional image 106 to correspond to the size correction. For example, the target area specifying unit 86 projects the position (coordinates) of the specific structure pattern P (target area 104) in each of the plurality of tomographic images 100 based on the position (coordinates) of the radiation source 29, and derives the position (coordinates) of the specific structure pattern P (target area 104) in the synthesized two-dimensional image 106. Also, for example, the target area specifying unit 86 projects the mask image 102 corresponding to the plurality of tomographic images 100 based on the position (coordinates) of the radiation source 29, and derives the position (coordinates) of the specific structure pattern P (target area 104) in the synthesized two-dimensional image 106. Also, for example, the target area specifying unit 86 projects a multi-value image representing the likelihood corresponding to the plurality of tomographic images 100 based on the position (coordinates) of the radiation source 29, obtains a multi-value image representing the likelihood in the synthesized two-dimensional image 106, and derives the area where the value is equal to or greater than the threshold as the specific structure pattern P (target area 104) in the synthesized two-dimensional image 106.

[0138] (Modification Example 6: Modification Example of the Determination Method in the Lesion Diagnosis Determination Unit 88) A modification example in which the lesion diagnosis determination unit 88 focuses on the determination regarding the diagnosis of the lesion for the target area 104 will be described with reference to FIG. 21. In the example shown in FIG. 21, for the lesion diagnosis model 66 using the sliding window method, a modification example is shown in which the sliding method of the window 122 is changed to focus on the determination regarding the diagnosis of the lesion for the target area 104.

[0139] As shown in FIG. 21, the lesion diagnosis determination unit 88 applies the lesion diagnosis model 66 with the slide width of the window 122 that scans (slides) the synthesized two-dimensional image 106 being smaller in the target area 104 than in other areas. In this way, by reducing the slide width, the number of determinations in the target area 104 can be made larger than the number of determinations in other areas, and the determination regarding the diagnosis of the lesion for the target area 104 can be focused.

[0140] Note that the present modification is not limited thereto. For example, regarding the window 122 in the lesion diagnosis model 66 by the lesion diagnosis determination unit 88, the window 122 in the key target area 104 may be made smaller than the window 122 in other areas, so that the number of determination times in the key target area 104 may be made larger than the number of determination times in other areas.

[0141] In each of the above embodiments, the form of detecting a specific structure pattern P from a plurality of tomographic images 100 has been described. However, the image to be the detection target of the specific structure pattern P is not limited to the plurality of tomographic images 100, and may be a plurality of series of projection images used to obtain the plurality of tomographic images 100.

[0142] As described above, the image processing apparatus 16 of each of the above embodiments includes a CPU 60A. The CPU 60A detects a specific structure pattern P representing a lesion candidate structure for the breast U in a plurality of series of projection images obtained by tomosynthesis imaging of the breast U or in a plurality of tomographic images 100 obtained from the plurality of projection images. Further, the CPU 60A synthesizes a plurality of tomographic images 100 to generate a synthesized two-dimensional image 106, specifies a key target area 104 in which the specific structure pattern P exists in the synthesized two-dimensional image 106, and makes a determination regarding the diagnosis of a lesion based on the synthesized two-dimensional image 106 and the key target area 104.

[0143] In the synthesized two-dimensional image 106, tissues such as mammary glands may overlap or be hidden in the shadow of normal tissues, so that it may be difficult to detect the specific structure pattern P. On the other hand, in each of the above embodiments, since the structure pattern P is detected from a plurality of series of projection images or a plurality of tomographic images 100, it becomes easier to detect the specific structure pattern P. Further, from the synthesized two-dimensional image 106, only a determination regarding the diagnosis of the lesion is made for the specific structure pattern P detected as a lesion. Therefore, it is possible to specialize in the determination regarding the diagnosis of a lesion. Therefore, according to the image processing apparatus 16 of each of the above embodiments, it is possible to accurately make a determination regarding the diagnosis of a breast lesion.

[0144] Note that the method for detecting the specific structure pattern P is not limited to the method of applying the CAD algorithm based on the probability indicating that it is the specific structure pattern P described above. For example, filtering processing by a filter for detecting the specific structure pattern P, a detection model obtained by machine learning such as deep learning to detect the specific structure pattern P, etc., may be used to detect the specific structure pattern P from a plurality of tomographic images 100. Further, as the structure pattern detector 68, a learned model learned by machine learning may be used. As a model applied as such a structure pattern detector 68, a Classification method using a convolutional neural network (CNN) such as ResNet is adapted in a sliding window method. Alternatively, a wide range of machine learning models such as a Segmentation method such as U-Net or an Object-Detection method such as Faster-RCNN are assumed. In addition, an MLP (MultiLayer Perceptron) or the like may be applied. Further, a structure pattern detector 68 generated by subjecting a geometric structure pattern to machine learning with respect to a machine learning model, a structure pattern detector 68 generated by subjecting simulation image data based on a mathematical model to machine learning, or a structure pattern detector 68 generated by subjecting a radiation image obtained by photographing a breast to machine learning using it as learning data can be applied.

[0145] Also, in the above-described embodiment, the case where the determination regarding the diagnosis of the lesion is made by applying the lesion diagnosis model 66 to the synthesized two-dimensional image 106 regardless of the detection result of the structure pattern detection unit 84 has been described. However, depending on the detection result of the structure pattern detection unit 84, whether or not to make the determination regarding the diagnosis of the lesion by applying the lesion diagnosis model 66 may be varied. For example, when a specific structure pattern P of a lesion that can be surely said to be malignant or a lesion that can be surely said to be benign is detected by the structure pattern detector 68, the diagnosis of the lesion by applying the lesion diagnosis model 66 may not be performed, and the determination regarding the diagnosis of the lesion may be made based on the detection result of the structure pattern detector 68.

[0146] Also, as described above, the lesion diagnosis model 66 may apply, for example, an MLP (MultiLayer Perceptron) or the like in addition to the Classification method using a convolutional neural network (CNN) such as ResNet.

[0147] In addition, in the above-described embodiment, the form in which the image processing apparatus 16 performs learning of the lesion diagnosis model 66 and determination regarding diagnosis of a lesion using the lesion diagnosis model 66 has been described. However, a form in which learning of the lesion diagnosis model 66 is performed by a learning apparatus other than the image processing apparatus 16 may be employed. That is, the apparatus that performs learning of the lesion diagnosis model 66 and the apparatus that makes a determination regarding diagnosis of a lesion using the lesion diagnosis model 66 may be different apparatuses. FIG. 22 shows a configuration diagram schematically showing an example of the overall configuration of the radiation image imaging system 1 when learning of the lesion diagnosis model 66 is performed by a learning apparatus 18 other than the image processing apparatus 16. The radiation image imaging system 1 shown in FIG. 22 further includes a learning apparatus 18. The learning apparatus 18 includes a computer system including a control unit 18A and a storage unit 18B such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The storage unit 18B stores the learning program 63A, the learning data 64, and the lesion diagnosis model 66 that were provided in the storage unit 62 of the image processing apparatus 16 in the above-described embodiments. The learning apparatus 18 generates the lesion diagnosis model 66 by performing learning using the learning data 64 when the control unit 18A executes the learning program 63A. The lesion diagnosis model 66 generated by the learning apparatus 18 is transmitted to the image processing apparatus 16 and stored in the storage unit 62 of the image processing apparatus 16. In this case, unlike the image processing apparatus 16 (see FIG. 3) in the above-described embodiments, the learning program 63A and the learning data 64 do not have to be stored in the storage unit 62 of the image processing apparatus 16.

[0148] Also, in the above-described embodiment, for example, as the hardware structure of the processing unit that executes various processes such as the tomographic image generation unit 80, the synthesized two-dimensional image generation unit 82, the structural pattern detection unit 84, the key target area specification unit 86, the lesion diagnosis determination unit 88, and the display control unit 89, and the hardware structure of the processing unit that executes various processes such as the learning data acquisition unit 90 and the lesion diagnosis model generation unit 92, the following various processors can be used. As described above, in addition to the CPU, which is a general-purpose processor that executes software (program) and functions as various processing units, the above various processors include a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing, and a dedicated electric circuit, which is a processor having a circuit configuration specifically designed to execute specific processes, such as an ASIC (Application Specific Integrated Circuit).

[0149] One processing unit may be configured by one of these various processors, or may be configured by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, a plurality of processing units may be configured by one processor.

[0150] As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as clients and servers, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Second, as represented by a System On Chip (SoC), etc., there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with a single integrated circuit (IC) chip is used. Thus, various processing units are configured using one or more of the above various processors as a hardware structure.

[0151] Furthermore, as a hardware structure of these various processors, more specifically, circuitry combining circuit elements such as semiconductor elements can be used.

[0152] Also, in the above-described forms, the mode in which the learning program 63A and the image processing program 63B are pre-stored (installed) in the storage unit 62 has been described, but it is not limited to this. Each of the learning program 63A and the image processing program 63B may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a USB (Universal Serial Bus) memory. Also, each of the learning program 63A and the image processing program 63B may be in a form downloaded from an external device via a network.

Explanation of Reference Numerals

[0153] 1 Radiographic imaging system 10 Mammography device 12 Console 14 PACS 16 Image processing device 17 Network 18 Learning device, 18A Control unit, 18B Storage unit 191 to 197, 19 t Irradiation position 20 Radiation detector, 20A Detection surface 24 Photography table, 24A Photography surface 26 Base 27 Shaft portion 28 Arm portion 29 Radiation source 30 Compression plate 32 Compression unit 40, 60 Control unit 42, 50, 62 Memory unit 44 User I / F unit 46, 74 Communication I / F unit 52 Radiation image group 60A CPU, 60B ROM, 60C RAM 63A Learning program, 63B Image processing program 64 Learning data 66 Lesion diagnosis model 68 Structure pattern detector, 681 Spherical structure pattern detector, 682 Radial structure pattern detector 70 Display unit 72 Operation unit 79 Bus 80 Tomographic image generation unit 82 Synthetic two-dimensional image generation unit 84 Structure pattern detection unit 86 Key target area specification unit 87 Key target area extraction unit 88 Lesion diagnosis determination unit 89 Display control unit 90 Learning data acquisition unit 92 Lesion diagnosis model generation unit 100 Tomographic image 102, 1021, 1022, 103 Mask image 104 Key target area 106, 107 Synthetic two-dimensional image 108 Area 109 Likelihood map 110, 1101, 1102 Judgment result 111 Correct data 120 Patch 122 Window 200 Input layer 201 Intermediate layer 202, 206 Convolution layer 204, 208 Pooling layer 210 Flatten layer 211, 213A, 213B, 213C Node 212 Output layer cmp1~cmp4 Image feature map F, F1, F2 Filter Ip Pixel of interest DI Input data, DIc Output data L1, L2 Length M, U Breast P, P1, P2 Structural pattern R Radiation, RC Radiation axis α, β Angle

Claims

1. Comprising at least one processor, wherein the processor, detects a specific structural pattern representing a lesion candidate structure of the breast in a series of a plurality of projection images obtained by breast tomosynthesis imaging or in a plurality of tomographic images obtained from the plurality of projection images, and generates a plurality of first mask images in which the positions of the specific structural pattern appear, synthesizes the plurality of tomographic images to generate a synthesized two-dimensional image, identifies a key target region in the synthesized two-dimensional image where the specific structural pattern exists from the plurality of first mask images, and generates a second mask image representing the key target region, makes a determination regarding the diagnosis of a lesion based on the synthesized two-dimensional image and the second mask image An image processing apparatus.

2. wherein the processor, makes a determination regarding the diagnosis of the lesion more intensively for the key target region in the synthesized two-dimensional image than for other regions, The image processing apparatus according to Claim 1.

3. wherein the processor, extracts the key target region from the synthesized two-dimensional image, and makes a determination regarding the diagnosis of the lesion for the extracted key target region The image processing apparatus according to Claim 1 or Claim 2.

4. wherein the processor, extracts the key target region based on conditions corresponding to the type of the specific structural pattern, The image processing apparatus according to Claim 3.

5. wherein the processor, detects the specific structural pattern for each type of the specific structural pattern, The image processing apparatus according to any one of Claims 1 to 4.

6. wherein the processor, identifies the type of the specific structural pattern, and identifies the key target region for each identified type The image processing apparatus according to any one of Claims 1 to 5.

7. wherein the processor, identifies the type of the specific structural pattern, and makes a determination regarding the diagnosis of the lesion based on the identified type and the key target region The image processing apparatus according to any one of Claims 1 to 6.

8. wherein the processor, as a determination regarding the diagnosis of the lesion, determines whether it is a benign lesion or a malignant lesion The image processing apparatus according to Claim 7.

9. wherein the processor, as a determination regarding the diagnosis of the lesion, determines whether there is a lesion The image processing apparatus according to Claim 7.

10. wherein the processor, As a determination regarding the diagnosis of the lesion, determine whether it is a malignant lesion. The image processing apparatus according to claim 7.

11. The processor As a determination regarding the diagnosis of the lesion, determine whether it is a benign lesion, a malignant lesion, or other than a lesion. The image processing apparatus according to claim 7.

12. The processor As a determination regarding the diagnosis of the lesion, determine the degree of malignancy. The image processing apparatus according to claim 7.

13. The processor A detector that outputs, as a detection result, information representing the specific structural pattern from the input plurality of projection images or the plurality of tomographic images, and identifies the type of the specific structural pattern using a plurality of detectors provided for each type of the specific structural pattern. The image processing apparatus according to any one of claims 1 to 12.

14. The processor For the specific structural pattern, using a detector generated by machine-learning a geometric structural pattern against a machine learning model, a detector generated by machine-learning simulation image data by a mathematical model, or a detector generated by machine-learning a machine learning model using a radiation image of a breast as learning data, detect the specific structural pattern. The image processing apparatus according to any one of claims 1 to 13.

15. The processor When the size of the breast included in the plurality of tomographic images is different from the size of the breast included in the synthesized two-dimensional image, perform processing to make the sizes equal, and identify a key target region where the specific structural pattern exists in the synthesized two-dimensional image. The image processing apparatus according to any one of claims 1 to 14.

16. In the process of synthesizing the plurality of tomographic images to generate a synthesized two-dimensional image, a process of identifying a key target region where the specific structural pattern exists in the synthesized two-dimensional image is incorporated. The image processing apparatus according to any one of claims 1 to 15.

17. Detect a specific structural pattern representing a lesion candidate structure of the breast in a series of a plurality of projection images obtained by tomosynthesis imaging of the breast, or a plurality of tomographic images obtained from the plurality of projection images, and generate a plurality of first mask images in which the positions of the specific structural pattern appear. synthesize the plurality of tomographic images to generate a synthesized two-dimensional image; identify a target region of interest in the synthesized two-dimensional image where the specific structural pattern exists, from the plurality of first mask images, and generate a second mask image representing the target region of interest; make a determination regarding the diagnosis of a lesion based on the synthesized two-dimensional image and the second mask image; An image processing method executed by a computer.

18. detect a specific structural pattern representing a lesion candidate structure of the breast in a series of a plurality of projection images obtained by tomosynthesis imaging of the breast, or in a plurality of tomographic images obtained from the plurality of projection images, and generate a plurality of first mask images in which the positions of the specific structural pattern appear; synthesize the plurality of tomographic images to generate a synthesized two-dimensional image; identify a target region of interest in the synthesized two-dimensional image where the specific structural pattern exists, from the plurality of first mask images, and generate a second mask image representing the target region of interest; make a determination regarding the diagnosis of a lesion based on the synthesized two-dimensional image and the second mask image; An image processing program for causing a computer to execute the processing.

Images