Image processing device, image processing method, and storage medium

Abstract

In this image processing device, a mammary gland structure extraction means extracts a mammary gland structure from a chest image. A diagnostically difficult site detection means detects, from information of the chest image and the mammary gland structure, a diagnostically difficult site, which is a site where the shape of a mammary gland is complicated and therefore is difficult to diagnose. In lesion detection and identification, the location of a lesion is inferred from information of the chest image, the mammary gland structure, and the diagnostically difficult site. A result display means displays the chest image and the location of the lesion. The image processing device can be used, for example, to assist decision making in diagnosis.

Description

Image processing device, image processing method, and recording medium

[0001] The present disclosure relates to techniques for processing images.

[0002] Image diagnosis using mammography and ultrasound is important for the early detection and diagnosis of breast cancer. Patent Document 1 describes a system that prevents doctors from overlooking abnormalities and improves the efficiency of image interpretation by providing only detection information for abnormal shadow candidates in medical images.

[0003] Japanese Patent Application Laid-Open No. 2006-340835

[0004] When doctors diagnose breast images taken by mammography, they may overlook lesions or make a mistake in diagnosis, for example, when breast density is high.

[0005] One of the objects of the present disclosure is to provide an image processing device that can accurately infer information about lesions from chest images.

[0006] In order to solve the above problems, in one aspect of the present invention, an image processing device comprises: a mammary gland structure extraction means for extracting mammary gland structure from a chest image; a difficult-to-diagnose position detection means for detecting, from information on the chest image and the mammary gland structure, a difficult-to-diagnose position where the shape of the mammary gland is complex and diagnosis is difficult; a lesion detection and identification means for inferring the position of a lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose position; and a result display means for displaying the chest image and the position of the lesion.

[0007] In another aspect of the present invention, an image processing method includes: extracting a mammary gland structure from a chest image; detecting a difficult-to-diagnose location where the mammary gland has a complex shape and is difficult to diagnose from information on the chest image and the mammary gland structure; performing an inference process to infer the location of a lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose location; and displaying the chest image and the location of the lesion.

[0008] In yet another aspect of the present invention, a recording medium records a program that causes a computer to execute the following processes: extracting mammary gland structure from a chest image; detecting difficult-to-diagnose locations where the mammary gland has a complex shape and is difficult to diagnose from information about the chest image and the mammary gland structure; performing an inference process to infer the location of a lesion from information about the chest image, the mammary gland structure, and the difficult-to-diagnose locations; and displaying the chest image and the location of the lesion.

[0009] According to the present disclosure, an image processing device can accurately infer information about lesions from chest images.

[0010] 1 shows an example of a schematic configuration of an image processing system. 2 shows an example of a hardware configuration of an image processing device. 3 is a block diagram showing an example of a functional configuration of an image processing device. 4 is an example of a chest image. 5 is an example of an image showing mammary gland tissue. 6 is a diagram explaining the type and progression of a lesion. 7 is an example of a display of a position that is difficult to diagnose on a mammary gland structure. 8 is an example of a display of the position, type, and progression of a lesion on a mammary gland structure. 9 is an example of a diagnostic assistance screen. 10 is a flowchart of lesion inference processing.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. [Embodiment] (Configuration) Fig. 1 shows an example of a schematic configuration of an image processing system 100 to which an image processing device of the present disclosure is applied. The image processing system 100 infers information about lesions from images of breasts (hereinafter also referred to as "chest images") taken with an X-ray device and displays the inference results, thereby assisting doctors in diagnosing chest images and preventing lesions from being overlooked. In other words, the image processing system 100 can support medical professionals such as doctors in making diagnostic decisions.

[0012] In the image processing system 100 of the present disclosure, a mammography device 10 and an image processing device 20 are communicatively connected via a network 5 such as the Internet or a local area network (LAN). The mammography device 10 is an X-ray device dedicated to breast imaging, and the captured chest images are stored in a predetermined database (DB). In the present disclosure, chest images of a subject captured by the mammography device 10 are stored in a chest image DB 27 connected to the image processing device 20 via the network 5. The subject here refers to, for example, a patient receiving medical treatment at a medical institution or a person undergoing breast cancer screening.

[0013] The image processing device 20 is a personal computer (PC), tablet, or the like that displays chest images captured by the mammography device 10 and acquires chest images from the chest image DB 27. The image processing device 20 is an information processing device that processes, stores, and transmits / receives various data, and extracts mammary gland structures from the chest images. Furthermore, the image processing device 20 detects, from the information on the chest images and mammary gland structures, locations where the mammary gland shape is complex, visibility is low, and diagnosis is difficult (hereinafter also referred to as "difficult-to-diagnose locations"). The image processing device 20 then infers information about the lesion, such as its location, type, and progression, from the information on the chest images, mammary gland structures, and difficult-to-diagnose locations, and displays the inference results together with the chest images, mammary gland structures, and difficult-to-diagnose locations.

[0014] 2 is a block diagram showing an example of the hardware configuration of the image processing device 20. As shown in the figure, the image processing device 20 includes an interface 21, a processor 22, a memory 23, a recording medium 24, a display unit 25, and an input unit 26. These components and a chest image DB 27 are interconnected via a bus.

[0015] The interface 21 exchanges data with the mammography device 10. The interface 21 is used when receiving chest images from the mammography device 10. The interface 21 is also used when the image processing device 20 exchanges data with a specific device connected by wire or wirelessly.

[0016] The processor 22 is a computer such as a CPU (Central Processing Unit), and executes a program prepared in advance to control the entire image processing device 20. The processor 22 may be a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0017] The memory 23 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 23 stores programs executed by the processor 22. The memory 23 is also used as a working memory while the processor 22 is executing various processes.

[0018] The recording medium 24 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the image processing device 20. The recording medium 24 records various programs executed by the processor 22. When the image processing device 20 executes the lesion inference process, the programs recorded on the recording medium 24 are loaded into the memory 23 and executed by the processor 22.

[0019] The display unit 25 displays a predetermined image on, for example, an LCD (Liquid Crystal Display), etc. The input unit 26 is a keyboard, mouse, touch panel, etc., and is used by users such as doctors and nurses when performing predetermined operations.

[0020] The chest image DB 27 stores chest images of subjects acquired from the mammography device 10 in association with the subject's identification information.

[0021] 3 is a block diagram showing an example of the functional configuration of the image processing device 20. Functionally, the image processing device 20 includes a chest image acquisition unit 31, a mammary gland structure extraction unit 32, a difficult-to-diagnose position detection unit 33, a lesion detection and identification unit 34, and a result display unit 35. The chest image acquisition unit 31, the mammary gland structure extraction unit 32, the difficult-to-diagnose position detection unit 33, the lesion detection and identification unit 34, and the result display unit 35 are realized by the processor 22 executing a program.

[0022] The chest image acquisition unit 31 acquires chest images of a subject to be diagnosed by a doctor from the chest image DB 27. Fig. 4 shows an example of a chest image. Here, the chest image of a subject to be diagnosed by a doctor is a chest image from which the image processing device 20 infers information about a lesion, such as the location, type, and progression of the lesion, and is also referred to as an original image in the present disclosure.

[0023] The mammary gland structure extraction unit 32 extracts the mammary gland structure from the original image using an algorithm for extracting the mammary gland structure from a chest image. FIG. 5 shows an example of a mammary gland structure. The mammary gland structure is the structure of mammary gland tissue, and in the present disclosure, it is an image in which the mammary gland tissue is shown in white, as shown in FIG. 5. Note that the algorithm for extracting the mammary gland structure from a chest image can be, for example, the method described in the following paper: Karla K Evans, Tamara Miner Haygood, Julie Cooper, Anne-Marie Culpan, Jeremy M Wolfe, "A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast", https: / / doi.org / 10.1073 / pnas.1606187113

[0024] The difficult-to-diagnose position detection unit 33 detects difficult-to-diagnose positions from the original image and information on the mammary gland structure extracted from the original image. A difficult-to-diagnose position is a position where the shape of the mammary gland is complex and visibility is low, making diagnosis difficult, such as an area with high mammary gland density or an area that can be confused with other normal areas. The difficult-to-diagnose position is, for example, an image showing mammary gland tissue or coordinate information indicating the difficult-to-diagnose position on a chest image.

[0025] Specifically, the difficult-to-diagnose position detection unit 33 uses a machine learning model to detect difficult-to-diagnose positions from the original image and information on the mammary gland structure extracted from the original image. Specifically, a machine learning model is generated in advance when a chest image and information on the mammary gland structure extracted from the chest image are input, and the machine learning model outputs information indicating areas in the mammary gland structure of the chest image where the mammary gland has a complex shape and is low in visibility as difficult-to-diagnose positions, and the generated model is used in the difficult-to-diagnose position detection unit 33. In this way, a machine learning model that outputs optimized difficult-to-diagnose positions when a chest image and information on the mammary gland structure extracted from the chest image are input is called a "difficult-to-diagnose position output model." This makes it possible to detect optimized difficult-to-diagnose positions.

[0026] Training data is used to generate the difficult-to-diagnose location output model. The training data is data that associates input data input in training the difficult-to-diagnose location output model with correct answer data corresponding to the input data. Specifically, the input data is a chest image and information on mammary gland structure extracted from the chest image, and the correct answer data is the difficult-to-diagnose location in the chest image. The correct answer data is created based on information that can be read from the mammary gland structure, such as areas of high mammary gland density, as well as the opinions of the doctor who diagnosed the chest image. The difficult-to-diagnose location output model is trained to output the difficult-to-diagnose location based on the chest image input as input data and the information on the mammary gland structure extracted from the chest image. An example of a machine learning model is a model that uses a neural network.

[0027] The difficult-to-diagnose position output model is stored in a predetermined DB or memory 23, and the difficult-to-diagnose position detection unit 33 can use the difficult-to-diagnose position output model by reading it from the DB or memory 23. The difficult-to-diagnose position output model can be updated by learning it using new training data.

[0028] The lesion detection and classification unit 34 detects and classifies lesions by inferring information about the lesion, such as its location, type, and progression, from the original image, the mammary gland structure extracted from the original image, and information about the difficult-to-diagnose location. A lesion is a change in a living body that occurs during the course of a disease, and the location of a lesion is the location of the change in the living body. For example, the location of a lesion may be an image showing mammary gland tissue or coordinate information indicating the location of the lesion on a chest image.

[0029] Specifically, the lesion detection and classification unit 34 uses a machine learning model to infer the location, type, and progression of a lesion from the original image, the mammary gland structure extracted from the original image, and information on the difficult-to-diagnose location, thereby detecting and classifying the lesion. Specifically, a machine learning model is generated in advance that outputs information indicating the location, type, and progression of a lesion when a chest image, the mammary gland structure extracted from the chest image, and information on the difficult-to-diagnose location are input. In this way, a machine learning model that outputs an optimized location, type, and progression of a lesion when a chest image, the mammary gland structure extracted from the chest image, and information on the difficult-to-diagnose location are input is called a "lesion output model." This allows the optimized location, type, and progression of a lesion to be inferred, detected, and classified.

[0030] Training data is used to generate the lesion output model. The training data is data that associates input data input in the learning of the lesion output model with correct answer data corresponding to the input data. Specifically, the input data is a chest image, a mammary gland structure extracted from the chest image, and information on difficult-to-diagnose locations in the chest image, while the correct answer data is the location, type, and progression of a lesion detected and differentiated from the chest image. The correct answer data is created based on the diagnosis results of the chest image by a doctor, etc. The lesion output model is trained to output the location, type, and progression of a lesion based on the chest image input as input data, the mammary gland structure extracted from the chest image, and information on difficult-to-diagnose locations in the chest image. An example of a machine learning model is a model that uses a neural network.

[0031] The lesion output model is stored in a predetermined DB or memory 23, and the lesion detection and classification unit 34 can read and use the lesion output model from the predetermined DB or memory 23. The lesion output model can be updated by learning it using new training data.

[0032] Figure 6 is a diagram illustrating the types and progression of lesions. As shown in Figure 6, in this disclosure, the types of lesions include whether the lesion is a mass or a calcification. A mass is a lump that forms inside the breast and is some kind of mass that is different from normal tissue. Calcification is a state in which calcium is deposited inside the mammary gland.

[0033] The type of lesion can be categorized by whether the shape of the lesion is circumscribed, micro-lobulated, indistinctive, obscured, or spiculated. A circumscribed lesion has a clear boundary and a smooth edge. A micro-lobulated lesion has a clearly defined boundary, but upon closer inspection, it is found to be finely irregular and fuzzy. An indistinct boundary is a lesion where the boundary cannot be traced due to infiltration or progression into the surrounding area. A spiculated lesion is a lesion where the edge is hidden because part of the lesion overlaps with or is adjacent to normal breast tissue. A spiculated lesion is a lesion with sharp spicules.

[0034] The progression of the lesion is expressed as a five-stage scale from progression stage 1 (SMC-1) to progression stage 5 (SMC-5), as shown in Figure 6, where progression stage 1 is the stage where the disease has spread the least and progression stage 5 is the stage where the disease has spread the most.

[0035] The result display unit 35 displays on the display unit 25 the chest image, the mammary gland structure, the difficult-to-diagnose location, and the location, type, and progression of the lesion detected and identified by inference by the lesion detection and identification unit 34. The result display unit 35 also acquires information on the selection of the threshold value used to extract the mammary gland structure and the rectangular display mode, as will be described in detail later.

[0036] 7 shows an example of displaying a difficult-to-diagnose position on a mammary gland structure. As shown in FIG. 7, the result display unit 35 displays a rectangle with a white line surrounding the difficult-to-diagnose position on an image showing mammary gland tissue, based on the coordinate information detected by the difficult-to-diagnose position detection unit 33. By displaying a rectangle surrounding the difficult-to-diagnose position in this way, it is possible to visualize and present to the user areas where the mammary gland structure is complex and difficult to diagnose.

[0037] 8 shows an example of displaying the location, type, and progression of a lesion on a mammary gland structure. As shown in FIG. 8, the result display unit 35 displays a thick white rectangle surrounding the lesion on an image showing the mammary gland tissue, based on the coordinate information inferred by the lesion detection and identification unit 34. By displaying a rectangle surrounding the location of the lesion in this way, the location of the lesion detected by the image processing device 20 can be visualized and presented to the user.

[0038] It is assumed that the lesion indicated by rectangle 41 in FIG. 8 is inferred by the lesion detection and classification unit 34 to be calcified and unclearly bordered, and to have a stage of progression of 4. In this case, the result display unit 35 displays "Calc," indicating calcification, "Indistinct," indicating unclearly bordered, and "SMC-4," indicating the stage of progression, near the rectangle 41, as shown in FIG. 8. It is also assumed that the lesion indicated by rectangle 42 in FIG. 8 is inferred by the lesion detection and classification unit 34 to be a mass accompanied by spicules, and to have a stage of progression of 5. In this case, the result display unit 35 displays "Mass," indicating a mass, "Spiculated," indicating spicules accompanied, and "SMC-5," indicating the stage of progression, near the rectangle 42, as shown in FIG. 8. In this way, the type and stage of progression of the lesion are displayed near the rectangle surrounding the lesion.

[0039] Specifically, the result display unit 35 displays a diagnostic auxiliary screen in response to a predetermined operation by the user. Fig. 9 is an example of the diagnostic auxiliary screen. As shown in Fig. 9, the diagnostic auxiliary screen includes a chest image display unit 51, a mammary gland image display unit 52, a display bar 53, a detection button 54, and a display switching button 55. The chest image display unit 51 displays the original image, and the mammary gland image display unit 52 displays an image showing mammary gland tissue as a mammary gland structure extracted from the original image.

[0040] The display bar 53 is a bar for adjusting the degree to which the mammary gland structure is extracted, and more specifically, for adjusting the threshold value used to extract the mammary gland tissue. The mammary gland structure extraction unit 32 extracts the mammary gland structure from the original image using the threshold value indicated by the display bar 53. For example, moving the right end of the black bar to the right increases the threshold value, and only large mammary gland tissue is displayed in the mammary gland image display unit 52. Moving the right end of the black bar to the left decreases the threshold value, and even fine mammary gland tissue is displayed in the mammary gland image display unit 52. The result display unit 35 then displays an image showing the mammary gland structure extracted by the mammary gland structure extraction unit 32.

[0041] When the detection button 54 is pressed, the breast image display unit 52 displays an image showing mammary gland tissue and a thick white rectangle indicating the location of a lesion. When the display switch button 55 is pressed, it switches between displaying a rectangle indicating the location of a lesion and a rectangle indicating a difficult-to-diagnose location. Specifically, when the user presses the display switch button 55 multiple times, the result display unit 35 acquires the selection result by selecting one of a display mode that displays only a rectangle indicating the location of a lesion, a display mode that displays only a rectangle indicating a difficult-to-diagnose location, and a display mode that displays a rectangle indicating the location of a lesion and a rectangle indicating a difficult-to-diagnose location. The result display unit 35 then displays a rectangle indicating the location of a lesion and a rectangle indicating a difficult-to-diagnose location in a display mode corresponding to the selection result. When the rectangle indicating the location of a lesion is displayed, the type and progression of the detected lesion are displayed in text near the rectangle indicating the location of the lesion.

[0042] The diagnostic assistance screen does not display an image of mammary gland tissue, and only displays the chest image, display bar 53, detection button 54, and display switch button 55, so that the rectangles indicating the location of the lesion and the rectangles indicating the difficult-to-diagnose location do not interfere with the doctor's diagnosis, until the detection button 54 is pressed. When the user wants to check the location, type, and progression of the lesion or the difficult-to-diagnose location, which are the inference results of the image processing device 20, while diagnosing the chest image, the user presses the detection button 54. Then, by pressing the display switch button 55, the user selects and displays a mode that displays either or both the rectangle indicating the location of the lesion and the rectangle indicating the difficult-to-diagnose location. This allows the image processing device 20 to use the diagnostic assistance screen to present the user with the chest image, the mammary gland structure, the difficult-to-diagnose location, and the location, type, and progression of the lesion detected and differentiated by the inference of the lesion detection and differentiation unit 34, as needed.

[0043] In the present disclosure, difficult-to-diagnose locations are displayed as white rectangles and lesion locations are displayed as thick white rectangles, but this is not limited to this and any display method can be used as long as it can indicate difficult-to-diagnose locations and lesion locations, respectively. In addition, in the present disclosure, the type and progression of the lesion are displayed in text near the rectangle surrounding the lesion, but this is not limited to this and any display method can be used as long as it is possible to determine the correspondence between the location of the lesion and the type and progression of the lesion.

[0044] In addition, in the present disclosure, the screen shown in Figure 9 is displayed as a diagnostic auxiliary screen, but this is not limited to this, and the layout, design, and display timing of images and user interfaces (UIs) can be set arbitrarily.

[0045] In addition, in the present disclosure, the diagnostic assistance screen includes a chest image display unit 51 and a mammary gland image display unit 52, and displays a chest image and an image showing mammary gland tissue separately. However, this is not limited to this, and the image showing mammary gland tissue may be superimposed on the chest image. In this case, it is desirable to display the mammary gland tissue in a color that is easy to understand even when superimposed on the chest image. Rectangles indicating the location of a lesion and rectangles indicating difficult-to-diagnose locations are displayed on the image showing mammary gland tissue superimposed on the chest image. In this case, the type and progression of the lesion are also displayed in text near the rectangle indicating the location of the lesion.

[0046] Alternatively, the diagnostic assistance screen may display a rectangle indicating the location of a lesion or a difficult-to-diagnose location on the chest image without displaying an image of the mammary gland tissue. In this case, the type and progression of the lesion are also displayed in text form near the rectangle indicating the location of the lesion.

[0047] In the above configuration, the mammary gland structure extraction unit 32, the difficult-to-diagnose position detection unit 33, the lesion detection and identification unit 34, and the result display unit 35 of the image processing device 20 are examples of the mammary gland structure extraction means, the difficult-to-diagnose position detection means, the lesion detection and identification means, and the result display means of the image processing device of the present disclosure, respectively. Also, the result display unit 35 of the image processing device 20 is an example of the selection result acquisition means and the threshold acquisition means of the present disclosure.

[0048] (Lesion inference processing) Next, a description will be given of the lesion inference processing performed by the image processing device 20. Fig. 10 is a flowchart of the lesion inference processing performed by the image processing device 20. This processing is realized by the processor 22 shown in Fig. 2 executing a program prepared in advance.

[0049] First, the image processing device 20 acquires an original image, which is a chest image of a subject to be diagnosed by a doctor (step S101). Next, the image processing device 20 extracts mammary gland structures from the original image using an algorithm for extracting mammary gland structures from chest images (step S102). Next, the image processing device 20 detects difficult-to-diagnose locations from the original image and information on the mammary gland structures extracted from the original image (step S103). Specifically, the image processing device 20 inputs the original image and information on the mammary gland structures extracted from the original image into a difficult-to-diagnose location output model, and detects optimized difficult-to-diagnose locations output from the difficult-to-diagnose location output model.

[0050] Next, the image processing device 20 detects and differentiates the lesion by inferring the position, type, and progression of the lesion from the original image, the mammary gland structure extracted from the original image, and information on the difficult-to-diagnose location (step S104). Specifically, the image processing device 20 inputs the original image, the mammary gland structure extracted from the original image, and information on the difficult-to-diagnose location into the lesion output model, and outputs the position, type, and progression of the lesion as the inference results.

[0051] The image processing device 20 then displays a diagnostic assistance screen to present the original image, the mammary gland structure in the original image, the location of the difficult-to-diagnose lesion, and the location, type, and progression of the lesion to the user (step S105). This completes the lesion inference process.

[0052] In the present disclosure, when displaying the location of a lesion, the type and progression level are displayed nearby, but this is not limited to this, and whether or not to display the type and progression level of the lesion can be set arbitrarily. For example, the type and progression level of the lesion may not be displayed, or only the type or progression level of the lesion may be displayed near a rectangle indicating the location of the lesion. Furthermore, the type of lesion may be displayed as either a mass or calcification, or only the shape of the lesion.

[0053] As described above, the image processing device 20 detects difficult-to-diagnose locations from chest images, thereby making it possible to appropriately present to the user risk locations, which are tissues with low visibility on images, such as lesions along the complex course of mammary gland tissue. Furthermore, the image processing device 20 can infer information about the lesion, such as its location, type, and progression, taking into account the difficult-to-diagnose locations that are risk locations, and present the inference results to the user. This can assist doctors in diagnosing chest images and prevent lesions from being overlooked.

[0054] [Modifications] (First Modification) In the image processing system 100, in addition to the mammography apparatus 10 and the image processing device 20, an image storage server may be communicably connected via the network 5. In this case, the chest image DB 27 is stored in the image storage server, not in the image processing device 20. When making a diagnosis, the image processing device 20 acquires chest images from the chest image DB 27 via the network 5.

[0055] (Second Modification) In the image processing system 100, in addition to the mammography device 10 and the image processing device 20, a PC used by a user such as a doctor may be communicably connected via the network 5. In this case, the image processing device 20 can transmit a diagnostic auxiliary screen to the PC used by the user and display it by performing a predetermined operation.

[0056] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0057] (Supplementary Note 1) An image processing device comprising: a mammary gland structure extraction means for extracting mammary gland structure from a chest image; a difficult-to-diagnose position detection means for detecting, from information on the chest image and the mammary gland structure, a difficult-to-diagnose position where the shape of the mammary gland is complex and diagnosis is difficult; a lesion detection and identification means for inferring the position of a lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose position; and a result display means for displaying the chest image and the position of the lesion.

[0058] (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the result display means further displays the mammary gland structure.

[0059] (Supplementary Note 3) The image processing device according to Supplementary Note 2, wherein the result display means further displays the difficult-to-diagnose location.

[0060] (Appendix 4) The image processing device according to Appendix 1, wherein the lesion detection and identification means further infers the type of the lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose location, and the result display means further displays the type of the lesion.

[0061] (Appendix 5) The image processing device according to Appendix 4, wherein the lesion detection and discrimination means infers whether the lesion is a mass or calcification as the type of the lesion, and the result display means displays whether the lesion is a mass or calcification as the type of the lesion.

[0062] (Appendix 6) The image processing device according to Appendix 4, wherein the lesion detection and identification means infers whether the shape of the lesion is clearly defined, finely lobulated, unclearly defined, difficult to evaluate, or accompanied by spicules, as the type of the lesion, and the result display means displays whether the shape of the lesion is clearly defined, finely lobulated, unclearly defined, difficult to evaluate, or accompanied by spicules, as the type of the lesion.

[0063] (Appendix 7) The image processing device according to Appendix 1, wherein the lesion detection and identification means further infers the degree of progression of the lesion from the information on the chest image, the mammary gland structure, and the difficult-to-diagnose location, and the result display means further displays the degree of progression of the lesion.

[0064] (Supplementary Note 8) The image processing device according to Supplementary Note 3, wherein the result display means displays rectangles indicating the position of the lesion and the difficult-to-diagnose position on the mammary gland structure.

[0065] (Supplementary Note 9) The image processing device according to Supplementary Note 3, wherein the result display means displays the mammary gland structure so as to be superimposed on the chest image, and displays rectangles on the mammary gland structure indicating the position of the lesion and the difficult-to-diagnose position, respectively.

[0066] (Supplementary Note 10) The image processing device according to Supplementary Note 8 further comprises a selection result acquisition means for acquiring which of a display mode that displays only a rectangle indicating the position of the lesion, a display mode that displays only a rectangle indicating the difficult-to-diagnose position, and a display mode that displays a rectangle indicating the position of the lesion and a rectangle indicating the difficult-to-diagnose position has been selected, wherein the result display means displays a rectangle indicating the position of the lesion and a rectangle indicating the difficult-to-diagnose position according to the selected display mode.

[0067] (Appendix 11) The image processing device according to Appendix 1 further comprises a threshold acquisition means for acquiring a threshold for extracting the mammary gland structure from the chest image, wherein the mammary gland structure extraction means extracts the mammary gland structure from the chest image using the acquired threshold, and the result display means displays the extracted mammary gland structure.

[0068] (Supplementary Note 12) The image processing device according to Supplementary Note 1, wherein the original image is a chest image from which the position of the lesion is to be inferred, the mammary gland structure extraction means extracts the mammary gland structure from the original image, the difficult-to-diagnose position detection means detects the difficult-to-diagnose position using a machine learning model trained to output an optimized difficult-to-diagnose position in response to input of information on the original image and the mammary gland structure, and the lesion detection and identification means infers the position of the lesion using a machine learning model trained to output an optimized lesion position in response to input of information on the original image, the mammary gland structure, and the difficult-to-diagnose position.

[0069] (Supplementary Note 13) An image processing method that extracts a mammary gland structure from a chest image, detects a difficult-to-diagnose location where the mammary gland has a complex shape and is difficult to diagnose from information on the chest image and the mammary gland structure, performs an inference process to infer the location of a lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose location, and displays the chest image and the location of the lesion.

[0070] (Supplementary Note 14) The image processing method according to Supplementary Note 13, wherein the mammary gland structure is displayed in addition to the chest image and the location of the lesion.

[0071] (Supplementary Note 15) The image processing method according to Supplementary Note 14, wherein the difficult-to-diagnose location is displayed in addition to the chest image, the location of the lesion, and the mammary gland structure.

[0072] (Supplementary Note 16) The image processing method according to Supplementary Note 13, further inferring the type of the lesion from the information on the chest image, the mammary gland structure, and the difficult-to-diagnose location by the inference process, and displaying the type of lesion in addition to the chest image and the location of the lesion.

[0073] (Supplementary Note 17) A recording medium having recorded thereon a program that causes a computer to execute the following processes: extracting mammary gland structure from a chest image; detecting difficult-to-diagnose locations where the mammary gland has a complex shape and is difficult to diagnose from information on the chest image and the mammary gland structure; performing an inference process to infer the location of a lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose location; and displaying the chest image and the location of the lesion.

[0074] (Supplementary Note 18) The recording medium according to Supplementary Note 17, having recorded thereon a program for causing a computer to execute a process of displaying the breast image, the location of the lesion, and the mammary gland structure.

[0075] (Supplementary Note 19) A recording medium according to Supplementary Note 18, having recorded thereon a program for causing a computer to execute a process of displaying the difficult-to-diagnose location in addition to the chest image, the location of the lesion, and the mammary gland structure.

[0076] (Appendix 20) A recording medium according to Appendix 17, which stores a program that causes a computer to execute the process of further inferring the type of lesion from information on the chest image, the mammary gland structure, and the difficult-to-diagnose location through the inference process, and displaying the type of lesion in addition to the chest image and the location of the lesion.

[0077] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. In other words, the present disclosure naturally includes various modifications and alterations that would be possible for a person skilled in the art in accordance with the entire disclosure, including the claims, and the technical concept.

[0078] 5 Network 10 Mammography device 20 Image processing device 21 Interface 22 Processor 23 Memory 24 Recording medium 25 Display unit 26 Input unit 27 Chest image DB 31 Chest image acquisition unit 32 Mammary gland structure extraction unit 33 Difficult-to-diagnose position detection unit 34 Lesion detection and differentiation unit 35 Result display unit 100 Image processing system

Claims

1. An image processing apparatus comprising: a breast structure extraction means for extracting a breast structure from a chest image; a diagnosis difficulty position detection means for detecting a diagnosis difficulty position, which is a position where the shape of the breast is complex and diagnosis is difficult, from the chest image and the information of the breast structure; a lesion detection and discrimination means for inferring the position of a lesion from the chest image, the breast structure, and the information of the diagnosis difficulty position; and a result display means for displaying the chest image and the position of the lesion.

2. The image processing apparatus according to claim 1, wherein the result display means further displays the breast structure.

3. The image processing apparatus according to claim 2, wherein the result display means further displays the diagnosis difficulty position.

4. The lesion detection and discrimination means further infers the type of the lesion from the chest image, the breast structure, and the information of the diagnosis difficulty position, and the result display means further displays the type of the lesion. The image processing apparatus according to claim 1.

5. The lesion detection and discrimination means infers whether the lesion is a tumor or calcification as the type of the lesion, and the result display means displays whether the lesion is a tumor or calcification as the type of the lesion. The image processing apparatus according to claim 4.

6. The lesion detection and discrimination means infers whether the shape of the lesion is clearly demarcated, finely lobulated, ill-defined, difficult to evaluate, or spiculated as the type of the lesion, and the result display means displays whether the shape of the lesion is clearly demarcated, finely lobulated, ill-defined, difficult to evaluate, or spiculated as the type of the lesion. The image processing apparatus according to claim 4.

7. The lesion detection and discrimination means further infers the degree of progression of the lesion from the chest image, the breast structure, and the information of the diagnosis difficulty position, and the result display means further displays the degree of progression of the lesion. The image processing apparatus according to claim 1.

8. The image processing apparatus according to claim 3, wherein the result display means displays rectangles indicating the position of the lesion and the diagnosis difficulty position on the breast structure, respectively.

9. The image processing apparatus according to claim 3, wherein the result display means displays the breast structure so as to be superimposed on the chest image, and displays rectangles indicating the position of the lesion and the diagnosis difficulty position on the breast structure, respectively.

10. The image processing apparatus according to claim 8, further comprising selection result acquisition means for acquiring which one of a display mode for displaying only a rectangle indicating the position of the lesion, a display mode for displaying only a rectangle indicating the diagnostically difficult position, and a display mode for displaying a rectangle indicating the position of the lesion and a rectangle indicating the diagnostically difficult position is selected; and the result display means displays a rectangle indicating the position of the lesion and a rectangle indicating the diagnostically difficult position according to the selected display mode.

11. The image processing apparatus according to claim 1, further comprising threshold acquisition means for acquiring a threshold for extracting the breast structure from the chest image; the breast structure extraction means extracts the breast structure from the chest image using the acquired threshold; and the result display means displays the extracted breast structure.

12. The original image is a chest image that is a target for inferring the position of the lesion; the breast structure extraction means extracts the breast structure from the original image; the diagnostically difficult position detection means detects the diagnostically difficult position using a machine learning model that is learned to output the optimized diagnostically difficult position in response to the input of the original image and information on the breast structure; and the lesion detection and discrimination means infers the position of the lesion using a machine learning model that is learned to output the optimized position of the lesion in response to the input of the original image, information on the breast structure, and information on the diagnostically difficult position.

13. An image processing method for extracting a breast structure from a chest image, detecting a diagnostically difficult position, which is a position where the shape of the breast is complex and diagnosis is difficult, from the chest image and information on the breast structure, performing an inference process for inferring the position of a lesion from the chest image, information on the breast structure, and information on the diagnostically difficult position, and displaying the chest image and the position of the lesion.

14. The image processing method according to claim 13, further displaying the breast structure in addition to the chest image and the position of the lesion.

15. The image processing method according to claim 14, further displaying the diagnostically difficult position in addition to the chest image, the position of the lesion, and the breast structure.

16. The image processing method according to claim 13, further inferring the type of the lesion from the chest image, information on the breast structure, and information on the diagnostically difficult position by the inference process, and displaying the type of the lesion in addition to the chest image and the position of the lesion.

17. A recording medium storing a program that causes a computer to perform an inference process of extracting a breast structure from a chest image, detecting a difficult diagnosis position, which is a position where the shape of the breast is complex and diagnosis is difficult, from the information of the chest image and the breast structure, inferring the position of a lesion from the information of the chest image, the breast structure, and the difficult diagnosis position, and displaying the chest image and the position of the lesion.

18. The recording medium according to claim 17, storing a program that causes a computer to perform a process of displaying the breast structure in addition to the chest image and the position of the lesion.

19. The recording medium according to claim 18, storing a program that causes a computer to perform a process of displaying the difficult diagnosis position in addition to the chest image, the position of the lesion, and the breast structure.

20. The recording medium according to claim 17, storing a program that causes a computer to perform a process of inferring the type of the lesion from the information of the chest image, the breast structure, and the difficult diagnosis position by the inference process, and displaying the type of the lesion in addition to the chest image and the position of the lesion.

Images