Image processing device, image processing method, and storage medium

Abstract

An image processing device 1X primarily comprises a mammary gland structure extraction means 32X, an adjustment means 34X, and a display control means 35X. The mammary gland structure extraction means 32X extracts a mammary gland structure from each of a first chest image that results from imaging the chest of a subject and a second chest image that results from imaging the chest of the subject at a different time from the first chest image. On the basis of the extracted mammary gland structure, the adjustment means 34X adjusts the first chest image, treating the second chest image as a reference. The display control means 35X displays the adjusted first chest image and the second chest image on a display device. The image processing device 1X is used, for example, in the decision-making of a medical worker.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images.

[0002] Image processing systems that process chest images, such as mammography images, using computer image analysis have been known for some time. For example, Patent Document 1 discloses an information processing system that generates and displays image data that indicates areas where lesions are likely to be overlooked, based on mammography images and additional data.

[0003] Japanese Patent Application Laid-Open No. 2021-170174

[0004] When a medical professional such as a doctor refers to multiple chest images of the same subject taken at different times, the correspondence between these chest images may not be clear due to differences in the specifications of the imaging device, making it difficult for the medical professional to accurately grasp changes in the subject's chest over time.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that can suitably display chest images of the same subject taken at different times.

[0006] One aspect of the image processing device is an image processing device having: a mammary gland structure extraction means for extracting a mammary gland structure from a first chest image of a subject's chest and a second chest image of the subject's chest that is captured at a different time from the first chest image; an adjustment means for adjusting the first chest image based on the extracted mammary gland structure and using the second chest image as a reference; and a display control means for displaying the adjusted first chest image and second chest image on a display device.

[0007] One aspect of the image processing method is an image processing method in which a computer extracts mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest that is captured at a different time from the first chest image, adjusts the first chest image based on the extracted mammary gland structures and using the second chest image as a reference, and displays the adjusted first chest image and second chest image on a display device.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: extracting mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest that is taken at a different time from the first chest image; adjusting the first chest image based on the extracted mammary gland structures and using the second chest image as a reference; and displaying the adjusted first and second chest images on a display device.

[0009] As an example of an effect of the present disclosure, it becomes possible to preferably display chest images of the same subject taken at different times.

[0010] FIG. 1 shows a schematic configuration of a chest image processing system. FIG. 2 shows an overview of adjustment of a first chest image performed by an image processing device. FIG. 3 shows a specific example of first adjustment by affine transformation based on non-soft tissue. FIG. 4 shows an example of functional blocks of a processor of an image processing device. FIG. 5 shows a first display example of a display screen. FIG. 6 shows a second display example of a display screen. FIG. 7 shows a third display example of a display screen. FIG. 8 shows an example of a flowchart outlining processing performed by an image processing device. FIG. 9 is a block diagram of an image processing device. FIG. 10 shows an example of a flowchart performed by an image processing device.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Fig. 1 shows a schematic configuration of a chest image processing system 100. The chest image processing system 100 is a system that adjusts one of two chest images of a patient's chest, which are captured at different times, to match the other chest image, and presents the adjusted chest image to an examiner. As shown in Fig. 1, the chest image processing system 100 mainly includes an image processing device 1, a display device 3, and an operation device 4.

[0013] The image processing device 1 adjusts one of two chest images taken at different times of the chest of the same patient to match the other chest image. The image processing device 1 also controls the display of the adjusted chest image on the display device 3 and performs various processes based on operation signals received from the operation device 4.

[0014] The chest image described above is generated by a chest image generating device that generates a chest image (an X-ray image of the breast) by irradiating X-rays toward a patient's breast and detecting the X-rays that have passed through the breast. The chest image generating device may be a mammography device based on the CR (Computed Radiography) method or a mammography device based on the FPD (Flat Panel Detector) method. The chest image generated by the chest image generating device is digital data that can be analyzed by the image processing device 1. For example, pixels in the chest image are displayed with a higher pixel value (brightness) as the X-ray absorption increases, and pixels with higher pixel values ​​are drawn whiter on the chest image. The image processing device 1 is used, for example, for decision-making by medical professionals.

[0015] Hereinafter, the chest image to be adjusted (more specifically, transformed for adjustment) will be referred to as the "first chest image," and the chest image that is fixed without adjustment (i.e., the chest image that serves as the reference for adjusting the first chest image) will be referred to as the "second chest image." The first and second chest images may be generated by different chest image generating devices or by the same chest image generating device. For example, one chest image may be an image captured during a patient's medical checkup, and the other chest image may be an image captured before surgery or during a detailed examination because a diagnosis based on the first chest image indicated the need for surgery or a detailed examination.

[0016] The display device 3 performs a predetermined display based on a display signal supplied from the image processing device 1. Examples of the display device 3 include displays such as a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), as well as a projector.

[0017] The operation device 4 generates an operation signal based on an operation by a user, such as a doctor, of the image processing device 1. Examples of the operation device 4 include a button, a keyboard, a pointing device such as a mouse, a touch panel, a remote controller, a voice input device, and any other user interface.

[0018] 1 also shows an example of the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, and an interface 13. These elements are connected via a data bus 19.

[0019] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0020] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for the processing of the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs and other information necessary for the image processing device 1 to execute each process in this embodiment.

[0021] For example, the memory 12 stores a first chest image and a second chest image. In this case, the first chest image and the second chest image may be stored in association with at least one of patient information and examination information of the patient who is the subject of the chest images. Examples of patient information include patient identification information (patient ID) for identifying the patient and other patient attribute information (name, gender, date of birth, etc.). Examples of examination information include examination identification information (examination ID) for identifying the examination, examination date and time information, examination conditions (examination area, laterality (left, right), direction (e.g., CC direction, mediolateral oblique direction (MLO), compressed breast thickness), etc.

[0022] The memory 12 also stores model information related to an inference model (inference engine), which is a machine learning model that infers an image obtained by adjusting the first chest image. The model information is information necessary to configure an inference model that has undergone machine learning and includes trained parameters of the inference model. Details of the inference model will be described later. The inference model is, for example, a deep learning model that includes a neural network in its architecture. Representative models of such neural networks include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. When the inference model is configured using a neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weights of each element of each filter are pre-stored in the memory 12 as trained parameters.

[0023] At least one of the chest image and model information described above may be stored in any storage device that performs data communication with the image processing device 1, instead of being stored in the memory 12. In this case, the storage device is capable of data communication with the image processing device 1, and performs processing according to this embodiment based on information acquired from the storage device. The storage device may be a server device that performs data communication with the image processing device 1 via a communication network.

[0024] The interface 13 acts as an interface between the image processing device 1 and external devices. For example, the interface 13 is electrically connected to external devices such as the display device 3 and the operation device 4. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with external devices, or may be a hardware interface conforming to USB (Universal Serial Bus) or SATA (Serial AT Attachment). The interface 13 may also act as an interface with external devices such as the display device 3 and the operation device 4 via a communication network such as the Internet. Examples of external devices include the chest image generation device and the storage device described above.

[0025] The configuration of the chest image processing system 100 shown in FIG. 1 is an example, and various modifications may be made.

[0026] For example, the image processing device 1 may be configured integrally with at least one of the display device 3 and the operation device 4. In another example, the image processing device 1 may include an audio output device that outputs information by audio. In yet another example, the image processing device 1 may be configured from multiple devices.

[0027] (2) Adjustment of the First Chest Image In general, the image processing device 1 first applies an adjustment (also referred to as the "first adjustment") to the first chest image to roughly align the first chest image with the second chest image. Next, the image processing device 1 performs a final adjustment (also referred to as the "second adjustment") to the first chest image based on an inference model that utilizes the mammary gland structures of the first and second chest images. The first adjustment corresponds to pre-processing (intermediate adjustment) performed before the second adjustment, and the second adjustment corresponds to fine adjustment (final adjustment) for generating the first chest image to be finally displayed. Hereinafter, the first chest image to which the first adjustment has been applied will be referred to as the "intermediate-adjusted chest image," and the first chest image to which the second adjustment has been applied will be referred to as the "finally-adjusted chest image."

[0028] FIG. 2 is a diagram showing an outline of the adjustments (first adjustment and second adjustment) of the first chest image performed by the image processing device 1.

[0029] First, the image processing device 1 acquires a first chest image and a second chest image, which are chest images of a patient captured at different times. Fig. 2 shows an unadjusted first chest image 21 and a fixed second chest image 22. The first chest image 21 and the second chest image 22 differ in image size and other characteristics because they were generated by chest image generation devices with different specifications.

[0030] The image processing device 1 applies a first adjustment to the first chest image 21 to generate an intermediate-adjusted chest image 23, which is a first chest image adjusted so that the positions and brightness distributions of corresponding tissues in the second chest image match those in the first chest image 21. In this case, the image processing device 1 extracts regions of specific tissues that are unlikely to change shape from each of the first chest image 21 and the second chest image 22, and performs affine transformation (i.e., non-rigid registration) of the first chest image using the extracted regions as a reference to make the first chest image 21 appear closer to the second chest image 22. The "specific tissues that are unlikely to change shape" are specific tissues that are rigid and do not change easily over time, and are hereinafter also referred to as "non-soft tissues." Examples of non-soft tissues include pectoral muscles, ligaments, nipples, etc. The image processing device 1 also performs brightness adjustment (so-called gamma correction) on the first chest image 21 after the affine transformation so that the brightness distribution of the entire image of the intermediate-adjusted chest image 23 matches the brightness distribution of the entire image of the second chest image 22, thereby generating the intermediate-adjusted chest image 23.

[0031] 3 is a diagram showing a specific example of the first adjustment using affine transformation based on non-soft tissue. In FIG. 3, the image processing device 1 executes a process of extracting pectoral muscle regions, which are an example of non-soft tissue, from each of a first chest image 21 and a second chest image 22. As a result, the image processing device 1 generates a first mask image 21A representing the pectoral muscle regions in the first chest image 21 and a second mask image 21B representing the pectoral muscle regions in the second chest image 22. In the first mask image 21A and the second mask image 21B, the pectoral muscle regions are white and the other regions are black.

[0032] Next, the image processing device 1 calculates affine transformation parameters required to match or approximate the first mask image 21A to the second mask image 21B through affine transformation. In this case, the image processing device 1 may calculate the affine transformation parameters (affine transformation matrix) using any method used for aligning medical images. Examples of such methods include a method using image intensity and a method using feature points.

[0033] The image processing device 1 then applies affine transformation parameters to the first chest image 21 to deform it, and performs gamma correction based on the luminance distribution of the entire image, thereby generating an intermediate adjusted chest image 23.

[0034] Next, the process relating to the second adjustment will be described with reference to FIG.

[0035] The image processing device 1 performs processing to extract the mammary gland structure of the first chest image 21 and the second chest image 22, and generates a first mammary gland structure image 24 representing the mammary gland structure of the first chest image 21 and a second mammary gland structure image 25 representing the mammary gland structure of the second chest image 22. The mammary gland structure is the structure of mammary gland tissue, and the first mammary gland structure image 24 and the second mammary gland structure image 25 shown in Fig. 2 are mask images in which mammary gland tissue is colored white and the rest is colored black. Note that the image processing device 1 may extract the mammary gland structure of the intermediate-adjusted chest image 23 instead of extracting the mammary gland structure of the first chest image 21, and generate a second mammary gland structure image 25 representing the mammary gland structure of the intermediate-adjusted chest image 23.

[0036] Here, the algorithm for extracting mammary gland structures from chest images can be, for example, the method described in the following paper: Karla K Evans, Tamara Miner Haygood, Julie Cooper, Anne-Marie Culpan, Jeremy M Wolfe, "A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast", https: / / www.pnas.org / doi / full / 10.1073 / pnas.1606187113

[0037] In the second adjustment, the image processing device 1 uses the inference model to generate a final-adjusted chest image 26, which is the first chest image 21 that has been finally adjusted. In this case, the image processing device 1 generates an input image to be input to the inference model, in which the second chest image 22, the intermediate-adjusted chest image 23, the first mammary gland structure image 24, and the second mammary gland structure image 25 are superimposed in the channel direction. In this case, the image processing device 1 may perform normalization or the like so that the size of each channel of the input image (i.e., the number of vertical and horizontal pixels) becomes a predetermined size to match the input format to the inference model. The image processing device 1 then inputs the generated input image to an inference model configured with reference to the model information stored in the memory 12, and obtains a final-adjusted chest image 26, which is the inference result output by the inference model based on the input. As a result, the image processing device 1 obtains the final-adjusted chest image 26, which is an image that approximates the appearance of the first chest image to the second chest image while maintaining the correspondence between the mammary gland structures of the first and second chest images. The final adjusted chest image 26 corresponds to an image obtained by bringing the intermediate adjusted chest image 23 closer to the second chest image 22 through non-rigid registration and brightness correction.

[0038] After the second adjustment is performed, the image processing device 1 displays the acquired final adjusted chest image 26 together with the second chest image 22 on the display device 3 so that the examiner can visually recognize them. At this time, the image processing device 1 may also display a first mammary gland structure image 24 and a second mammary gland structure image 25 on the display device 3. Specific display examples will be described later.

[0039] In this way, the image processing device 1 can align (register) the first chest image 21 and the second chest image 22 captured by different systems, taking into account mammary gland density. Therefore, even if the patient's symptoms have changed over time during the imaging period between the first chest image 21 and the second chest image 22, the examiner can accurately follow up on such changes by comparing the final adjusted chest image 26 with the second chest image 22.

[0040] Here, we will provide additional information about the inference model used in the second adjustment. The inference model infers, as the final adjusted chest image, an image in which the appearance of the first chest image is made to resemble that of the second chest image without damaging the mammary gland structure so as not to destroy the correspondence between the mammary gland structure between the first chest image and the second chest image. The inference model is a machine-learned model that determines the relationship between the first chest image (which may be an intermediate adjusted chest image after the first adjustment), the second chest image, the first mammary gland structure image, and the second mammary gland structure image and the final adjusted chest image.

[0041] Machine learning is performed in advance on the inference model so that it outputs a final-adjusted chest image when an input image in which a first chest image, a second chest image, a first mammary gland structure image, and a second mammary gland structure image are superimposed in the channel direction is input. Parameters of the inference model obtained by machine learning are stored in the memory 12 or the like as model information. The inference model is trained using a training dataset including multiple records that combine an input image in which a first chest image (or an intermediate-adjusted chest image), a second chest image, a first mammary gland structure image, and a second mammary gland structure image are superimposed in the channel direction with a correct final-adjusted chest image to be output by the inference model. The correct final-adjusted chest image used in this case may be the second chest image included in the corresponding input image, or may be an image obtained by manually deforming the first chest image to approximate the second chest image. In addition, the input images input to the inference model are not limited to images in which the first chest image (or intermediate adjusted chest image), the second chest image, the first mammary gland structure image, and the second mammary gland structure image are superimposed in the channel direction, but may be images based on the first chest image, the second chest image, the first mammary gland structure image, and the second mammary gland structure image.

[0042] (3) Functional Blocks Figure 4 shows an example of functional blocks of the processor 11 of the image processing device 1. Functionally, the processor 11 of the image processing device 1 has a chest image acquisition unit 30, a non-soft tissue extraction unit 31, a mammary gland structure extraction unit 32, a first adjustment unit 33, a second adjustment unit 34, and a display control unit 35. Note that in Figure 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to other functional block diagrams described below.

[0043] The chest image acquisition unit 30 acquires two chest images of the patient's chest captured at different times. In this case, the chest image acquisition unit 30 may read the chest images from the memory 12 or may receive the chest images from an external device via the interface 13. The chest image acquisition unit 30 then uses one of the two acquired chest images as the first chest image and the other as the second chest image, and supplies the first chest image and the second chest image to the non-soft tissue extraction unit 31, the mammary gland structure extraction unit 32, and the display control unit 35, respectively. The chest image acquisition unit 30 may select the chest image captured earlier from the two chest images as the first chest image, or may determine the combination of the first and second chest images based on an input from the operation device 4.

[0044] The non-soft tissue extraction unit 31 extracts common non-soft tissue regions from the first chest image and the second chest image. For example, the non-soft tissue extraction unit 31 generates a mask image representing the non-soft tissue region in the first chest image and a mask image representing the non-soft tissue region in the second chest image. In this case, the non-soft tissue extraction unit 31 may use any image recognition method to extract the non-soft tissue region. For example, machine-learned parameters of a deep learning model (region extraction model) that extracts non-soft tissue regions from an input image are stored in the memory 12, etc. Then, the non-soft tissue extraction unit 31 inputs the chest image into the region extraction model configured with reference to the parameters, and acquires a mask image representing the non-soft tissue region in the input chest image based on the inference result output by the region extraction model. Such a region extraction model is machine-trained in advance using training data in which chest images and correct mask images representing the non-soft tissue regions in the chest image are records. The non-soft tissue extraction unit 31 supplies the first chest image and the second chest image, and the extraction results of the non-soft tissue regions of the first chest image and the second chest image (e.g., the above-mentioned mask image) to the first adjustment unit 33.

[0045] The first adjuster 33 performs a first adjustment on the first chest image based on the extraction results of non-soft tissue regions from the first and second chest images provided by the non-soft tissue extractor 31, generating an intermediate-adjusted chest image that serves as the first chest image after the first adjustment. In this case, the first adjuster 33 affinely transforms the second chest image so that the non-soft tissue regions extracted from the first and second chest images by the non-soft tissue extractor 31 have the same shape and size. Furthermore, the first adjuster 33 aggregates the luminance distributions of the first and second chest images and adjusts the luminance of the first chest image so that the luminance distribution of the adjusted first chest image matches the luminance distribution of the second chest image (e.g., matches a statistical quantity such as the average value). The first adjuster 33 provides the second chest image and the intermediate-adjusted chest image to the second adjuster 34.

[0046] The mammary gland structure extraction unit 32 extracts the mammary gland structure of the first and second chest images and supplies the extraction results to the second adjustment unit 34. In this case, the mammary gland structure extraction unit 32 supplies the extraction results, that is, a first mammary gland structure image representing the mammary gland structure of the first chest image and a second mammary gland structure image representing the mammary gland structure of the second chest image, to the second adjustment unit 34 and the display control unit 35.

[0047] The second adjustment unit 34 generates a final-adjusted chest image by finally adjusting the first chest image based on the information supplied from the mammary gland structure extraction unit 32 and the first adjustment unit 33, respectively, and an inference model constructed by referring to the memory 12, etc. In this case, the second adjustment unit 34 inputs an input image in which an intermediate-adjusted chest image obtained by adjusting the first chest image, the second chest image, the first mammary gland structure image, and the second mammary gland structure image are superimposed in the channel direction into the inference model, and obtains a final-adjusted chest image, which is an inference result output by the inference model in response to the input. The second adjustment unit 34 supplies the generated final-adjusted chest image to the display control unit 35.

[0048] The display control unit 35 controls the display of the display device 3 based on the first and second chest images supplied from the chest image acquisition unit 30, the first and second mammary gland structure images supplied from the mammary gland structure extraction unit 32, and the finally adjusted chest image supplied from the second adjustment unit 34. In this case, the display control unit 35 generates a display signal and supplies the generated display signal to the display device 3, thereby causing the display device 3 to display the above-mentioned image. Display examples displayed by the display device 3 based on the control of the display control unit 35 will be described later.

[0049] The components of the chest image acquisition unit 30, the non-soft tissue extraction unit 31, the mammary gland structure extraction unit 32, the first adjustment unit 33, the second adjustment unit 34, and the display control unit 35 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0050] (4) Display Examples Fig. 5 shows a first display example of a display screen that the display control unit 35 causes the display device 3 to display. The first display example is a display example displayed before the first chest image is adjusted, and the display control unit 35 outputs display information generated based on the first chest image, the second chest image, etc. supplied from the chest image acquisition unit 30 to the display device 3. The display control unit 35 transmits the display information to the display device 3, thereby causing the display screen shown in Fig. 5 to be displayed on the display device 3.

[0051] The display control unit 35 displays on the display screen a chest image of a patient named "XXX" taken at △△ Hospital on March 3, 2024, and a chest image of the same patient taken at □□ Hospital on April 2, 2024. The display control unit 35 also provides an image adjustment button 72 on the display screen and accepts input to instruct image adjustment related to registration.

[0052] 6 shows a second display example of the display screen that the display control unit 35 causes the display device 3 to display. In the second display example, the display control unit 35 detects that the image adjustment button 72 in the first display example has been selected, and displays a pop-up window 73 for confirming that adjustment of the alignment of the two chest images displayed in the first display example will begin.

[0053] The window 73 is provided with a first button 74 for instructing adjustment of the chest image captured on March 3, 2024 (Image A) to match the chest image captured on April 2, 2024 (Image B), and a second button 75 for instructing adjustment of Image B to match Image A. When the display control unit 35 detects that the first button 74 has been selected, it performs adjustment using the chest image captured on March 3, 2024 (Image A) as the first chest image and the chest image captured on April 2, 2024 (Image B) as the second chest image. On the other hand, when the display control unit 35 detects that the second button 75 has been selected, it performs adjustment using the chest image captured on March 3, 2024 (Image A) as the second chest image and the chest image captured on April 2, 2024 (Image B) as the first chest image.

[0054] Thus, according to the second display example, the display control unit 35 can accept a user input for determining a combination of the first and second chest images from two chest images. Note that instead of accepting a user input for determining a combination of the first and second chest images, the display control unit 35 may automatically determine that the chest image captured earlier (image A in this case) will be the first chest image and the chest image captured later (image B in this case) will be the second chest image.

[0055] 7 shows a third display example of the display screen that the display control unit 35 causes the display device 3 to display. In the third display example, the image processing device 1 uses the chest image captured on March 3, 2024 as the first chest image and the chest image captured on April 2, 2024 as the second chest image, and generates a final adjusted chest image by finally adjusting the chest image captured on March 3, 2024.

[0056] 7, the display control unit 35 displays the generated final adjusted chest image alongside the first mammary gland structure image in the first chest image display area 70. In this case, the final adjusted chest image is labeled "original image (adjusted)" and the first mammary gland structure image is labeled "mammary gland structure." Note that the mammary gland structure extraction unit 32 may perform a process to extract the mammary gland structure from the final adjusted chest image, and the display control unit 35 may display an image representing the mammary gland structure extracted from the final adjusted chest image instead of the first mammary gland structure image.

[0057] The display control unit 35 also displays the second chest image alongside the second mammary gland structure image in the second chest image display area 71. In this case, the second chest image is displayed as "original image (unadjusted)" and the second mammary gland structure image is displayed as "mammary gland structure."

[0058] Thus, according to the third display example, the display control unit 35 adjusts and displays the final adjusted chest image obtained by adjusting the first chest image and the second chest image so that the examiner can easily compare them, thereby allowing the examiner to easily follow the deterioration that occurs over time in the patient's chest.

[0059] (5) Processing Flow FIG. 8 is an example of a flowchart showing an outline of the processing executed by the image processing device 1 in the first embodiment.

[0060] First, the image processing device 1 acquires a first chest image and a second chest image of the same subject (step S11), and then extracts non-soft tissue areas that are common to both the first and second chest images (step S12).

[0061] The image processing device 1 then applies a first adjustment to the first chest image based on the non-soft tissue region extracted in step S12 (step S13). In this case, the image processing device 1 performs affine transformation on the first chest image so that the position and shape of the non-soft tissue region match, and further performs gamma correction on the first chest image in accordance with the luminance distribution of the second chest image. This generates an intermediate adjusted chest image.

[0062] Next, the image processing device 1 generates mammary gland structure images of the first and second chest images (step S14). In this case, the image processing device 1 generates a first mammary gland structure image of the first chest image and a second mammary gland structure image of the second chest image. The image processing device 1 may perform the process of step S14 in parallel with steps S12 and S13, or may perform the process before steps S12 and S13.

[0063] Next, the image processing device 1 acquires the first chest image after the second adjustment (i.e., the final adjusted chest image) based on the mammary gland structure images of the first and second chest images, the first chest image after the first adjustment (i.e., the intermediate adjusted chest image), the second chest image, and the inference model (step S15). In this case, the image processing device 1 generates an input image based on the mammary gland structure image and the first and second chest images after the first adjustment, and inputs the input image into the inference model to acquire an image output by the inference model.

[0064] Next, the image processing device 1 causes the display device 3 to display the first chest image after the second adjustment (i.e., the final adjusted chest image) and the second chest image (step S16).

[0065] (6) Modification The image processing device 1 may perform only the second adjustment without performing the first adjustment. For example, the first and second chest images may be stored in the memory 12 or the like in a state in which the first adjustment has been performed beforehand or in a state in which they roughly match to such an extent that the first adjustment is not necessary. The image processing device 1 generates a final adjusted chest image based on these first and second chest images using these mammary gland structure images and an inference model. Even in this case, the image processing device 1 can fine-tune the first chest image to match the second chest image and display the adjusted first and second chest images so that the examiner can compare them.

[0066] Alternatively, instead of generating a final adjusted chest image using an inference model, the second adjuster 34 may perform non-rigid registration of the first or intermediate adjusted chest image so that the mammary gland structure image of the first or intermediate adjusted chest image matches or approximates the second mammary gland structure image of the second chest image. In this case, the second adjuster 34 may perform the non-rigid registration using any non-rigid registration method applicable to medical images.

[0067] Second Embodiment Fig. 9 is a block diagram of an image processing device 1X. The image processing device 1X mainly includes a mammary gland structure extraction unit 32X, an adjustment unit 34X, and a display control unit 35X. The image processing device 1X may be composed of multiple devices. The image processing device 1X is used, for example, for decision-making by medical professionals.

[0068] The mammary gland structure extracting means 32X extracts mammary gland structures from a first chest image of the subject's chest and a second chest image of the subject's chest taken at a different time from the first chest image. The mammary gland structure extracting means 32X can be, for example, the mammary gland structure extractor 32 in the first embodiment.

[0069] The adjustment means 34X adjusts the first chest image based on the extracted mammary gland structure and the second chest image as a reference. The adjustment means 34X can be, for example, the second adjustment unit 34 in the first embodiment.

[0070] The display control means 35X displays the adjusted first and second chest images on a display device. The display control means 35X can be, for example, the display control unit 35 in the first embodiment.

[0071] 10 is an example of a flowchart showing the processing procedure executed by the image processing device 1X. The mammary gland structure extraction means 32X extracts mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest captured at a different time from the first chest image (step S21). Next, the adjustment means 34X adjusts the first chest image based on the extracted mammary gland structures and using the second chest image as a reference (step S22). The display control means 35X displays the adjusted first and second chest images on a display device (step S23).

[0072] According to the second embodiment, the image processing device 1X can suitably present to the user the first chest image and the second chest image adjusted based on the mammary gland structure.

[0073] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0074] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0075] [Supplementary Note 1] An image processing device comprising: a mammary gland structure extraction means for extracting a mammary gland structure from each of a first chest image of a subject's chest and a second chest image of the subject's chest taken at a different time from that of the first chest image; an adjustment means for adjusting the first chest image based on the extracted mammary gland structure and using the second chest image as a reference; and a display control means for displaying the adjusted first and second chest images on a display device. [Supplementary Note 2] The image processing device according to Supplementary Note 1, wherein the adjustment means adjusts the first chest image based on a machine learning model, the extracted mammary gland structure, the first chest image, and the second chest image, and the machine learning model is a model trained by machine learning to infer the first chest image adjusted based on the second chest image as a reference when an input image based on the extracted mammary gland structure, the first chest image, and the second chest image is input. [Supplementary Note 3] The image processing device according to Supplementary Note 1, further comprising region extraction means for extracting a region of a predetermined tissue from the first chest image and the second chest image, wherein the adjustment means performs the following: a first adjustment for adjusting the first chest image based on the region; and a second adjustment for adjusting the first chest image to which the first adjustment has been applied based on the extracted mammary gland structure and using the second chest image as a reference. [Supplementary Note 4] The image processing device according to Supplementary Note 3, wherein the adjustment means, in the first adjustment, performs affine transformation on the first chest image so that the region of the first chest image and the region of the second chest image coincide or approximate each other. [Supplementary Note 5] The image processing device according to Supplementary Note 3, wherein the predetermined tissue includes at least one of pectoral muscle, ligament, and nipple. [Supplementary Note 6] The image processing device according to Supplementary Note 1, wherein the display control means displays the adjusted first and second chest images on the display device together with an image representing the mammary gland structure extracted from the first chest image and an image representing the mammary gland structure extracted from the second chest image. [Supplementary Note 7] The image processing device according to Supplementary Note 1, wherein the display control means displays two chest images of the subject's chest on the display device and accepts an input for determining a combination of the first and second chest images from the two chest images.[Supplementary Note 8] The image processing device according to Supplementary Note 2, wherein the adjustment means generates the input image by superimposing, in a channel direction, an image representing the mammary gland structure extracted from the first chest image, an image representing the mammary gland structure extracted from the second chest image, the first chest image or an image obtained by adjusting the first chest image, and the second chest image. [Supplementary Note 9] An image processing method, wherein a computer extracts mammary gland structures from a first chest image obtained by capturing an image of a subject's chest and a second chest image obtained by capturing an image of the subject's chest at a different time from that of the first chest image, adjusts the first chest image based on the extracted mammary gland structure and uses the second chest image as a reference, and displays the adjusted first chest image and second chest image on a display device. [Supplementary Note 10] A program that causes a computer to execute the following processes: extracting mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest that is taken at a different time from the first chest image; adjusting the first chest image based on the extracted mammary gland structures using the second chest image as a reference; and displaying the adjusted first and second chest images on a display device. [Supplementary Note 11] A storage medium that stores the program described in Supplementary Note 10.

[0076] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0077] REFERENCE SIGNS LIST 1, 1X image processing device 3 display device 4 operation device 11 processor 12 memory 13 interface 100 chest image processing system

Claims

1. An image processing device having: a mammary gland structure extraction means for extracting mammary gland structure from a first chest image of a subject's chest and a second chest image of the subject's chest taken at a different time from the first chest image; an adjustment means for adjusting the first chest image based on the extracted mammary gland structure and using the second chest image as a reference; and a display control means for displaying the adjusted first and second chest images on a display device.

2. The image processing device of claim 1, wherein the adjustment means adjusts the first chest image based on a machine learning model, the extracted mammary gland structure, the first chest image, and the second chest image, and the machine learning model is a model that has been trained by machine learning to infer the first chest image adjusted based on the second chest image when an input image based on the extracted mammary gland structure, the first chest image, and the second chest image is input.

3. An image processing device as described in claim 1, further comprising an area extraction means for extracting an area of ​​a predetermined tissue from the first chest image and the second chest image, wherein the adjustment means performs a first adjustment for adjusting the first chest image based on the area, and a second adjustment for adjusting the first chest image to which the first adjustment has been applied based on the extracted mammary gland structure and using the second chest image as a reference.

4. An image processing device according to claim 3, wherein said adjustment means, in said first adjustment, performs affine transformation on said first chest image so that said area of ​​said first chest image coincides with or approximates said area of ​​said second chest image.

5. The image processing device according to claim 3, wherein the predetermined tissue includes at least one of a pectoral muscle, a ligament, and a nipple.

6. The image processing device according to claim 1, wherein the display control means displays the adjusted first and second chest images on the display device together with an image representing the mammary gland structure extracted from the first chest image and an image representing the mammary gland structure extracted from the second chest image.

7. The image processing device according to claim 1, wherein the display control means displays two chest images of the subject's chest on the display device and accepts input for determining a combination of the first chest image and the second chest image from the two chest images.

8. The image processing device according to claim 2, wherein the adjustment means generates the input image by superimposing, in the channel direction, an image representing the mammary gland structure extracted from the first chest image, an image representing the mammary gland structure extracted from the second chest image, the first chest image or an image obtained by adjusting the first chest image, and the second chest image.

9. An image processing method in which a computer extracts mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest taken at a different time from the first chest image, adjusts the first chest image based on the extracted mammary gland structures and using the second chest image as a reference, and displays the adjusted first and second chest images on a display device.

10. A storage medium containing a program that causes a computer to execute the following process: extracting mammary gland structures from a first chest image of a subject's chest and a second chest image of the subject's chest taken at a different time from the first chest image; adjusting the first chest image based on the extracted mammary gland structures and using the second chest image as a reference; and displaying the adjusted first and second chest images on a display device.

Images