Image processing device, image processing method and program

Abstract

This image processing device 1X is provided with a detection means 31X, an acquisition means 32X, a deformation means 33X, and a display control means 34X. The detection means 31X detects a dense part in which tissue is densely packed from a chest image. The acquisition means 32X acquires a first image obtained by cutting out a region based on the dense part from the chest image. The deformation means 33X deforms the first image on the basis of information on the density of tissue in the first image. The display control means 34X displays a second image obtained by deforming the first image on a display device. The invention can be used to assist a user in decision making, etc.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images.

[0002] Image processing systems that process chest images such as mammography images using computer image analysis have been known for some time. For example, Patent Literature 1 discloses an information processing system that generates and displays oversight prevention data, including image data indicating areas where lesions are likely to be overlooked, based on mammography images and additional data.

[0003] Japanese Patent Application Laid-Open No. 2021-170174

[0004] Areas where lesions are likely to be overlooked are usually areas where tissue is densely packed and visibility is low, so even if areas where lesions are likely to be overlooked are detected and displayed as in Patent Document 1, it is difficult for the examiner to confirm whether or not there is an abnormality in the displayed area.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of suitably displaying areas where tissue is densely packed in a chest image.

[0006] One aspect of the image processing device is an image processing device having: a detection means for detecting dense tissue areas from a chest image; an acquisition means for acquiring a first image obtained by cutting out an area based on the dense tissue areas from the chest image; a transformation means for transforming the first image based on information about the tissue density in the first image; and a display control means for displaying a second image obtained by transforming the first image on a display device.

[0007] One aspect of the image processing method is an image processing method in which a computer detects tissue-dense areas from a chest image, obtains a first image by cutting out an area based on the tissue-dense areas from the chest image, deforms the first image based on information about the tissue density in the first image, and displays a second image obtained by deforming the first image on a display device.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: detect tissue-dense areas from a chest image; obtain a first image by cutting out an area based on the tissue-dense areas from the chest image; deform the first image based on information about the tissue density in the first image; and display a second image obtained by deforming the first image on a display device.

[0009] As an example of an effect of the present disclosure, it becomes possible to suitably display areas where tissue is densely packed in a chest image.

[0010] 1 shows a schematic configuration of a chest image processing system. FIG. 2 shows an overview of image processing performed by an image processing device in the first embodiment. FIG. 3 shows an example of functional blocks of a processor of an image processing device in the first embodiment. (A) shows an example of a target tissue mask image. (B) shows a target tissue density image calculated based on the target tissue mask image. (A) shows a target tissue density image. (B) is a diagram visualizing a deformation vector for each pixel calculated based on the target tissue density image. (A) shows a target tissue density image that clearly indicates the target tissue density for each pixel. (B) shows the correspondence between each pixel of the target tissue density image and X-Y coordinates when the area of ​​the target tissue density image is considered to be an X-Y coordinate space. FIG. 4 shows a first display example of a display screen. (B) shows a second display example of a display screen. (C) shows a third display example of a display screen. FIG. 5 shows an example of a flowchart showing an overview of processing performed by an image processing device in the first embodiment. FIG. 6 is a block diagram of an image processing device in the second embodiment. FIG. 7 shows an example of a flowchart performed by an image processing device in the second embodiment.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Fig. 1 shows a schematic configuration of a chest image processing system 100. As shown in Fig. 1, the chest image processing system 100 is a system that deforms an image of a patient (a subject) that shows areas with high tissue density and low visibility by non-rigid deformation to improve visibility, and presents the deformed image to an examiner. The chest image processing system 100 mainly includes an image processing device 1, a chest image generating device 2, a display device 3, and an operation device 4.

[0013] The image processing device 1 detects areas of dense tissue, cuts out areas based on the detected areas of dense tissue, and deforms the cut-out images based on the chest images of the patient supplied from the chest image generating device 2. The image processing device 1 also performs display control to display information based on the deformed images on the display device 3, and performs various processes based on operation signals received from the operation device 4.

[0014] The chest image generation device 2 generates chest images by capturing images of the patient's chest, which is the imaging target region, and supplies the generated chest images to the image processing device 1. For example, the chest image generation device 2 generates chest images (breast X-ray images) by performing X-ray photography, in which X-rays are irradiated toward the patient's breasts and the X-rays that pass through the breasts are detected. The chest image generation device 2 may be a mammography device based on the CR (Computed Radiography) method or a mammography device based on the FPD (Flat Panel Detector) method. The chest images generated by the chest image generation device 2 are digital data that can be analyzed by the image processing device 1. For example, pixels in the chest image are displayed with a larger pixel value (brightness) as the X-ray absorption increases, and pixels with a larger pixel value are drawn whiter on the chest image. The chest images generated by the chest image generating device 2 are not limited to mammography images, but may also be ultrasound images, MRI images, CT images, chest X-ray images, angio images, and the like.

[0015] Furthermore, in addition to the chest images, the chest image generating device 2 may also supply at least one of patient information and examination information of the patient who is the subject of the chest images to the image processing device 1. In this case, examples of the patient information include patient identification information (patient ID) for identifying the patient and other patient attribute information (name, gender, date of birth, etc.). Examples of the examination information include examination identification information (examination ID) for identifying the examination, examination date and time information, examination conditions (examination area, laterality (left, right), direction (e.g., CC direction, mediolateral oblique direction (MLO), compressed breast thickness), etc.

[0016] The display device 3 performs a predetermined display based on a display signal supplied from the image processing device 1. Examples of the display device 3 include displays such as a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), as well as a projector.

[0017] The operation device 4 generates an operation signal based on an operation by a user, such as a doctor, of the image processing device 1. Examples of the operation device 4 include a button, a keyboard, a pointing device such as a mouse, a touch panel, a remote controller, a voice input device, and any other user interface.

[0018] 1 also shows an example of the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, and an interface 13. These elements are connected via a data bus 19.

[0019] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0020] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for the processing of the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs and other information necessary for the image processing device 1 to execute each process in this embodiment.

[0021] The interface 13 acts as an interface between the image processing device 1 and external devices. For example, the interface 13 is electrically connected to the chest image generation device 2, the display device 3, and the operation device 4. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with external devices, or may be a hardware interface conforming to USB (Universal Serial Bus) or SATA (Serial AT Attachment). The interface 13 may also act as an interface with external devices such as the chest image generation device 2, the display device 3, and the operation device 4 via a communication network such as the Internet.

[0022] The configuration of the chest image processing system 100 shown in FIG. 1 is an example, and various modifications may be made.

[0023] For example, the image processing device 1 may be integrated with at least one of the chest image generating device 2, the display device 3, and the operation device 4. In another example, the image processing device 1 may include an audio output device that outputs information by audio. In yet another example, the image processing device 1 may be composed of multiple devices.

[0024] In another example, the chest image generating device 2 may be a storage device that stores pre-generated chest images of a patient. In this case, the storage device is capable of data communication with the image processing device 1, and the image processing device 1 acquires chest images of a patient designated by, for example, the operation device 4 from the storage device and executes the processing according to this embodiment. The storage device may be a server device that performs data communication with the image processing device 1 via a communication network, or may be built into the image processing device 1 as the memory 12 of the image processing device 1.

[0025] (2) Overview FIG. 2 is a diagram showing an overview of the image processing executed by the image processing device 1 in the first embodiment.

[0026] When the image processing device 1 acquires a chest image of a patient from the chest image generating device 2, it performs image analysis on the acquired chest image to detect dense areas of tissue within the breast (also referred to as "target tissue"). Here, "target tissue" refers to tissue within the breast that appears in the chest image, and examples of target tissue include mammary glands and milk ducts (including milk ducts formed as part of mammary glands). In Figure 2, the detected areas of dense areas are highlighted by rectangular frames.

[0027] Next, the image processing device 1 acquires an image (also referred to as a "first crop image") obtained by cropping a region of the chest image based on the detected region of the crowded area. The first crop image may be an image obtained by cropping the detected region of the crowded area from the chest image, or an image obtained by cropping a region smaller or larger than the detected region of the crowded area from the chest image so that the image fits within a predetermined size.

[0028] The image processing device 1 then non-rigidly deforms the first crop image so as to equalize the density of the target tissue appearing in the first crop image. Hereinafter, the image obtained by non-rigidly deforming the first crop image will also be referred to as the "second crop image." In this case, the image processing device 1 deforms the first crop image so as to equalize the degree of density of the target tissue on the image. As a result, the second crop image becomes an image obtained by non-rigidly deforming the first crop image so as to eliminate the density of the target tissue. The image processing device 1 then displays information based on the second crop image on the display device 3. This allows the examiner to more easily identify dense areas of the target tissue that are difficult to identify in the first crop image using the second crop image.

[0029] (3) Functional Blocks Figure 3 shows an example of functional blocks of the processor 11 of the image processing device 1 in the first embodiment. Functionally, the processor 11 of the image processing device 1 has a chest image acquisition unit 30, a crowded area detection unit 31, a first crop image acquisition unit 32, an image transformation unit 33, and a display control unit 34. Note that in Figure 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to other functional block diagrams described later.

[0030] The chest image acquisition unit 30 receives the chest images of the patient generated by the chest image generation device 2 from the chest image generation device 2 via the interface 13. Then, the chest image acquisition unit 30 supplies the acquired chest images to the crowded area detection unit 31, the first crop image acquisition unit 32, and the display control unit 34.

[0031] The dense area detection unit 31 detects dense areas of target tissue from the chest images supplied from the chest image acquisition unit 30 and supplies the detection results to the first crop image acquisition unit 32 .

[0032] Here, specific examples (first and second examples) of detecting a crowded area will be described in order.

[0033] In a first example of detecting dense areas, the dense area detection unit 31 infers dense areas of target tissue in a chest image based on an inference model (inference engine) that outputs detection results of dense areas of target tissue from an input chest image. The inference model described above is a model trained, for example, by machine learning, and is a model that learns the relationship between the chest image input to the inference model and the dense areas of target tissue in the image. Specifically, when a chest image is input, the inference model is trained to output an inference result of the dense areas of target tissue in the input chest image. The inference result output by the inference model may be position information indicating an area on the image surrounding the dense areas of target tissue (e.g., vertex coordinate information in the case of a bounding box), or position information indicating a representative position on the image of the dense areas of target tissue.

[0034] The inference model is trained using a training dataset including a plurality of records each of which is a combination of a sample chest image and correct answer information indicating a concentrated area of ​​correct answers in the chest image. The inference model may be a model (including a statistical model, the same applies hereinafter) employed in any machine learning method, such as a neural network or a support vector machine. Representative models of such neural networks include, for example, a Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. When the inference model is configured using a neural network, various parameters, such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter, are stored as trained parameters in memory 12 or the like.

[0035] The inference model may be trained to detect areas where target tissues are densely packed and where lesions are suspected as the dense areas. In this case, the inference model is trained using a training dataset including multiple records that are combinations of sample chest images and ground truth information that indicates areas in the chest images where target tissues are densely packed and where lesions are suspected.

[0036] The inference model may output inference results indicating multiple dense areas of the target tissue. Furthermore, when the inference model is configured using a neural network, it may output a confidence level for each detected dense area as an inference result along with location information of the detected dense areas. In this case, for example, the dense area detection unit 31 adopts an inference result for a dense area with a confidence level equal to or greater than a predetermined threshold as a detection result for a dense area to be used in subsequent processing, and supplies the detection result to the first crop image acquisition unit 32 and the display control unit 34.

[0037] In a second example of detecting a dense area, the dense area detection unit 31 calculates the degree of density of the target tissue (also referred to as "target tissue density") for each pixel in the chest image and detects the dense area based on the target tissue density. In this case, for example, the dense area detection unit 31 may detect, as a dense area, an area in which a predetermined number or more of pixels are connected, each pixel having a target tissue density equal to or greater than a predetermined threshold. The predetermined threshold and the predetermined number are stored in, for example, the memory 12. The method for detecting the target tissue and the method for calculating the target tissue density are the same as the method for detecting the target tissue and the method for calculating the target tissue density executed by the image transformation unit 33, and will be described in detail below.

[0038] Based on the detection result of the dense areas of the target tissue supplied from the dense area detection unit 31, the first crop image acquisition unit 32 performs a process of cutting out a first crop image from the chest image supplied from the chest image acquisition unit 30, and supplies the cut out first crop image to the image transformation unit 33.

[0039] In this case, for example, when the detection result of the dense tissue area of ​​the target tissue provided by the dense area detection unit 31 indicates an area within the chest image, the first crop image acquisition unit 32 acquires a first crop image by cutting out the area indicated by the detection result of the dense area. Note that in this case, the first crop image acquisition unit 32 may acquire a first crop image by cutting out an area from the chest image that is larger or smaller than the area indicated by the detection result of the dense area, so that the first crop image has a predetermined size. In this case, the first crop image acquisition unit 32 cuts out the first crop image from the chest image so that the center of the area indicated by the detection result of the dense area overlaps with the center of the first crop image. Furthermore, when the detection result of the dense tissue area of ​​the target tissue provided by the dense area detection unit 31 indicates a representative position within the chest image, the first crop image acquisition unit 32 acquires a first crop image by cutting out an area of ​​a predetermined size and shape from the chest image, with the representative position as the center of the first crop image.

[0040] The image deformation unit 33 non-rigidly deforms the first crop image supplied from the first crop image acquisition unit 32 so as to homogenize the density of the target tissue for each pixel, and supplies the second crop image obtained by deforming the first crop image to the display control unit 34. In this case, the image deformation unit 33 sequentially performs the following operations: (a) detection of the target tissue in the first crop image; (b) calculation of the target tissue density for each pixel in the first crop image; (c) calculation of a deformation vector field based on the target tissue density; and (d) deformation of the first crop image based on the deformation vector field (so-called image warping). Details of the processing by the image deformation unit 33 will be described later.

[0041] The display control unit 34 controls the display by the display device 3 based on the second crop image etc. supplied from the image transformation unit 33. In this case, the display control unit 34 generates a display signal and supplies the generated display signal to the display device 3, thereby causing the display device 3 to display the second crop image etc. Note that, based on information supplied from the chest image acquisition unit 30 and the crowded area detection unit 31, the display control unit 34 may display a chest image indicating the detected area of ​​the crowded area detected by the crowded area detection unit 31, together with the second crop image, on the display device 3. An example of a display displayed by the display device 3 based on the control of the display control unit 34 will be described later.

[0042] The components of the chest image acquisition unit 30, the crowded area detection unit 31, the first crop image acquisition unit 32, the image transformation unit 33, and the display control unit 34 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize the components. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0043] (4) Details of Image Deformation Next, each of the above-mentioned processes (a) to (d) included in the image deformation process by the image deformation unit 33 will be described.

[0044] First, "(a) Detection of target tissue in first crop image" will be described.

[0045] The image transformation unit 33 performs threshold processing on the luminance value (pixel value) associated with each pixel of the first crop image to generate a mask image indicating the presence or absence of target tissue for each pixel. In this case, the image transformation unit 33 generates a mask image in which pixels with luminance values ​​higher than a predetermined threshold have different pixel values ​​from pixels with luminance values ​​equal to or lower than the predetermined threshold. The above-mentioned threshold is stored in advance in, for example, the memory 12.

[0046] Preferably, the image transformation unit 33 performs an arbitrary filter process (edge ​​detection process) in addition to the above-described brightness threshold process. The filter used in the above-described filter process may be an arbitrary filter such as a Canny filter, a Sobel filter, or a Laplacian filter. In this case, the image transformation unit 33 acquires an image obtained by inputting the image obtained by the brightness threshold process into the above-described filter as a mask image indicating the presence or absence of the target tissue for each pixel.

[0047] Hereinafter, the mask image obtained by the process (a) will be referred to as a “target tissue mask image,” pixels in the target tissue mask image that correspond to the target tissue will be referred to as “target tissue pixels,” and pixels that do not correspond to the target tissue will be referred to as “non-target tissue pixels.” The target tissue mask image is an example of “information indicating the presence or absence of tissue in each unit region.”

[0048] Next, "(b) Calculation of target tissue density for each pixel in the first crop image" will be described.

[0049] The image transformation unit 33 calculates the target tissue density for each pixel of the first crop image using the target tissue mask image. In this case, the image transformation unit 33 selects one pixel at a time from the first crop image, detects target tissue pixels within the offset distance for each selected pixel, and determines the sum of weights according to the distance between the detected target tissue pixels and the selected pixel as the target tissue density of the selected pixel. Hereinafter, an image in which the pixel value of each pixel is the target tissue density calculated by process (b) will also be referred to as a "target tissue density image."

[0050] The granularity of the target tissue mask image and the target tissue density image does not need to be the granularity of one pixel in a chest image, but may be multiple pixels or subpixels. Instead of calculating the target tissue density, which increases as the density of the target tissue increases, the image transformation unit 33 may calculate an index, which decreases as the density of the target tissue increases, and use this index instead of the target tissue density. The same applies when the dense area detection unit 31 calculates the target tissue density when detecting a dense area. The target tissue density image is information indicating the distribution of the target tissue density for each unit area (one pixel, multiple pixels, or subpixel), and is an example of "distribution information."

[0051] Fig. 4(A) shows an example of a target tissue mask image, and Fig. 4(B) shows a target tissue density image calculated based on the target tissue mask image of Fig. 4(A). Here, for simplicity of explanation, the target tissue mask image is assumed to be a 4 × 4 image, with the pixel in the first row and second column being the target tissue pixel and the other pixels being non-target tissue pixels.

[0052] In this case, the image transformation unit 33 selects all pixels of the target tissue mask image one by one. Then, if a target tissue pixel is present within a predetermined offset distance (here, a distance of two pixels) from the selected pixel, the image transformation unit 33 adds a value corresponding to the distance between the selected pixel and the target tissue pixel as the target tissue density of the selected pixel. The "value corresponding to the distance" increases as the distance decreases and is maximized when the distance is 0 (i.e., the selected pixel and the target tissue pixel are the same pixel). In the examples of FIGS. 4(A) and 4(B), the target tissue density of the pixel in the first row and second column, which is the target tissue pixel, is the maximum value (first value), the target tissue density of the pixel one pixel away from the pixel in the first row and second column is the second highest value (second value), the target tissue density of the pixel two pixels away from the pixel in the first row and second column is the third value lower than the second value, and the target tissue density of the other pixels is 0.

[0053] In addition, if there are multiple target tissue pixels within the offset distance, the image deformation unit 33 sets the sum of the values ​​corresponding to the distance between the selected pixel and each of the multiple target tissue pixels as the target tissue density of the selected pixel.

[0054] As a result, the image deformation unit 33 can preferably calculate a target tissue density image. Note that the granularity for calculating the target tissue density (i.e., the granularity of the target tissue density image) does not need to be one pixel, but may be multiple pixels or subpixels.

[0055] Next, "(c) Calculation of deformation vector field based on target tissue density" will be described.

[0056] The image deformation unit 33 regards the region of the target tissue density image as a solvent, solves a diffusion equation assuming that a solute with a concentration equal to the target tissue density indicated by the target tissue density image is dissolved, and calculates a movement vector for each pixel to the next step (i.e., the dissolved state). This movement vector corresponds to a deformation vector, and the space where the deformation vectors are arranged in the region of the target tissue density image corresponds to a deformation vector field. Note that the deformation vector is a vector for matching an arbitrary point on the first crop image, which is the image to be deformed, with a corresponding point on the second crop image, which is the target image.

[0057] 5A shows a target tissue density image, and FIG. 5B is a diagram visualizing the deformation vectors for each pixel calculated based on the target tissue density image shown in FIG. 5A. As shown in FIGS. 5A and 5B, each deformation vector points in a direction from a position where the target tissue density is high (i.e., high concentration) to a position where the target tissue density is low. In this way, deformation vectors are calculated that diffuse the dense portions of the target tissue so that the density of the target tissue is uniform.

[0058] Here, a specific example of a method for calculating a deformation vector field will be described with reference to Figures 6(A) and 6(B). Figure 6(A) shows a target tissue density image in which the target tissue density for each pixel is indicated by a numerical value, and Figure 6(B) shows the correspondence between the target tissue density image and X-Y coordinates when the area of ​​the target tissue density image is considered to be an X-Y coordinate space. The image deformation unit 33 then solves the diffusion equation for the target tissue density "u(x, y, t)" shown in the following formula by substituting each value of "u(x, y, t)" identified by Figure 6(B) into the equation.

[0059] Note that "x" and "y" represent coordinate values ​​ranging from 0 to 1 in the XY coordinate space, "t" represents time, and "t = 0" represents the initial state corresponding to the target tissue density image. Furthermore, "C" represents a movement vector (i.e., a deformation vector) on the region of the target tissue density image. The image deformation unit 33 then fixes the value of one point (e.g., the origin) at the edge of the region of the target tissue density image. For example, the image deformation unit 33 sets "u(0,0,t) = 0." When solving the diffusion equation, the image deformation unit 33 inputs the following values ​​according to FIG. 6(B):

[0060] u(0,1,0)=5, u(1 / 3,1,0)=8, u(2 / 3,1,0)=5, u(1,1,0)=2, u(0,2 / 3,0)=5, u(1 / 3,2 / 3,0)=5, u(2 / 3,2 / 3,0)=5, u(1,2 / 3,0)=2, u(0,1 / 3,0)=2, u(1 / 3,1 / 3,0)=2, u(2 / 3,1 / 3,0)=2, u(1,1 / 3,0)=2, u(0,0,0)=0, u(1 / 3,0,0)=0, u(2 / 3,0,0)=0, u(1,0,0)=0.

[0061] The image deformation unit 33 then solves the above-mentioned diffusion equation by approximating the derivative to a differential form using a differential method, thereby enabling the image deformation unit 33 to suitably calculate a movement vector (i.e., a deformation vector) in the target tissue density image.

[0062] Next, "(d) Deformation of the first cropped image based on the deformation vector field" will be described.

[0063] The image deformation unit 33 regards the motion vector for each pixel on the target tissue density image calculated by the process (c) as a deformation vector field from the first crop image to the second crop image, and deforms the first crop image based on the deformation vector field. In this case, the image deformation unit 33 may use any image warping technique, such as BackAward Image Warping. This allows the image deformation unit 33 to obtain the second crop image by deforming the first crop image so as to equalize the target tissue density.

[0064] (5) Display Example Fig. 7 shows a first display example of a display screen that the display control unit 34 causes the display device 3 to display. The display control unit 34 outputs display information generated based on the chest image supplied from the chest image acquisition unit 30, the second cropped image supplied from the image transformation unit 33, and the like to the display device 3. The display control unit 34 transmits the display information to the display device 3, thereby causing the display screen shown in Fig. 7 to be displayed on the display device 3.

[0065] In the first display example shown in FIG. 7, the display control unit 34 provides a chest image display area 70 and a second crop image display area 71 on the display screen.

[0066] Here, the display control unit 34 displays the chest image acquired by the chest image acquisition unit 30 from the chest image generation device 2 in the chest image display area 70. Furthermore, the display control unit 34 superimposes a frame 73 on the chest image to highlight the detected area of ​​the crowded area based on the detection result of the crowded area supplied from the crowded area detection unit 31. Note that the display control unit 34 may also display a frame 73 that highlights the corresponding area of ​​the first crop image based on information supplied from the first crop image acquisition unit 32.

[0067] The display control unit 34 also displays the second cropped image supplied from the image transformation unit 33 in the second cropped image display area 71. In this second cropped image, the density of target cells such as milk ducts is reduced, improving the visibility of the target cells. Therefore, the examiner who is the viewer can check in detail in the second cropped image display area 71 whether or not there is an abnormality, such as a lesion, in areas of the chest image where the target cells are particularly dense. Furthermore, by referring to the chest image display area 70 in which the frame 73 is displayed, the viewer can also identify the area of ​​the patient's entire chest that corresponds to the second cropped image.

[0068] 8 shows a second display example of the display screen displayed by the display control unit 34 on the display device 3. In the second display example, the image processing device 1 detects multiple locations where target cells are concentrated, and generates and displays second cropped images corresponding to the detected concentrated locations. In the second display example, the display control unit 34 provides a chest image display area 70 and a second cropped image display area 71 on the display screen.

[0069] In the second display example, because the crowded area detection unit 31 detected multiple crowded areas, the display control unit 34 displays frames 73A and 73B on the chest image to highlight the areas of the crowded areas detected by the crowded area detection unit 31. In this case, the image transformation unit 33 acquires second cropped images by transforming the first cropped images corresponding to the respective detected areas of the crowded areas, and the display control unit 34 displays the second cropped images supplied from the image transformation unit 33 in association with the frames 73A and 73B indicating the corresponding detected areas of the crowded areas in the second cropped image display area 71. In the second display example, as an example, the display control unit 34 emphasizes, with a solid line frame, the outer frame of the second cropped image corresponding to the detected area of ​​the crowded area indicated by the solid line frame 73A, and emphasizes, with a dashed line frame, the outer frame of the second cropped image corresponding to the detected area of ​​the crowded area indicated by the dashed line frame 73B.

[0070] In this way, even when there are multiple detection regions of densely packed areas, the display control unit 34 can suitably present second crop images that have been modified to improve the visibility of the target cells in each of the densely packed areas. Note that, instead of the example of FIG. 8 , the display control unit 34 may accept a user selection of frames 73A and 73B on the display screen and display only the second crop image corresponding to the selected frame in the second crop image display area 71. This allows the display control unit 34 to clearly display the second crop image corresponding to the area in which the viewer is interested in the second crop image display area 71.

[0071] 9 shows a third display example of the display screen that the display control unit 34 causes the display device 3 to display. In the third display example, the image processing device 1 displays information indicating the relative position of the second cropped image in the entire chest (more specifically, breast) instead of displaying the chest images displayed in the first and second display examples. In the third display example, the display control unit 34 provides a second cropped image display area 71 and a cropped portion display area 72 on the display screen.

[0072] In this case, the display control unit 34 displays the second cropped image supplied from the image transformation unit 33 in the second cropped image display area 71, and also displays a schematic diagram of the breast divided into four regions (upper outer side, lower outer side, upper inner side, and lower inner side) in the cropped portion display area 72. In the cropped portion display area 72, the display control unit 34 emphasizes the region of the breast corresponding to the region indicated by the second cropped image, "upper outer side of the breast," and also emphasizes the region indicated by the second cropped image with a frame 74. In this case, the display control unit 34 may determine the relative position of the region indicated by the second cropped image in the entire breast using, for example, any method for determining the three-dimensional position of a region corresponding to an arbitrary image region specified in the chest image (including a method for three-dimensionally reconstructing the chest from the chest image). In another example, if information indicating the correspondence between each pixel of the chest image and the schematic diagram of the breast is stored in advance in the memory 12, the display control unit 34 may refer to the information to determine the relative position of the region indicated by the second cropped image in the entire breast.

[0073] In this way, in the third display example, even when the display control unit 34 does not display a chest image, the display control unit 34 can allow the viewer to understand the relative position of the area corresponding to the second crop image in the entire patient's chest and to check for the presence or absence of abnormalities in areas in the chest image where target cells are particularly concentrated. In each display example, the display control unit 34 may further display information about the patient or the examination on the display screen based on at least one of the patient information and the examination information received together with the chest image from the chest image generation device 2. Furthermore, for example, the display control unit 34 may determine a countermeasure based on the patient's examination information and a model generated by machine learning the correspondence between the examination information and the countermeasure for the chest image, and further display the countermeasure on the display screen. The method of determining a countermeasure is not limited to the above-described method.

[0074] (7) Processing Flow FIG. 10 is an example of a flowchart showing an outline of the processing executed by the image processing device 1 in the first embodiment.

[0075] First, the image processing device 1 acquires a chest image of a patient's chest from the chest image generating device 2 (step S11). Next, the image processing device 1 detects areas where target tissue is concentrated from the chest image acquired in step S11 (step S12).

[0076] The image processing device 1 then determines whether a dense area of ​​target tissue has been detected from the chest image (step S13). If the image processing device 1 determines that a dense area of ​​target tissue has been detected from the chest image (step S13; Yes), it acquires a first crop image cut out from the chest image based on the detection result of the dense area of ​​target tissue (step S14). The image processing device 1 then deforms the first crop image to improve the visibility of the target tissue (step S15). In this case, the image processing device 1 generates a target tissue density image, which is a map of the target tissue density in the first crop image, and acquires a second crop image by deforming the first crop image based on the target tissue density image so as to homogenize the target tissue density.

[0077] After step S15, the image processing device 1 displays at least the second cropped image on the display device 3 (step S16). In this case, for example, the image processing device 1 causes the display device 3 to display a display screen based on any one of the first to third display examples. On the other hand, if the image processing device 1 determines in step S13 that no dense area of ​​target tissue has been detected in the chest image (step S13; No), it causes the display device 3 to display the chest image (step S17). In this case, since there is no dense area of ​​target tissue that is difficult for the examiner to see, the examiner can preferably check the condition of the patient's chest based on the chest image.

[0078] 11 is a block diagram of an image processing device 1X according to a second embodiment. The image processing device 1X includes a detection unit 31X, an acquisition unit 32X, a transformation unit 33X, and a display control unit 34X. The image processing device 1X may be composed of multiple devices.

[0079] The detecting means 31X detects tissue-dense areas from chest images, and may be, for example, the dense area detecting unit 31 in the first embodiment.

[0080] The acquisition unit 32X acquires a first image obtained by cropping a region based on the crowded area from the chest image. The "region based on the crowded area" may be a region of the chest image that directly corresponds to the crowded area, or may be a region of the chest image that is expanded or reduced based on the crowded area. The acquisition unit 32X may be, for example, the first crop image acquisition unit 32 in the first embodiment.

[0081] The deformation means 33X deforms the first image based on information about tissue density in the first image. Examples of "information about tissue density in the first image" include the "target tissue mask image" and "target tissue density image" in the first embodiment. The deformation means 33X can be, for example, the image deformation unit 33 in the first embodiment.

[0082] The display control means 34X displays a second image obtained by modifying the first image on a display device. Here, the "display device" may be configured integrally with the image processing device 1X, or may be a device separate from the image processing device 1X. The display control means 34X may be, for example, the display control unit 34 in the first embodiment.

[0083] 12 is an example flowchart showing the processing procedure in the second embodiment. The detection means 31X detects tissue-dense areas from the chest image (step S21). Next, the acquisition means 32X acquires a first image by cutting out an area based on the tissue-dense areas from the chest image (step S22). The transformation means 33X transforms the first image based on information about tissue density in the first image (step S23). The display control means 34X displays a second image, which is the transformed version of the first image, on the display device (step S24).

[0084] According to the second embodiment, the image processing device 1X deforms a first image of a tissue-dense area based on information about the tissue density, and displays a second image obtained by deforming the first image, thereby allowing the user to conveniently check the state of the tissue-dense area.

[0085] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0086] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0087] [Supplementary Note 1] An image processing device comprising: a detection means for detecting dense tissue areas from a chest image; an acquisition means for acquiring a first image obtained by cutting out an area based on the dense tissue area from the chest image; a transformation means for transforming the first image based on information about the tissue density in the first image; and a display control means for displaying a second image obtained by transforming the first image on a display device. [Supplementary Note 2] The image processing device of Supplementary Note 1, wherein the display control means displays the second image and information about the area based on the dense tissue area on the display device. [Supplementary Note 3] The image processing device of Supplementary Note 2, wherein the display control means displays the second image and the chest image in which the area based on the dense tissue area is highlighted on the display device. [Supplementary Note 4] The image processing device of Supplementary Note 2, wherein the display control means displays the second image and information indicating the relative position of the area based on the dense tissue area within the entire chest on the display device. [Supplementary Note 5] The image processing device according to Supplementary Note 1, wherein the deformation means generates, as information about the congestion, distribution information indicating a distribution of the density of the tissue in a region based on the crowded portion, based on the first image, and deforms the first image based on the distribution information. [Supplementary Note 6] The image processing device according to Supplementary Note 5, wherein the deformation means generates, based on the first image, information indicating the presence or absence of the tissue for each unit region in the region based on the crowded portion, and generates the distribution information based on the information indicating the presence or absence of the tissue. [Supplementary Note 7] The image processing device according to Supplementary Note 5, wherein the deformation means deforms the first image based on the distribution information to uniform the density. [Supplementary Note 8] The image processing device according to Supplementary Note 5, wherein the deformation means calculates a deformation vector for each unit region in the region based on the crowded portion, based on the distribution information, and deforms the first image based on the deformation vector. [Supplementary Note 9] The image processing device described in Supplementary Note 1, wherein the detection means detects the crowded areas based on the chest image and an inference model, and the inference model is a machine-learned model of the relationship between the chest image and the crowded areas.[Supplementary Note 10] An image processing method in which a computer detects dense tissue areas from a chest image, obtains a first image by cutting out a region based on the dense tissue areas from the chest image, deforms the first image based on information about the tissue density in the first image, and displays a second image obtained by deforming the first image on a display device. [Supplementary Note 11] A storage medium storing a program that causes a computer to execute processes of detecting dense tissue areas from a chest image, obtains a first image by cutting out a region based on the dense tissue areas from the chest image, deforms the first image based on information about the tissue density in the first image, and displays a second image obtained by deforming the first image on a display device.

[0088] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0089] REFERENCE SIGNS LIST 1, 1X image processing device 2 chest image generating device 3 display device 4 operation device 11 processor 12 memory 13 interface 100 chest image processing system

Claims

1. a detection means for detecting tissue-dense areas from chest images; an acquisition means for acquiring a first image obtained by cutting out an area based on the crowded area from the chest image; a deformation means for deforming the first image based on information about the tissue density in the first image; a display control means for displaying a second image obtained by modifying the first image on a display device; An image processing device having:

2. The image processing device according to claim 1 , wherein the display control means displays the second image and information about the area based on the crowded portion on the display device.

3. The image processing device according to claim 2 , wherein the display control means displays the second image and the chest image in which the region based on the crowded portion is emphasized on the display device.

4. The image processing device according to claim 2 , wherein the display control means displays the second image and information indicating a relative position of the region based on the crowded portion in the entire chest on the display device.

5. 2. The image processing device according to claim 1, wherein the deformation means generates, as the information regarding the congestion, distribution information indicating a distribution of the density of the tissue in an area based on the congestion location based on the first image, and deforms the first image based on the distribution information.

6. 6. The image processing device according to claim 5, wherein the deformation means generates information indicating the presence or absence of the tissue for each unit area in the area based on the dense location based on the first image, and generates the distribution information based on the information indicating the presence or absence of the tissue.

7. The image processing device according to claim 5 , wherein the deformation means deforms the first image so as to equalize the density based on the distribution information.

8. The image processing device according to claim 5 , wherein the deformation means calculates a deformation vector for each unit area in the area based on the dense portion based on the distribution information, and deforms the first image based on the deformation vector.

9. the detecting means detects the crowded area based on the chest image and an inference model; The image processing device according to claim 1 , wherein the inference model is a model that is machine-learned based on the relationship between the chest image and the crowded areas.

10. The computer Detects areas of dense tissue from chest images; obtaining a first image obtained by cutting out an area based on the crowded area from the chest image; deforming the first image based on information about the tissue density in the first image; a second image obtained by transforming the first image is displayed on a display device; Image processing methods.

11. Detects areas of dense tissue from chest images; obtaining a first image obtained by cutting out an area based on the crowded area from the chest image; deforming the first image based on information about the tissue density in the first image; A program that causes a computer to execute a process of displaying a second image obtained by modifying the first image on a display device.