Image processing device, medical image capture system, image processing method, and image processing program

Abstract

Provided is an image processing device comprising a processor, wherein said processor: acquires a radiographic image that has been captured of a breast in a state of being compressed by a compression member; acquires a plurality of ultrasonic images of the breast, which have been captured while scanning is performed with an ultrasonic probe, in a compressed state that is considered to be the same as that in the radiographic image; and superimposes, on a radiographic image for display corresponding to the radiographic image, the shape of a region of interest based on a first position of the region of interest, which has been detected from the ultrasonic image.

Description

Image processing device, medical image capturing system, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, a medical image capturing system, an image processing method, and an image processing program.

[0002] When observing a radiographic image of a breast, an ultrasound image of the breast is also referred to. For example, Japanese Patent Application Laid-Open No. 2005-125080 describes a method for observing a region of interest in a radiographic image and a region of interest in an ultrasound image.

[0003] However, even when radiographic images and ultrasound images are taken consecutively while the breast is compressed by a compression member, and the compression state is considered to be the same, the appearance of the breast in each image differs. For example, radiographic images make it easy to extract calcifications, but they become difficult to see when the breast density is high. On the other hand, ultrasound images make it easy to extract tumors, and even when the breast density is high, they are not difficult to see like radiographic images. Because the appearance of radiographic images and ultrasound images differs, it is difficult for radiologists to compare the radiographic and ultrasound images.

[0004] The present disclosure provides an image processing device, a medical image capturing system, an image processing method, and an image processing program that enable an image reader to easily compare a radiological image with an ultrasonic cross-sectional image.

[0005] An image processing device of a first aspect of the present disclosure includes a processor that acquires a radiological image of the breast compressed by a compression member, acquires multiple ultrasound images of the breast compressed in a state that can be considered identical to the radiological image and are acquired while scanning with an ultrasound probe, and superimposes the shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

[0006] An image processing device according to a second aspect of the present disclosure is the image processing device according to the first aspect, wherein the processor also displays a scanning position image showing the scanning position of the ultrasound probe with the shape of the region of interest superimposed thereon.

[0007] An image processing device of a third aspect of the present disclosure is the image processing device of the first aspect, wherein, when a second position of a region of interest detected from a radiological image for display overlaps with a first position, the processor shifts the shape of the region of interest to be superimposed on the radiological image for display by a predetermined amount for the overlapping portion.

[0008] An image processing device of a fourth aspect of the present disclosure is the image processing device of the first aspect, in which the processor also displays an ultrasound cross-sectional image that is approximately parallel to the imaging table and is synthesized from multiple ultrasound images, with the shape of the region of interest superimposed on it.

[0009] An image processing device of a fifth aspect of the present disclosure is the image processing device of the fourth aspect, in which the processor displays an ultrasound image of a cross section that corresponds to a specified position in the ultrasound cross-sectional image and intersects with the ultrasound cross-sectional image, synthesized from multiple ultrasound images, with the shape of the region of interest based on the first position superimposed on it.

[0010] An image processing device of a sixth aspect of the present disclosure is the image processing device of the fourth aspect, in which the processor also displays an ultrasound image corresponding to a specified position in the ultrasound cross-sectional image with the shape of the region of interest based on the first position superimposed thereon.

[0011] An image processing device of a seventh aspect of the present disclosure is the image processing device of the sixth aspect, in which information representing the position of the ultrasound probe when a specified position is photographed is superimposed on an ultrasound cross-sectional image and displayed.

[0012] An image processing device of an eighth aspect of the present disclosure is the image processing device of the first aspect, in which the processor also displays an ultrasound cross-sectional image that is approximately parallel to the imaging table and is synthesized from multiple ultrasound images, with the shape of the region of interest based on a second position of the region of interest detected from the radiation image for display superimposed thereon.

[0013] An image processing device of a ninth aspect of the present disclosure is the image processing device of the fourth aspect, in which a processor displays multiple ultrasound cross-sectional images corresponding to breast depths side by side, with the shape of a region of interest based on a first position at each depth superimposed.

[0014] An image processing device of a tenth aspect of the present disclosure is the image processing device of the first aspect, wherein the radiological image is a radiological image obtained by tomosynthesis imaging, the radiological image for display is each of a plurality of radiological tomographic images reconstructed from the radiological image, and the processor displays the plurality of radiological tomographic images side by side with the shape of the region of interest superimposed based on a first position at a depth corresponding to each of the radiological tomographic images.

[0015] An image processing device of an eleventh aspect of the present disclosure is the image processing device of the tenth aspect, wherein the processor also displays the shape of the region of interest based on a second position of the region of interest detected from each of the plurality of radiation tomographic images, superimposed on each of the plurality of radiation tomographic images.

[0016] An image processing device of a twelfth aspect of the present disclosure is the image processing device of the first aspect, in which the processor also displays the shape of the region of interest based on a second position of the region of interest detected from the radiological image for display, superimposed on the radiological image for display.

[0017] An image processing device of a thirteenth aspect of the present disclosure is the image processing device of the first aspect, in which a processor displays a plurality of radiation images for display side by side, each of which has a shape of a region of interest based on a first position at a different depth in the breast superimposed thereon.

[0018] A medical imaging system according to a fourteenth aspect of the present disclosure includes the image processing device of the present disclosure, a radiation imaging device, and an ultrasound imaging device.

[0019] In addition, an image processing method of a fifteenth aspect of the present disclosure includes a processor provided in an image processing device, which acquires a radiological image of the breast in a compressed state by a compression member, acquires multiple ultrasound images of the breast in a compressed state that can be considered identical to the radiological image, which are acquired while scanning with an ultrasound probe, and superimposes the shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

[0020] In addition, the image processing program of a sixteenth aspect of the present disclosure causes a processor provided in an image processing device to acquire a radiological image taken with the breast compressed by a compression member, acquire multiple ultrasound images of the breast in a compressed state that can be considered identical to the radiological image taken while scanning with an ultrasound probe, and perform a process of superimposing the shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

[0021] According to the present disclosure, it is possible for an image reader to easily compare a radiological image of a breast captured in a compressed state with an ultrasound image.

[0022] 1 is a configuration diagram schematically illustrating an example of the overall configuration of an imaging system according to an embodiment; FIG. 2 is a side view illustrating an example of the appearance of a mammography apparatus according to an embodiment; FIG. 3 is a block diagram illustrating an example of the configuration of an ultrasound imaging apparatus according to an embodiment; FIG. 4 is a diagram illustrating an example of an ultrasound image and an ultrasound probe position image displayed on a display unit of the ultrasound imaging apparatus; FIG. 5 is a block diagram illustrating an example of the configuration of an image processing device according to an embodiment; FIG. 6 is a functional block diagram illustrating an example of the configuration of an image processing device according to an embodiment; FIG. 7 is a flowchart illustrating an example of a flow of capturing a radiation image and an ultrasound image using an imaging system according to an embodiment; FIG. 8 is a flowchart illustrating an example of an image processing flow in an image processing device according to an embodiment; FIG. 9 is a diagram illustrating an example of a display state of a radiation image, an ultrasound cross-sectional image, and a scanning position image in a state in which a first shape of a region of interest is superimposed in accordance with an embodiment; FIG. 10 is a functional block diagram illustrating an example of the configuration of an image processing device according to a first modification; FIG. 11 is a diagram illustrating an example of a display state of a radiation image, an ultrasound cross-sectional image, and a scanning position image in a state in which a first shape and a second shape of a region of interest are superimposed in accordance with a first modification; FIG. 12 is a diagram illustrating another example of a display state of a radiation image in a state in which a first shape and a second shape of a region of interest are superimposed in accordance with a first modification; Fig. 10 is a diagram showing an example of a state in which a plurality of ultrasound cross-sectional images at different depths are displayed side by side in Modification 2. Fig. 11 is a functional block diagram showing an example of the configuration of an image processing device of Modification 3. Fig. 12 is a diagram showing an example of a state in which a plurality of radiation tomographic images at different depths are displayed side by side in Modification 3. Fig. 13 is a diagram showing an example of a state in which an ultrasound cross-sectional image and an ultrasound image of a cross section intersecting the ultrasound cross-sectional image at a specified position are displayed in Modification 4.

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the embodiment.

[0024] First, an example of the overall configuration of the medical image capturing system of this embodiment will be described. Fig. 1 shows a configuration diagram illustrating an example of the overall configuration of a medical image capturing system 1 of this embodiment.

[0025] As shown in FIG. 1, the medical image capturing system 1 of this embodiment includes a radiation image capturing system 2, an ultrasonic image capturing device 16, an image processing device 18, and an image storage system 19.

[0026] First, a description will be given of the configuration of the radiation image capturing system 2. The radiation image capturing system 2 includes a mammography apparatus 10 and a console 12.

[0027] The mammography device 10 of this embodiment is a device that takes a radiographic image of a subject's breast by irradiating the breast with radiation R (e.g., X-rays). Note that the mammography device 10 may be a device that takes images of the subject's breast not only when the subject is standing (standing position), but also when the subject is sitting in a chair (including a wheelchair) or the like (seated position).

[0028] Figure 2 shows a side view of an example of the appearance of the mammography device 10 of this embodiment. Note that Figure 2 is a side view of the mammography device 10 as seen from the right side of the subject. As shown in Figure 2, the mammography device 10 includes a radiation source 36R, a radiation detector 30, an imaging table 40 disposed between the radiation source 36R and the radiation detector 30, and a compression member 34 that compresses the breast between the radiation source 36R and the imaging table 40.

[0029] The imaging stand 40 includes a control unit 20, a storage unit 22, an I / F (Interface) unit 24, an operation unit 26, and a radiation detector 30. The control unit 20 controls the overall operation of the mammography apparatus 10 in accordance with the control of the console 12. The control unit 20 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. (not shown). The ROM stores various programs in advance, including a program executed by the CPU for controlling radiographic imaging. The RAM temporarily stores various data.

[0030] The storage unit 22 stores image data of radiographic images and various other information. The storage unit 22 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. Note that, hereinafter, "image data of radiographic images" will be simply referred to as "radiographic images." Similarly, "image data of ultrasound images" will be simply referred to as "ultrasound images."

[0031] The I / F unit 24 communicates various types of information with the console 12 via wired or wireless communication. Specifically, the I / F unit 24 receives information related to the control of the mammography apparatus 10 from the console 12. The I / F unit 24 also transmits radiographic images to the console 12.

[0032] The operation unit 26 is a part provided on the imaging stand 40 or the like that can be operated by the user with the hands or feet, and is, for example, a switch, a button, a touch panel, etc. Furthermore, for example, the operation unit 26 may receive voice input from the user.

[0033] The radiation detector 30 is disposed inside the imaging table 40 and detects radiation R that has passed through the breast, which is the subject. When imaging is performed in the mammography device 10 of this embodiment, the subject's breast is positioned on the imaging surface 40A of the imaging table 40 by a user such as a doctor or technician. The imaging surface 40A, which comes into contact with the subject's breast, is made of, for example, carbon, from the viewpoints of radiation R transparency and strength.

[0034] The radiation detector 30 detects radiation R that has passed through the subject's breast and the imaging table 40, generates a radiographic image based on the detected radiation R, and outputs the generated radiographic image. The type of radiation detector 30 in this embodiment is not particularly limited, and may be, for example, an indirect conversion type radiation detector that converts radiation R into light and then converts the converted light into electric charges, or a direct conversion type radiation detector that directly converts radiation R into electric charges.

[0035] The radiation source 36R is provided in the radiation irradiation unit 36. As shown in FIG. 2, the radiation irradiation unit 36 ​​is provided on an arm 42 together with an imaging table 40 and a compression unit 46. As shown in FIG. 2, the mammography apparatus 10 of this embodiment includes the arm 42, a base 44, and a shaft 45. The arm 42 is held by the base 44 so as to be movable in the vertical direction (Z-axis direction). The shaft 45 connects the arm 42 to the base 44. The arm 42 is rotatable relative to the base 44, with the shaft 45 serving as the axis of rotation.

[0036] 2 , the compression member 34 is attached to the compression unit 46. The compression unit 46 and the arm 42 are rotatable independently relative to the base 44, with the shaft 45 as a rotation axis. In this embodiment, the shaft 45, the arm 42, and the compression unit 46 are each provided with a gear (not shown), and the arm 42 and the compression unit 46 are each connected to the shaft 45 by switching between an engaged state and a disengaged state of these gears. One or both of the arm 42 and the compression unit 46 connected to the shaft 45 rotates integrally with the shaft 45.

[0037] The compression member 34 in this embodiment is a plate-shaped member that moves up and down (in the Z-axis direction) by a compression plate driver (not shown) provided in the compression unit 46, and compresses the breast of the subject between the compression member 34 and the imaging table 40. As shown in Fig. 2, with regard to the direction of movement of the compression member 34, the direction in which the breast is compressed, in other words, the direction toward the imaging surface 40A, is referred to as the "compression direction," and the direction in which compression of the breast is released, in other words, the direction toward the radiation irradiation unit 36, is referred to as the "compression release direction."

[0038] The compression member 34 is preferably optically transparent to allow for easy alignment and confirmation of the compression state during breast compression, and is also formed from a material with excellent radiation R transparency. Furthermore, the compression member 34 is desirably formed from a material that easily transmits ultrasound waves transmitted from the ultrasound probe 55 (see FIG. 3 , described in detail below) of the ultrasound imaging device 16. Examples of materials that can be used for the compression member 34 include resins such as polymethylpentene, polycarbonate, acrylic, and polyethylene terephthalate. Polymethylpentene is particularly suitable for the compression member 34 because it has low rigidity, excellent stretchability and flexibility, and suitable values ​​for acoustic impedance, which affects the reflectivity of ultrasound, and attenuation coefficient, which affects the attenuation of ultrasound. The material constituting the compression member 34 is not limited to that described in this embodiment. For example, the compression member 34 may be a film-like material.

[0039] The compression member 34 is not limited to one that compresses the entire breast, but may also be one that compresses a portion of the breast. In other words, the compression member 34 may be smaller than the breast. For example, a compression member 34 used in so-called spot imaging, which captures a radiographic image of only the area where a lesion is present, is known as such a compression member 34.

[0040] On the other hand, the console 12 of this embodiment has the function of controlling the mammography apparatus 10 using imaging orders and various information acquired from a RIS (Radiology Information System) 5 or the like via a wireless communication LAN (Local Area Network) or the like, and instructions given by the user via an operation unit 56 or the like. The console 12 of this embodiment is, for example, a server computer.

[0041] Next, the configuration of the ultrasound imaging device 16 will be described. Fig. 3 shows a block diagram illustrating an example of the configuration of the ultrasound imaging device 16. The ultrasound imaging device 16 is a device that allows a user to capture ultrasound images of a subject's breast as an object, and is a so-called cart-type ultrasound imaging device. Note that the ultrasound imaging device 16 may also be a so-called compact or handheld type ultrasound imaging device other than the cart type.

[0042] 3, the ultrasonic imaging device 16 includes a control unit 50, a storage unit 52, an I / F unit 54, an ultrasonic probe 55, an operation unit 56, a position detection sensor 57, and a display unit 58. The control unit 50, the storage unit 52, the I / F unit 54, the ultrasonic probe 55, the operation unit 56, the position detection sensor 57, and the display unit 58 are connected via a bus 59 such as a system bus or a control bus so as to be able to exchange various information with each other.

[0043] The control unit 50 of this embodiment controls the overall operation of the ultrasound imaging device 16. The control unit 50 includes a CPU 50A, a ROM 50B, and a RAM 50C. The ROM 50B stores various programs to be executed by the CPU 50A in advance. The RAM 50C temporarily stores various data.

[0044] The captured ultrasound images and various other information are stored in the storage unit 52. Specific examples of the storage unit 52 include an HDD and an SSD.

[0045] The ultrasonic probe 55 is moved by the user along the upper surface 34A of the compression member 34 (see FIG. 2 , the surface opposite to the side that contacts the subject's breast) and scans the breast with ultrasound to obtain an ultrasound image of the breast. Specifically, when ultrasound imaging is performed, the ultrasonic probe 55 is moved by the user along the upper surface 34A of the compression member 34 with an acoustic matching member (not shown) such as echo jelly applied to the upper surface 34A of the compression member 34 or with an acoustic matching member attached to the ultrasonic probe 55.

[0046] The ultrasonic probe 55 includes a plurality of ultrasonic transducers (not shown) arranged one-dimensionally or two-dimensionally. Each ultrasonic transducer transmits ultrasonic waves based on an applied drive signal, and receives ultrasonic echoes to output a reception signal.

[0047] Each of the multiple ultrasonic transducers is composed of a vibrator made of a piezoelectric material (piezoelectric body), such as a piezoelectric ceramic, typically PZT (lead zirconate titanate), or a polymer piezoelectric element, typically PVDF (polyvinylidene difluoride), with electrodes formed on both ends. When a pulsed or continuous wave drive signal is sent to the electrodes of the vibrator to apply a voltage, the piezoelectric body expands and contracts. This expansion and contraction generates pulsed or continuous wave ultrasonic waves from each vibrator, and these ultrasonic waves are combined to form an ultrasonic beam. Furthermore, each vibrator expands and contracts upon receiving propagating ultrasonic waves, generating electrical signals. These electrical signals are output as ultrasonic reception signals and input to the main body (not shown) of the ultrasonic imaging device 16 via a cable (not shown).

[0048] The position detection sensor 57 is a sensor for detecting the position of the ultrasonic probe 55. In this embodiment, a six-axis sensor is used as the position detection sensor 57. The six-axis sensor detects the movement direction, orientation, and rotation of the ultrasonic probe 55 and further calculates the movement distance, movement speed, and the like. Specifically, a six-axis sensor is realized by combining an acceleration sensor that can detect three directions (front / back, left / right, and up / down) with a geomagnetic sensor that can detect north / south / east / west, or an acceleration sensor with a gyro sensor that can detect the speed of rotation. Note that the position detection sensor 57 is not limited to the six-axis sensor used in this embodiment. For example, a magnetic sensor such as that described in Japanese Patent Application Laid-Open No. 2011-167331 may also be used as the position detection sensor 57. Furthermore, in this embodiment, a method of using a sensor such as the position detection sensor 57 is described as a method of detecting the position of the ultrasonic probe 55, but the method of detecting the position of the ultrasonic probe 55 is not limited to the method using a sensor. For example, a marker that can detect the position in three axes in space may be attached to the ultrasonic probe 55, and the position of the ultrasonic probe 55 may be identified by analyzing an image captured of the marker. The position detection sensor 57 may be provided inside the ultrasonic probe 55 or may be provided outside the ultrasonic probe 55. Also, unlike the present embodiment, the position detection sensor 57 may be provided separately from the ultrasonic imaging device 16. In any case, as long as the position detection sensor 57 can detect the position of the ultrasonic probe 55, there are no limitations on its arrangement or configuration.

[0049] The control unit 50 identifies the current position of the ultrasound probe 55. For example, the control unit 50 identifies the position of the ultrasound probe 55 relative to the imaging surface 40A of the imaging table 40, the position of the ultrasound probe 55 relative to the upper surface 34A of the compression member 34, or the position of the ultrasound probe 55 relative to the body surface of the breast compressed by the compression member 34.

[0050] The control unit 50 also displays an ultrasound probe position image indicating the position of the ultrasound probe 55 on the display unit 58 based on the detection result of the position detection sensor 57. As shown in Fig. 4, the control unit 50 of this embodiment displays an ultrasound image U and an ultrasound probe position image H on the display unit 58. The ultrasound probe position image H is an image in which an icon H1 indicating the current position of the ultrasound probe 55 and a travel trajectory H2 of the ultrasound probe 55 are superimposed on a schematic diagram that schematically shows a breast compressed by the compression members 34. The ultrasound image U displayed on the display unit 58 together with the ultrasound probe position image H is an ultrasound image captured when the ultrasound probe 55 was located at the position of the icon H1 in the ultrasound probe position image H.

[0051] In the medical image capturing system 1 of this embodiment, it is possible to associate any position in the ultrasound image captured by the ultrasound imaging device 16 with any position pixel in the radiation image captured by the mammography device 10 based on the position of the ultrasound probe 55. Therefore, the position of the ultrasound probe 55 detected by the position detection sensor 57 is added to the captured ultrasound image and output.

[0052] The operation unit 56 is used by the user to input instructions and various information related to capturing ultrasound images, etc. The operation unit 56 is not particularly limited, and examples thereof include various switches, a touch panel, a touch pen, and a mouse. The display unit 58 displays various information and ultrasound images corresponding to the reception signals received from the ultrasound probe 55. The operation unit 56 and the display unit 58 may be integrated into one unit to form a touch panel display.

[0053] The I / F unit 54 communicates various types of information with the RIS 5 and the image storage system 19 via wireless or wired communication. The ultrasound images captured by the ultrasound imaging device 16 are transmitted to the image storage system 19 via the I / F unit 54 via wireless or wired communication.

[0054] Next, the image storage system 19 will be described. The image storage system 19 is a system that stores radiographic images captured by the radiographic imaging system 2 and ultrasound images captured by the ultrasound imaging device 16. The image storage system 19 is connected to each of the console 12 and the ultrasound imaging device 16 via wireless or wired communication. The image storage system 19 extracts images from the stored radiographic images and ultrasound images in response to requests from the console 12, the ultrasound imaging device 16, and other image reading devices (not shown), and transmits the images to the requesting devices. A specific example of the image storage system 19 is a PACS (Picture Archiving and Communication System).

[0055] Next, we will explain the image processing device 18. The image processing device 18 has a function of acquiring, from the image storage system 19, each of the radiographic images captured by the radiographic imaging system 2 and the ultrasound images captured by the ultrasound imaging device 16, and performing predetermined image processing.

[0056] Fig. 5 is a block diagram showing an example of the configuration of the image processing device 18. As shown in Fig. 5, the image processing device 18 includes a control unit 60, a storage unit 62, an I / F unit 64, an operation unit 66, and a display unit 68. The control unit 60, the storage unit 62, the I / F unit 64, the operation unit 66, and the display unit 68 are connected via a bus 69 such as a system bus or a control bus so as to be able to exchange various information with each other.

[0057] The control unit 60 of this embodiment controls the overall operation of the image processing device 18. The control unit 60 includes a CPU 60A, a ROM 60B, and a RAM 60C. The ROM 60B stores in advance various programs, including an image processing program 61 (described later), which are executed by the CPU 60A. The RAM 60C temporarily stores various data.

[0058] The storage unit 62 stores radiographic images, ultrasound images, and various other information acquired from the image storage system 19. Specific examples of the storage unit 62 include an HDD and an SSD.

[0059] The operation unit 66 is used by the user to input instructions and various information related to image processing. The operation unit 66 is not particularly limited, and examples thereof include various switches, a touch panel, a touch pen, and a mouse. The display unit 68 displays various information. The operation unit 66 and the display unit 68 may be integrated into a touch panel display.

[0060] The I / F unit 64 communicates radiological images, ultrasound images, and various types of information with the image storage system 19 via wireless or wired communication.

[0061] 6 shows a functional block diagram illustrating an example of the functions of the image processing device 18. The image processing device 18 includes a radiographic image acquisition unit 70, an ultrasound image acquisition unit 71, a first region of interest shape detection unit 74, an ultrasound cross-sectional image generation unit 76, a scanning position image generation unit 78, and a display control unit 79. As an example, in the image processing device 18 of this embodiment, the CPU 60A of the control unit 60 executes the image processing program 61, so that the CPU 60A functions as the radiographic image acquisition unit 70, the ultrasound image acquisition unit 71, the first region of interest shape detection unit 74, the ultrasound cross-sectional image generation unit 76, the scanning position image generation unit 78, and the display control unit 79.

[0062] The radiographic image acquisition unit 70 has a function of acquiring a radiographic image X. In this embodiment, as an example, a set of radiographic images X and ultrasound images U obtained by continuous imaging, the details of which will be described later, that satisfy the display conditions input by the user via the operation unit 66 is acquired. Therefore, when the radiographic image acquisition unit 70 receives the display conditions input by the user, it acquires the radiographic image X from the set of radiographic images X and ultrasound images U that corresponds to the received display conditions from the image storage system 19 via the I / F unit 64. Examples of the display conditions include identification information for identifying the subject and the breast, information indicating the date and time of imaging, and identification information assigned to each set of imaging. The radiographic image acquisition unit 70 outputs the acquired radiographic image X to the display control unit 79.

[0063] The ultrasound image acquisition unit 71 has a function of acquiring an ultrasound image U. As described above, the ultrasound image acquisition unit 71 of this embodiment acquires an ultrasound image U corresponding to the display conditions used when the radiation image acquisition unit 70 acquires a radiation image from the image storage system 19 via the I / F unit 64. In the ultrasound imaging device 16 of this embodiment, an ultrasound image U of the entire breast is acquired by repeating imaging multiple times while scanning with the ultrasound probe 55. That is, multiple ultrasound images U of the entire breast are obtained by continuously capturing images of the entire breast. Therefore, the ultrasound image acquisition unit 71 acquires multiple ultrasound images U. For simplicity of explanation, these multiple ultrasound images U may be simply referred to as "ultrasound image U." The ultrasound image acquisition unit 71 outputs the acquired ultrasound image U to the first region of interest shape detection unit 74, the ultrasound cross-sectional image generation unit 76, and the scanning position image generation unit 78.

[0064] That is, in this embodiment, of a set of radiographic image X and ultrasound image U that are captured consecutively, the radiographic image acquisition unit 70 acquires the radiographic image X, and the ultrasound image acquisition unit 71 acquires the ultrasound image U. Note that the ultrasound image U in this embodiment is provided with a probe position detection result S. The probe position detection result S is the detection result of the position detection sensor 57 of the ultrasound imaging device 16. Specifically, each ultrasound image U is provided with the probe position detection result S, which is the detection result of the position of the ultrasound probe 55 when the ultrasound image U was captured, detected by the position detection sensor 57.

[0065] The first region-of-interest shape detector 74 has a function of detecting the shape of the region of interest based on a first position of the region of interest detected from the ultrasound image U. The first position is the position of the region of interest at an arbitrary depth in the breast. In this embodiment, the ultrasound probe 55 scans the upper surface 34A of the compression member 34 and transmits ultrasound from the upper surface 34A. Therefore, attention is focused on information in the depth direction from the compression member 34 toward the imaging table 40. Therefore, the "depth" of the breast refers to the depth from the surface of the breast on the imaging table 40 that contacts the compression member 34. Specifically, the "breast depth" refers to the distance from the surface opposite the upper surface 34A of the compression member 34 toward the imaging surface 40A of the imaging table 40, in the direction from the upper surface 34A of the compression member 34 toward the imaging surface 40A of the imaging table 40. Note that the "arbitrary depth" here may be a predetermined depth or a depth corresponding to the cross section in which the area of ​​the region of interest is largest. In this embodiment, the shape of the region of interest detected from the ultrasound image is referred to as the "first shape of the region of interest."

[0066] Furthermore, the region of interest may be, for example, a region occupied by an object of interest such as a lesion, such as a tumor, but the object of interest is not limited to a lesion. However, the object of interest is an object that can be detected from each of the ultrasound image U and the radiological image using a known image analysis method or AI (artificial intelligence) technology. Note that even for the same object of interest, the region of interest detected from the ultrasound image U and the region of interest detected from the radiological image may be at least partially different. In other words, even for the same object of interest, the region of interest detected from the ultrasound image U and the region of interest detected from the radiological image may not be completely the same region.

[0067] There is no limitation on the method by which the first region of interest shape detection unit 74 detects the first shape of the region of interest based on the first position of the region of interest detected from the ultrasound image U. For example, the first region of interest shape detection unit 74 may detect the region of interest from each ultrasound image U acquired by the ultrasound image acquisition unit 71, and identify the first shape of the region of interest at an arbitrary depth by detecting the region of interest at an arbitrary depth based on the depthwise position of the region of interest detected from each ultrasound image U. Alternatively, for example, the first region of interest may be detected and its shape identified from an ultrasound cross-sectional image corresponding to a cross-section at an arbitrary depth generated by the ultrasound cross-sectional image generation unit 76 described later.

[0068] Furthermore, as a method for detecting a region of interest from an ultrasound image U, for example, a method is available in which a region of interest is detected in which brightness features, such as differences in brightness, texture, shape, etc. in the mammary gland region differ from the surrounding area, since lesions and the like have characteristic brightness compared to normal areas in the ultrasound image U. Another example is a method in which a region of interest is detected from an ultrasound image U using a learning model trained to output a region of interest in an ultrasound image U using multiple ultrasound images U including the region of interest as training data.

[0069] The first region of interest shape detection unit 74 outputs the first shape of the region of interest at an arbitrary depth to the display control unit 79 .

[0070] The ultrasound cross-sectional image generating unit 76 has a function of combining multiple ultrasound images U acquired by the ultrasound image acquiring unit 71 to generate an ultrasound cross-sectional image of a plane parallel to the imaging surface 40A of the imaging table 40. Typically, radiographic images captured by the mammography device 10 are images of a plane parallel to the imaging surface 40A of the imaging table 40. In this way, to facilitate comparison with radiographic images of a plane parallel to the imaging surface 40A of the imaging table 40, the ultrasound cross-sectional image generating unit 76 generates an ultrasound cross-sectional image of a plane parallel to the imaging surface 40A of the imaging table 40. In this embodiment, the term "substantially parallel" in the context of "a plane substantially parallel to the imaging surface 40A of the imaging table 40" refers to a plane in which bending and tilt of the compression member 34, tilt of the ultrasound probe 55, etc. can be ignored, and the plane can be considered "parallel" if these are ignored. In this embodiment, the case where the bending and tilt of the compression member 34 and the tilt of the ultrasonic probe 55 are small enough to be ignored will be described, and therefore, hereinafter, the term "parallel" may be used. Note that, if the bending and tilt of the compression member 34 and the tilt of the ultrasonic probe 55 are large enough to be unignorable, these may be corrected to make the surface "substantially parallel to the imaging surface 40A of the imaging table 40."

[0071] When the upper surface 34A of the compression member 34 is parallel to the imaging surface 40A of the imaging table 40 and the inclination of the ultrasonic probe 55 during imaging is ignored, the ultrasonic cross-sectional image synthesized from the multiple ultrasonic images U becomes a so-called C-mode image, which is an image of the C-plane of the breast. Note that a C-mode image in ultrasonic imaging is an image of a cross-sectional plane perpendicular to the ultrasonic beam at a specific fixed diagnostic distance. Note that the method by which the ultrasonic cross-sectional image generating unit 76 generates the ultrasonic cross-sectional image is not limited, and for example, a known technology can be used to reconstruct a cross-sectional image of a plane parallel to the imaging surface 40A of the imaging table 40 by reconstructing the multiple ultrasonic images U.

[0072] In this embodiment, the ultrasonic cross-sectional image is described as a two-dimensional cross-sectional image that is a plane parallel to the imaging surface 40A of the imaging table 40, but the ultrasonic cross-sectional image is not limited to a two-dimensional image. For example, the ultrasonic cross-sectional image may be a three-dimensional image that has a cross section at any depth and has a thickness in the depth direction.

[0073] The ultrasound cross-sectional image generating unit 76 of this embodiment generates an ultrasound cross-sectional image at least at an arbitrary depth where the first region-of-interest shape detecting unit 74 detects the shape of the region of interest.

[0074] The ultrasonic cross-sectional image generating unit 76 outputs the generated ultrasonic cross-sectional image to the display control unit 79 .

[0075] The scanning position image generating unit 78 has a function of generating a scanning position image that indicates the scanning position of the ultrasonic probe 55. The scanning position image generating unit 78 of this embodiment generates an image that indicates the scanning position of the ultrasonic probe 55 in capturing the ultrasonic image U, based on the probe position detection result S that is attached to the ultrasonic image U acquired by the ultrasonic image acquiring unit 71. The scanning position image is, for example, an image in which the travel trajectory of the ultrasonic probe 55 is superimposed on a schematic diagram of the breast. In this case, when capturing of the ultrasonic image U is completed, the image becomes the same as the ultrasonic probe position image H that is displayed on the display unit 58 of the ultrasonic imaging device 16. Furthermore, for example, the scanning position image may be an image in which the travel trajectory of the ultrasonic probe 55 is superimposed on an ultrasonic cross-sectional image of an arbitrary depth or an image that indicates the outline of the breast extracted from a radiation image.

[0076] The scanning position image generating unit 78 outputs the generated scanning position image to the display control unit 79 .

[0077] The display control unit 79 superimposes the first shape of the region of interest detected by the first region of interest shape detection unit 74 onto the radiographic image X acquired by the radiographic image acquisition unit 70. As an example, the display control unit 79 of this embodiment superimposes a line representing the contour (outline) of the region of interest as the first shape of the region of interest. The display control unit 79 displays the radiographic image X on which the first shape of the region of interest is superimposed on the display unit 68. Note that in this embodiment, the radiographic image X is an example of a radiographic image for display according to the present disclosure.

[0078] In addition, the display control unit 79 superimposes the first shape of the region of interest detected by the first region of interest shape detection unit 74 on the ultrasound cross-sectional image generated by the ultrasound cross-sectional image generation unit 76, and displays the ultrasound cross-sectional image with the first shape of the region of interest superimposed on it on the display unit 68.

[0079] In addition, the display control unit 79 superimposes the first shape of the region of interest detected by the first region of interest shape detection unit 74 on the scanning position image generated by the scanning position image generation unit 78, and displays the scanning position image with the first shape of the region of interest superimposed on it on the display unit 68.

[0080] Next, the operation of the image processing device 18 of this embodiment will be described with reference to the drawings, but first, a description will be given of the flow of capturing a radiographic image X and an ultrasound image U using the medical image capturing system 1. Fig. 7 shows a flowchart illustrating an example of the flow of capturing a radiographic image X and an ultrasound image U using the medical image capturing system 1 of this embodiment.

[0081] First, the user positions the subject's breast, which will be the subject of the imaging, on the imaging surface 40A of the imaging table 40. Once positioning is complete, the user issues an instruction to compress the breast using the operation unit 26. Then, in step S10 of Figure 7, the control unit 20 of the mammography device 10 begins compressing the breast with the compression members 34. Specifically, upon receiving the instruction to compress the breast, the control unit 20 moves the compression members 34 in the compression direction, causing the breast to be compressed between the compression members 34 and the imaging surface 40A of the imaging table 40.

[0082] In the next step S12, a radiographic image X of the breast is captured by the mammography device 10. Specifically, the user operates an irradiation switch included in the operation unit 26 to cause the radiation source 36R to irradiate the breast with radiation R, and the radiation detector 30 captures the radiographic image X. The radiographic image X captured by the mammography device 10 is output to the console 12, and is then output from the console 12 to the image storage system 19 at a predetermined timing and stored in the image storage system 19.

[0083] In the next step S14, the technician scans the ultrasound probe 55 of the ultrasound imaging device 16 to capture multiple ultrasound images U of the breast compressed by the compression members 34. Specifically, after capturing the radiation image X, the user applies an acoustic matching material (not shown) such as echo jelly to the upper surface 34A of the compression members 34. The user then scans the upper surface 34A of the compression members 34, covered with the acoustic matching material, with ultrasound from the ultrasound probe 55 to capture multiple ultrasound images U.

[0084] As described above, the ultrasound image U and the ultrasound probe position image H captured by the ultrasound imaging device 16 are displayed on the display unit 58 (see FIG. 4 ). The ultrasound image U captured by the ultrasound imaging device 16 is temporarily stored in the storage unit 52, and then output from the ultrasound imaging device 16 to the image storage system 19 at a predetermined timing and stored in the image storage system 19.

[0085] After capturing the ultrasound image U, the compression of the breast by the compression members 34 is released in the next step S16. Specifically, the user issues an instruction to release the compression via the operation unit 26. Upon receiving the instruction to release the compression of the breast, the control unit 20 moves the compression members 34 in the compression release direction, moving the compression members 34 away from the imaging surface 40A of the imaging table 40, thereby releasing the compression of the breast by the compression members 34. When the processing of step S16 is completed in this manner, the continuous capture of the radiographic image X and the ultrasound image U is completed. In this embodiment, the continuous capture of the radiographic image X and the ultrasound image U while the breast W remains compressed by the compression members 34, as shown in FIG. 7 , is referred to as "continuous capture." The order of capturing the radiographic image X (step S12 in FIG. 7 ) and the ultrasound image U (step S14 in FIG. 7 ) is not limited. However, from the viewpoint of shortening the breast compression time, it is preferable to capture the radiographic image X first, as in this embodiment. Furthermore, the compression force applied to the breast by the compression member 34 may be reduced to the extent that it is considered that there is no change in the compression state of the breast between capturing the radiological image X and capturing the ultrasound image U. For example, the compression force applied by the compression member 34 may be reduced to the extent that it is considered that there is no change in the state of expansion of the mammary glands of the breast.

[0086] When the continuous capturing of the radiographic image X and the ultrasonic image U is completed in this manner, image processing is performed by the image processing device 18. As an example, when the image processing device 18 of this embodiment receives the display conditions for the radiographic image X and the ultrasonic image U to be displayed, input by the user via the operation unit 66 as described above, the CPU 60A of the control unit 60 executes the image processing program 61 stored in the ROM 60B, thereby performing the image processing, an example of which is shown in Fig. 8. Fig. 8 shows a flowchart illustrating an example of the flow of image processing in the image processing device 18 of this embodiment.

[0087] First, in step S100, the radiographic image acquisition unit 70 acquires a radiographic image X that satisfies the display conditions from the image storage system 19, as described above, and outputs it to the display control unit 79. In the next step S102, the ultrasound image acquisition unit 71 acquires a plurality of ultrasound images U that satisfy the display conditions from the image storage system 19, as described above, and outputs them to the first region of interest shape detection unit 74, the ultrasound cross-sectional image generation unit 76, and the scanning position image generation unit 78. Through the processing of steps S100 and S102, the image processing device 18 acquires a set of a radiographic image X and an ultrasound image U obtained by continuous imaging.

[0088] In the next step S104, the first region of interest shape detection unit 74 detects the first shape of the region of interest based on the first position of the region of interest detected from each ultrasound image U, as described above, and outputs the detection result to the display control unit 79.

[0089] In the next step S106, the display control unit 79 causes the display unit 68 to display the radiographic image X with the first shape of the region of interest 93 superimposed thereon, as described above. Fig. 9 shows an example of the radiographic image X with the first shape of the region of interest 93 superimposed thereon, displayed on the display unit 68. The radiographic image X includes a breast image 90x and an image of the region of interest 92x, and also has the first shape of the region of interest 93 superimposed thereon. The first shape of the region of interest 93 superimposed on the radiographic image X is detected from the ultrasound image U, and therefore may differ in shape from the image of the region of interest 92x included in the radiographic image X.

[0090] In the next step S108, the ultrasound cross-sectional image generation unit 76 generates an ultrasound cross-sectional image D as described above. Here, the ultrasound cross-sectional image generation unit 76 generates an ultrasound cross-sectional image D corresponding to the cross-section at the arbitrary depth at which the first shape of the region of interest was detected in step S104. The ultrasound cross-sectional image generation unit 76 outputs the generated ultrasound cross-sectional image D to the display control unit 79.

[0091] In the next step S110, the display control unit 79 displays the ultrasound cross-sectional image D on the display unit 68 with the first shape 93 of the region of interest superimposed thereon, as described above. Fig. 9 shows an example of the ultrasound cross-sectional image D on the display unit 68 with the first shape 93 of the region of interest superimposed thereon. The ultrasound cross-sectional image D includes a breast image 90d and an image 92d of the region of interest, and also has the first shape 93 of the region of interest superimposed thereon. In this embodiment, the ultrasound cross-sectional image D displayed on the display unit 68 and the depth of the region of interest at which the first shape 93 was detected are the same, and therefore the shape of the image 92d of the region of interest included in the ultrasound cross-sectional image D is the same as the shape of the first shape 93.

[0092] In the next step S112, the scanning position image generating unit 78 generates the scanning position image P as described above. The scanning position image generating unit 78 outputs the generated scanning position image P to the display control unit 79.

[0093] In the next step S114, the display control unit 79 causes the display unit 68 to display the scanning position image P with the first shape 93 of the region of interest superimposed thereon, as described above. Fig. 9 shows an example of the scanning position image P with the first shape 93 of the region of interest superimposed thereon, displayed on the display unit 68. The traveling trajectory 94 and the first shape 93 of the region of interest are superimposed on the scanning position image P. When the processing of step S114 ends, the image processing shown in Fig. 8 ends.

[0094] As described above, in the image processing device 18 of this embodiment, as shown in Fig. 9 , the radiographic image X, the ultrasound cross-sectional image D, and the scanning position image P, each having the first shape 93 of the region of interest detected from the ultrasound image U superimposed thereon, are displayed on the display unit 68. Note that, in the example shown in Fig. 9 , the radiographic image X, the ultrasound cross-sectional image D, and the scanning position image P are displayed side by side, but the display and non-display of the images may be switched in response to a user instruction. Furthermore, whether or not the first shape 93 of the region of interest is superimposed on each of the radiographic image X, the ultrasound cross-sectional image D, and the scanning position image P may be switched in response to a user instruction.

[0095] Note that this embodiment may be modified as follows. (Modification 1) In the above embodiment, a form has been described in which a first shape 93 of a region of interest detected from an ultrasound image U is displayed in a state where it is superimposed on a radiological image X, an ultrasound cross-sectional image D, and a scanning position image P. In this modification, a form will be described in which the shape of a region of interest detected from a radiological image X is also displayed in a state where it is superimposed on at least one of the radiological image X, the ultrasound cross-sectional image D, and the scanning position image P. Note that in this modification, the shape of the region of interest detected from a radiological image X is referred to as a "second shape of the region of interest."

[0096] As shown in FIG. 10 , the image processing device 18 of this modified example further includes a second region-of-interest shape detection unit 72. The second region-of-interest shape detection unit 72 has a function of detecting the shape of a region of interest from a radiological image X acquired by the radiological image acquisition unit 70. The method by which the second region-of-interest shape detection unit 72 detects a region of interest from a radiological image X is not limited. For example, the region of interest may be detected from the radiological image X by performing image analysis of the radiological image X using existing mammography CAD (Computer-Aided Detection). Another example is a method of detecting a region of interest from a radiological image X using a learning model trained to output a region of interest within the radiological image X using multiple radiological images X containing the region of interest as training data.

[0097] The second region of interest shape detection unit 72 outputs the second shape of the region of interest detected from the radiological image X to the display control unit 79 .

[0098] The display controller 79 of this modified example further superimposes the second shape of the region of interest detected by the second region of interest shape detector 72 onto the radiographic image X acquired by the radiographic image acquisition unit 70. As an example, the display controller 79 of this modified example superimposes a line representing the outline (outline) of the region of interest as the second shape of the region of interest. The display controller 79 displays the radiographic image X on the display unit 68, with the first and second shapes of the region of interest superimposed. FIG. 11 shows an example of the radiographic image X on the display unit 68, with the first and second shapes of the region of interest 93 and 95 superimposed. It is preferable that the first and second shapes of the region of interest 93 and 95 be different in at least one of color and line type to facilitate distinction. The second shape of the region of interest 95 superimposed on the radiographic image X was detected from the radiographic image X, and is therefore identical in shape to the image 92x of the region of interest included in the radiographic image X.

[0099] The display controller 79 also superimposes the second shape of the region of interest detected by the second region of interest shape detector 72 on the ultrasound cross-sectional image D generated by the first region of interest shape detector 74. The display controller 79 displays the ultrasound cross-sectional image D on the display unit 68, with the first and second shapes of the region of interest superimposed. Fig. 11 shows an example of the ultrasound cross-sectional image D on the display unit 68, with the first and second shapes 93 and 95 of the region of interest superimposed. The second shape 95 of the region of interest superimposed on the ultrasound cross-sectional image D was detected from the radiation image X, and may therefore differ in shape from the image 92d of the region of interest included in the ultrasound cross-sectional image D.

[0100] In addition, the display control unit 79 may further superimpose the second shape of the region of interest detected by the second region of interest shape detection unit 72 on the scanning position image P generated by the scanning position image generation unit 78 and display it on the display unit 68.

[0101] The display control unit 79 may also be configured to switch between displaying and hiding the first shape 93 of the region of interest and the second shape 95 of the region of interest in response to an instruction from the user.

[0102] Furthermore, when the first shape of the region of interest 93 and the second shape of the region of interest 95 overlap, the display control unit 79 may display the overlapping portion with the shapes of the superimposed region of interest shifted by a predetermined amount. Fig. 12 illustrates an example in which the display control unit 79 displays the first shape of the region of interest 93 and the second shape of the region of interest 95 superimposed on the radiation image X with the second shape of the region of interest 95 shifted by a predetermined amount in the overlapping portion.

[0103] Note that the specific amount by which the shapes are shifted may be determined depending on, for example, the size of the radiological image X to be superimposed, the size of the region of interest, the accuracy of the image, etc. Furthermore, which of the first shape 93 of the region of interest and the second shape 95 of the region of interest is to be shifted may be determined in advance or may be determined according to a user's designation. Furthermore, for example, which of the first shape 93 of the region of interest and the second shape 95 of the region of interest is to be shifted may be determined depending on whether the image to be superimposed is the radiological image X or the ultrasound cross-sectional image D.

[0104] (Variant 2) In this variant, as shown in FIG. 13 , a form is described in which a plurality of radiological images X are displayed side by side on the display unit 68, with a first shape 93 of a region of interest based on a first position at a different depth in the breast superimposed on each of the radiological images X.

[0105] In this modification, the ultrasound cross-sectional image generating unit 76 generates an arbitrary number of ultrasound cross-sectional images D or a plurality of ultrasound cross-sectional images D corresponding to cross sections at arbitrary depths. While the example shown in FIG. 13 illustrates the generation of six ultrasound cross-sectional images D, the number of ultrasound cross-sectional images D generated by the ultrasound cross-sectional image generating unit 76 or the arbitrary depth may be, for example, a predetermined number. Furthermore, the number or depth may correspond to at least one of the thickness of the breast compressed by the compression members 34 and the depth-wise length of the region of interest in the ultrasound image U. The first region of interest shape detecting unit 74 detects the shape of the region of interest from each of the plurality of ultrasound cross-sectional images D generated by the ultrasound cross-sectional image generating unit 76 and outputs the detection result to the display control unit 79.

[0106] The display control unit 79 superimposes the shape of the region of interest detected by the first region of interest shape detection unit 74 onto each of the plurality of radiological images X, and causes the display unit 68 to display the six radiological images X.

[0107] 13 , the display controller 79 causes the display unit 68 to display the radiographic image X in a state where a first shape 93_1 of the region of interest detected from the ultrasound cross-sectional image D corresponding to the cross-section at the shallowest depth, i.e., the position closest to the compression member 34, is superimposed thereon. In this manner, the display controller 79 causes the display unit 68 to sequentially display the radiographic images X in a line with the first shapes 93_1 to 93_6 of the region of interest detected from the ultrasound cross-sectional images D corresponding to the cross-sections at shallower depths superimposed thereon. Since the six radiographic images X displayed in a line are the same image, the breast image 90x and the region of interest image 92x included in each radiographic image X are similar images, and the superimposed second shapes 95 of the region of interest are similar shapes.

[0108] By displaying in this manner, in this modified example, it is possible to easily compare the first shape 93 of the region of interest detected at different depths from the ultrasound image U with the second shape 95 of the region of interest detected from the radiation image X.

[0109] In this modification, the display control unit 79 may also display a plurality of ultrasonic cross-sectional images D generated by the ultrasonic cross-sectional image generation unit 76 side by side on the display unit 68. In the example shown in Fig. 13, as described above, the ultrasonic cross-sectional image generation unit 76 generates six ultrasonic cross-sectional images D. Fig. 14 shows a case where six ultrasonic cross-sectional images D (D_1 to D_6) are displayed side by side.

[0110] 14 , the display control unit 79 displays the ultrasound cross-sectional image D_1 on the display unit 68 in a state where a first shape 93_1 of the region of interest detected from the ultrasound cross-sectional image D_1 corresponding to the cross-section at the shallowest depth, i.e., the position closest to the compression member 34, is superimposed thereon. In this manner, the display control unit 79 sequentially displays the ultrasound cross-sectional images D_1 to D_6 side by side on the display unit 68 in a state where the first shapes 93_1 to 93_6 of the region of interest detected from each of the ultrasound cross-sectional images D_1 to D_6 corresponding to the cross-sections at shallower depths are superimposed thereon. Since the ultrasound cross-sectional images D_1 to D_6 displayed side by side are different images according to depth, the breast images 90d_1 to 90d_6 and the region of interest images 92d_1 to 92d_6 included in each ultrasound cross-sectional image D_1 to D_6 are different images, and the superimposed first shapes 93_1 to 93_6 of the region of interest often have different shapes. On the other hand, the second shape 95 of the superimposed region of interest has the same shape in each of the ultrasonic cross-sectional images D_1 to D_6.

[0111] In this manner, in the image processing device 18 of this modified example, the display control unit 79 performs display as shown in Figures 13 and 14, making it possible to easily compare the first shape 93 of the region of interest at different depths of the region of interest detected from the ultrasound image U with the second shape 95 of the region of interest detected from the radiation image X.

[0112] (Modification 3) In this modification, a case will be described in which the radiographic images X acquired by the radiographic image acquisition unit 70 are a plurality of radiographic images obtained by so-called tomosynthesis imaging.

[0113] The mammography apparatus 10 shown in FIG. 2 is capable of so-called tomosynthesis imaging. Tomosynthesis imaging is an imaging method in which radiation is irradiated onto a subject (a breast in this embodiment) at different irradiation angles, and a radiographic image X (so-called projection image) is captured for each irradiation angle. When performing tomosynthesis imaging, the arm 42 rotates to move the radiation source 36R of the radiation irradiator 36 to each of a plurality of irradiation positions with different irradiation angles (projection angles). For example, the radiation source 36R is moved to irradiation positions with different irradiation angles by a predetermined angle. In other words, the radiation source 36R is moved to positions where the angle of incidence of radiation R with respect to the detection surface of the radiation detector 30 is different. At each irradiation position, radiation R is irradiated from the radiation source 36R in response to instructions from the console 12, and a radiographic image X is captured by the radiation detector 30. In this way, tomosynthesis imaging obtains a plurality of radiographic images X corresponding to the number of irradiation positions. In addition, the mammography device capable of tomosynthesis imaging may be different from the mammography device 10 shown in Figure 2, and may be, for example, a mammography device in which a radiation source 36R is provided for each irradiation position, and the entire device is equipped with multiple radiation sources 36R.

[0114] As shown in FIG. 15, the image processing device 18 of this modification further includes a radiation tomographic image generating unit 73 in addition to the components of the image processing device 18 of the first modification (see FIG. 10).

[0115] The radiation image acquisition unit 70 of this modified example acquires a plurality of radiation images X obtained by tomosynthesis imaging, and outputs them to the radiation tomographic image generation unit 73 .

[0116] The radiological tomographic image generating unit 73 generates, from the radiological images X, a plurality of radiological tomographic images corresponding to a plurality of cross sections at different heights from the imaging surface 40A of the imaging table 40. The number of radiological tomographic images generated by the radiological tomographic image generating unit 73 may be the same as the number of ultrasonic cross-sectional images D generated by the ultrasonic cross-sectional image generating unit 76. As described above, in an ultrasonic image, "depth" refers to the depth of the breast on the imaging table 40 from the surface that contacts the compression member 34. On the other hand, in a radiological tomographic image, the imaging surface 40A of the imaging table 40 is used as a reference to identify the position of the corresponding cross section in the height direction from the imaging surface 40A. In this case, the radiological tomographic image generating unit 73 converts the height in the radiological tomographic image into the depth in the ultrasonic image (ultrasonic cross-sectional image D). The farther away from the imaging table 40, the closer to the compression member 34 the breast is. Therefore, the higher the height from the imaging surface 40A of the imaging table 40, the shallower the depth of the breast from the surface that contacts the compression member 34.

[0117] The method by which the radiation tomographic image generating unit 73 generates a radiation tomographic image from the plurality of radiation images X is not particularly limited, and any known method may be used. The radiation tomographic image generating unit 73 outputs the generated plurality of radiation tomographic images to the second region of interest shape detecting unit 72.

[0118] The second region of interest shape detection unit 72 detects the second shape 95 of the region of interest from each of the multiple radiation tomographic images generated by the radiation tomographic image generation unit 73, as described above, and outputs the detection result to the display control unit 79.

[0119] 16, in this modification, the display control unit 79 superimposes second shapes 95_1 to 95_6 of the region of interest detected from each of the multiple (six in FIG. 16) radiological tomographic images G_1 to G_6 on the display unit 68. The display control unit 79 also superimposes first shapes 93_1 to 93_6 of the region of interest detected from the ultrasound cross-sectional image D at the corresponding depth on each of the radiological tomographic images G_1 to G_6.

[0120] 16 is a radiological tomographic image at the position with the highest height from the imaging table 40, i.e., the position with the shallowest depth from the compression member 34, and radiological tomographic image G_6 is a radiological tomographic image at the position with the lowest height from the imaging table 40, i.e., the position with the deepest depth from the compression member 34. That is, in the example shown in FIG. 16 , the display control unit 79 displays the radiological tomographic images G_1 to G_6 side by side on the display unit 68 in order from the shallowest position from the compression member 34, with the second shapes 95_1 to 95_6 of the regions of interest detected from each image superimposed thereon. Note that, as shown in FIG. 16 , since each radiological tomographic image G_1 is a radiological tomographic image corresponding to a cross section at a position with different heights as described above, the second shapes 95_1 to 95_6 of the regions of interest 92x_1 to 92x_6 included in each of the radiological tomographic images G_1 to G_6 may be different.

[0121] In this way, in the image processing device 18 of this modified example, the first shape 93 of the region of interest and the second shape 95 of the region of interest can be easily compared for each of the radiation tomographic images G_1 to G_ at different heights (or depths).

[0122] 17A, the display control unit 79 may also display, on the display unit 68, an ultrasound image B of a cross section that corresponds to a specified position on the ultrasound cross-sectional image D and that intersects with the ultrasound cross-sectional image D, which is synthesized from a plurality of ultrasound images U, with the first shape 93 of the region of interest superimposed thereon. The ultrasound image B of the cross section that intersects with the ultrasound cross-sectional image D is a so-called B-mode image.

[0123] For example, in the example shown in FIG. 17( a), the display control unit 79 displays an icon 99 on the ultrasound cross-sectional image D so that it can be moved by the image reader. The icon 99 has a length corresponding to the length of contact with the compression member 34 of the ultrasound probe 55 used to capture the ultrasound image U. The image reader specifies the position of the ultrasound image cross-section to be displayed by moving the icon 99. In other words, the image reader specifies the position at which the image reader wants to view the cross-section using the icon 99. The ultrasound cross-sectional image generation unit 76 reconstructs, from the multiple ultrasound images U, an ultrasound image B of a cross-section that intersects with the ultrasound cross-sectional image D and corresponds to the position specified by the icon 99.

[0124] As shown in (a) of Figure 17, the display control unit 79 displays the reconstructed ultrasound image B alongside the ultrasound cross-sectional image D on the display unit 68, with the first shape 93 of the region of interest superimposed on it.

[0125] The orientation of the icon 99 displayed in (a) of Figure 17 is a predetermined orientation, but if it is the same as the position of the ultrasound probe 55 when the position specified by the radiologist is photographed, the ultrasound image B will be the same as the ultrasound image U obtained by photographing.

[0126] Note that the orientation of the icon 99 in the ultrasound cross-sectional image D may differ from the position of the ultrasound probe 55 when the position specified by the image interpreter was imaged. In such a case, as shown in FIG. 17B, the display control unit 79 may change the display of the icon 99 to the position of the ultrasound probe 55 when the position specified by the image interpreter was imaged. In other words, the display control unit 79 may superimpose the icon 99, which is information representing the position of the ultrasound probe 55 when the position specified by the image interpreter was imaged, on the ultrasound cross-sectional image D. In this case, as shown in FIG. 17B, the ultrasound image B is the ultrasound image U obtained by imaging. The ultrasound image U obtained by actual imaging often has higher image quality than the reconstructed ultrasound image B. Therefore, by displaying the ultrasound image U obtained by actual imaging in this manner, more accurate image interpretation can be performed.

[0127] In this way, the image processing device 18 of this modified example displays a cross-sectional ultrasound image B at a position specified by the image reader using the icon 99, which is a cross-sectional ultrasound image B that intersects with the ultrasound cross-sectional image D, making it easier to observe the area of ​​interest in the depth direction of the object of interest.

[0128] As described above, in the image processing device 18 of the above embodiment and each modified example, the radiographic image acquisition unit 70 acquires a radiographic image captured with the breast compressed by a compression member. The ultrasound image acquisition unit 71 acquires multiple ultrasound images of the breast compressed while scanning with the ultrasound probe, which can be considered to be the same as the radiographic image. The first region of interest shape detection unit 74 detects a shape 93 of the region of interest based on a first position of the region of interest from the ultrasound image. The display control unit 79 superimposes the first shape 93 of the region of interest on the radiographic image X for display. Furthermore, the display control unit 79 displays the radiographic image X with the first shape 93 of the region of interest superimposed.

[0129] Therefore, the image processing device 18 of the above embodiment and each modification can make it easier for the image reader to compare the radiological image with the ultrasonic cross-sectional image.

[0130] In the above embodiment and each modified example, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the radiographic image acquisition unit 70, the ultrasonic image acquisition unit 71, the first region of interest shape detection unit 74, the ultrasonic cross-sectional image generation unit 76, the scanning position image generation unit 78, and the display control unit 79. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0131] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0132] Examples of configuring multiple processing units with a single processor include: first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0133] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0134] In the above embodiment, the image processing program 61 is pre-stored (installed) in the control unit 60 of the image processing device 18, but this is not limiting. The image processing program 61 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The image processing program 61 may also be downloaded from an external device via a network.

[0135] The present disclosure can also be applied to programs and program products. Specifically, the image processing program 61 in each of the above embodiments may be provided as a program product. The program product includes any type of product for providing a program. For example, the program product includes a program provided via a network such as the Internet, and a non-transitory computer-readable recording medium such as a CD-ROM or DVD on which the program is stored.

[0136] Furthermore, the configurations and operations of the medical image capturing system 1, the radiographic image capturing system 2, the mammography device 10, the ultrasound image capturing device 16, and the image processing device 18 described in the above embodiment and each modification are merely examples, and it goes without saying that they can be modified according to the circumstances without departing from the spirit of the present invention. It also goes without saying that the above embodiments can be combined as appropriate.

[0137] The following supplementary note is further disclosed regarding the above-described embodiment: (Supplementary Note 1) An image processing device comprising a processor, which acquires a radiographic image captured with the breast compressed by a compression member, acquires a plurality of ultrasound images of the breast in a compressed state that can be considered identical to the radiographic image, captured while scanning with an ultrasound probe, and superimposes a shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiographic image for display corresponding to the radiographic image.

[0138] (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the processor also displays a scanning position image indicating a scanning position of the ultrasound probe with the shape of the region of interest superimposed thereon.

[0139] (Supplementary Note 3) The image processing device according to Supplementary Note 1 or Supplementary Note 2, wherein when a second position of the region of interest detected from the radiation image for display overlaps with the first position, the processor shifts a shape of the region of interest to be superimposed on the radiation image for display by a predetermined amount for the overlapping portion.

[0140] (Supplementary Note 4) The image processing device according to any one of Supplementary Note 1 to Supplementary Note 3, wherein the processor also displays an ultrasound cross-sectional image that is substantially parallel to an imaging table and is synthesized from the plurality of ultrasound images, with the shape of the region of interest superimposed thereon.

[0141] (Supplementary Note 5) The image processing device described in Supplementary Note 4, wherein the processor also displays an ultrasound image of a cross section that corresponds to a specified position in the ultrasound cross-sectional image and intersects with the ultrasound cross-sectional image, synthesized from the plurality of ultrasound images, with the shape of the region of interest based on the first position superimposed thereon.

[0142] (Supplementary Note 6) The image processing device according to Supplementary Note 4, wherein the processor also displays the ultrasound image corresponding to a specified position in the ultrasound cross-sectional image with the shape of the region of interest based on the first position superimposed thereon.

[0143] (Supplementary Note 7) The image processing device according to Supplementary Note 6, wherein information indicating the position of the ultrasound probe when the specified position is imaged is displayed superimposed on the ultrasound cross-sectional image.

[0144] (Supplementary Note 8) The image processing device described in any one of Supplementary Note 1 to Supplementary Note 7, wherein the processor also displays an ultrasound cross-sectional image that is approximately parallel to the imaging table and is synthesized from the plurality of ultrasound images, with the shape of the region of interest based on a second position of the region of interest detected from the radiation image for display superimposed thereon.

[0145] (Supplementary Note 9) The image processing device according to Supplementary Note 4, wherein the processor displays a plurality of the ultrasound cross-sectional images according to the depth of the breast side by side with the shape of the region of interest based on the first position at each depth superimposed thereon.

[0146] (Supplementary Note 10) The image processing device described in any one of Supplementary Note 1 to Supplementary Note 9, wherein the radiological image is a radiological image obtained by tomosynthesis imaging, the radiological image for display is each of a plurality of radiological tomographic images reconstructed from the radiological image, and the processor displays the plurality of radiological tomographic images side by side with the shape of the region of interest superimposed on each of the plurality of radiological tomographic images based on the first position at a depth corresponding to each of the plurality of radiological tomographic images.

[0147] (Supplementary Note 11) The image processing device according to Supplementary Note 10, wherein the processor also displays a shape of the region of interest based on a second position of the region of interest detected from each of the plurality of radiation tomographic images in a superimposed state on each of the plurality of radiation tomographic images.

[0148] (Supplementary Note 12) The image processing device according to any one of Supplementary Note 1 to Supplementary Note 11, wherein the processor also displays a shape of the region of interest based on a second position of the region of interest detected from the radiation image for display, superimposed on the radiation image for display.

[0149] (Supplementary Note 13) The image processing device according to any one of Supplementary Note 1 to Supplementary Note 12, wherein the processor displays the plurality of radiation images for display side by side with shapes of the region of interest based on the first position at different depths of the breast superimposed on each of the radiation images for display.

[0150] (Supplementary Note 14) A medical image capturing system comprising: the image processing device according to any one of Supplementary Notes 1 to 13; a radiographic image capturing device; and an ultrasonic image capturing device.

[0151] (Supplementary Note 15) An image processing method in which a processor included in an image processing device acquires a radiographic image taken with the breast compressed by a compression member, acquires a plurality of ultrasound images of the breast in a compressed state that can be considered identical to the radiographic image, taken while scanning with an ultrasound probe, and superimposes a shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiographic image for display corresponding to the radiographic image.

[0152] (Supplementary Note 16) An image processing program for causing a processor included in an image processing device to execute the following processes: acquire a radiographic image captured with the breast compressed by a compression member; acquire a plurality of ultrasound images of the breast in a compressed state that can be considered identical to the radiographic image, captured while scanning with an ultrasound probe; and superimpose a shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiographic image for display corresponding to the radiographic image.

[0153] The disclosure of Japanese Patent Application No. 2023-170673, filed on September 29, 2023, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising: a processor which acquires a radiological image of the breast compressed by a compression member, acquires a plurality of ultrasound images of the breast in a compressed state which can be regarded as identical to the radiological image, the images being acquired while scanning with an ultrasound probe, and superimposes a shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

2. The image processing device according to claim 1, wherein the processor also displays a scanning position image showing the scanning position of the ultrasound probe with the shape of the region of interest superimposed thereon.

3. The image processing device according to claim 1, wherein, when a second position of the region of interest detected from the radiation image for display overlaps with the first position, the processor shifts the shape of the region of interest to be superimposed on the radiation image for display by a predetermined amount for the overlapping portion.

4. The image processing device according to claim 1, wherein the processor also displays an ultrasound cross-sectional image, which is synthesized from the multiple ultrasound images and is approximately parallel to the imaging table, with the shape of the region of interest superimposed thereon.

5. The image processing device according to claim 4, wherein the processor also displays an ultrasound image of a cross section that corresponds to a specified position in the ultrasound cross-sectional image and that intersects with the ultrasound cross-sectional image, synthesized from the multiple ultrasound images, with the shape of the region of interest based on the first position superimposed thereon.

6. The image processing device according to claim 4, wherein the processor also displays the ultrasound image corresponding to a specified position in the ultrasound cross-sectional image with the shape of the region of interest based on the first position superimposed thereon.

7. The image processing device according to claim 6, wherein information indicating the position of said ultrasonic probe when said specified position was imaged is displayed superimposed on said ultrasonic cross-sectional image.

8. The image processing device according to claim 1, wherein the processor also displays an ultrasound cross-sectional image, which is approximately parallel to the imaging table and which is synthesized from the multiple ultrasound images, with the shape of the region of interest based on a second position of the region of interest detected from the radiation image for display superimposed thereon.

9. The image processing device according to claim 4, wherein the processor displays a plurality of the ultrasound cross-sectional images corresponding to the depth of the breast side by side with the shape of the region of interest based on the first position at each depth superimposed thereon.

10. The image processing device according to claim 1, wherein the radiation image is a radiation image obtained by tomosynthesis imaging, the radiation image for display is each of a plurality of radiation tomographic images reconstructed from the radiation image, and the processor displays the plurality of radiation tomographic images side by side with the shape of the region of interest superimposed based on the first position at the corresponding depth for each of the radiation tomographic images.

11. The image processing device according to claim 10, wherein the processor also displays a shape of the region of interest based on a second position of the region of interest detected from each of the plurality of radiation tomographic images in a superimposed state on each of the plurality of radiation tomographic images.

12. The image processing device according to claim 1, wherein the processor also displays a shape of the region of interest based on a second position of the region of interest detected from the radiation image for display, superimposed on the radiation image for display.

13. The image processing device according to claim 1, wherein the processor displays the plurality of radiation images for display side by side with the shape of the region of interest based on the first position at a different depth of the breast superimposed on each of the radiation images for display.

14. A medical image capturing system comprising: an image processing device according to any one of claims 1 to 13; a radiation image capturing device; and an ultrasound image capturing device.

15. An image processing method comprising the steps of: acquiring a radiological image taken with the breast compressed by a compression member; acquiring a plurality of ultrasound images of the breast in a compressed state that can be regarded as identical to the radiological image, the images being taken while scanning with an ultrasound probe; and superimposing a shape of a region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

16. An image processing program for causing a processor included in an image processing device to execute the following processes: acquire a radiological image taken with the breast compressed by a compression member; acquire multiple ultrasound images of the breast in a compressed state that can be considered identical to the radiological image, taken while scanning with an ultrasound probe; and superimpose a shape of the region of interest based on a first position of the region of interest detected from the ultrasound image onto a radiological image for display corresponding to the radiological image.

Images