Image processing device, image processing method, and image processing program
The image processing device enhances interpolated image generation accuracy by separating and processing frame images into noise-reduced components, estimating displacement, and generating images with improved precision, addressing noise-related inaccuracies.
Patent Information
- Application Number
- JP2023082221
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-28
AI Technical Summary
Existing image processing devices struggle to accurately generate interpolated images when noise is present in frame images, leading to inaccuracies in motion vector detection and interpolated image generation.
An image processing device that acquires and processes multiple frame images, separates them into first and second component images, reduces noise in the first component images, estimates the displacement amount based on these components, and generates interpolated images using these components to improve accuracy.
The device can generate interpolated images with high accuracy even when noise is present, reducing the need for repeated radiation exposure and improving image quality in radiographic images.
Smart Images

Figure 2024165745000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program, and more particularly to an image processing device, an image processing method, and an image processing program that increase the number of frame images included in a moving image. [Background technology]
[0002] 2. Description of the Related Art Conventionally, image processing devices that increase the number of frame images included in a moving image are known (see, for example, Patent Document 1).
[0003] The above-mentioned Patent Document 1 discloses an interpolation frame generating device including a frame memory, a motion vector detection unit, and a frame frequency conversion memory. In the configuration disclosed in the above-mentioned Patent Document 1, the motion vector detection unit detects a motion vector between frames based on a current frame of an input image signal and a frame one frame before the current frame generated by the frame memory. The above-mentioned Patent Document 1 also discloses a configuration in which an interpolation frame is generated between a current frame and a frame one frame before the current frame based on a motion vector detected by the motion vector detection unit. The above-mentioned Patent Document 1 also discloses a configuration in which a frame frequency conversion memory generates an image signal having a frequency twice the frame frequency of the input image signal by alternately reading out the current frame and the interpolation frame. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-57866 A Summary of the Invention [Problem to be solved by the invention]
[0005] Here, as disclosed in the above Patent Document 1, when a motion vector is calculated based on a plurality of frames (a plurality of frame images) included in an image signal to generate an interpolated frame (an interpolated image), if the plurality of frame images contain noise, there is a problem that the motion vector cannot be detected with high accuracy. If the motion vector cannot be detected with high accuracy, there is a problem that the interpolated image cannot be generated with high accuracy.
[0006] The present invention has been made to solve the above-mentioned problems, and one object of the present invention is to provide an image processing device, an image processing method, and an image processing program that are capable of generating an interpolated image with high accuracy even when the interpolated image is generated based on multiple frame images containing noise. [Means for solving the problem]
[0007] An image processing device in a first aspect of the present invention is an image processing device that processes a plurality of frame images that constitute a moving image, and includes an image acquisition unit that acquires the plurality of frame images, and an image processing unit that processes the plurality of frame images acquired by the image acquisition unit, and is configured to acquire a first component image containing a first component in which noise has been reduced from each of two frame images that have different shooting times among the plurality of frame images, acquire a second component image containing a second component different from the first component from one of the two frame images, estimate an amount of displacement of the first component based on the two first component images, and generate an interpolated image of a time different from the two frame images based on the amount of displacement of the first components of the two first component images and the second component image.
[0008] An image processing device in a second aspect of the present invention is an image processing device that processes a plurality of frame images that constitute a radiographic image of a moving image, and includes an image acquisition unit that acquires the radiographic image that is a moving image, and an image processing unit that processes the radiographic image acquired by the image acquisition unit, and the image processing unit is configured to acquire a first component image containing a first component in which noise has been reduced from each of two frame images that have different shooting times among the plurality of frame images included in the radiographic image, acquire a second component image containing a second component different from the first component from one of the two frame images, estimate an amount of displacement of the first component based on the two first component images, and generate a radiographic interpolation image that is a radiographic image of a time different from the two frame images based on the amount of displacement of the first components of the two first component images and the second component image.
[0009] An image processing method according to a third aspect of the present invention is an image processing method for processing a plurality of frame images that constitute a moving image, and includes the steps of: acquiring a first component image containing a first component in which noise has been reduced from each of two frame images that have been captured at different times among the plurality of frame images; acquiring a second component image containing a second component different from the first component from one of the two frame images; estimating an amount of displacement of the first component based on the two first component images; and generating an interpolated image of a time different from the two frame images based on the amount of displacement of the first components of the two first component images and the second component image.
[0010] An image processing program in a fourth aspect of the present invention is an image processing program executed in an image processing device that processes a plurality of frame images that constitute a moving image, and causes the image processing device to execute the following processes: a process of acquiring a first component image containing a first component in which noise has been reduced from each of two frame images that were captured at different times among the plurality of frame images; a process of acquiring a second component image containing a second component different from the first component from one of the two frame images; a process of estimating the amount of displacement of the first component based on the two first component images; and a process of generating an interpolated image of a time different from the two frame images based on the amount of displacement of the first components of the two first component images and the second component image. Effect of the Invention
[0011] In the image processing device according to a first aspect of the present invention, the image processing method according to a third aspect, and the image processing program according to a fourth aspect, a plurality of first component images including a first component with reduced noise are obtained from two frame images captured at different times among a plurality of frame images, and a second component image including a second component different from the first component is obtained from one of the two frame images. In addition, the amount of displacement of the first component is estimated based on the two first component images, and an interpolated image of a time different from that of the two frame images is generated based on the amount of displacement of the first component of the two first component images and the second component image. As a result, the amount of displacement of the first component is estimated based on the first component image with reduced noise, so that the amount of displacement of the first component can be estimated more accurately than a configuration in which the amount of displacement of the first component is estimated based on an image in which noise is not reduced. Therefore, it is possible to accurately estimate the amount of displacement of the first component, so that the accuracy of the amount of displacement of the first component of the two first component images and the interpolated image generated based on the second component image can be improved. As a result, it is possible to provide an image processing device, an image processing method, and an image processing program that are capable of generating an interpolated image with high accuracy even when the interpolated image is generated based on multiple frame images containing noise.
[0012] Also, an image processing device according to a second aspect includes an image processing unit that acquires a plurality of first component images including a first component with reduced noise from two frame images with different capture times among a plurality of frame images included in a radiation image, acquires a second component image including a second component different from the first component from one of the two frame images, estimates a displacement amount of the first component based on the two first component images, and generates a radiation interpolation image, which is a radiation image at a time different from the two frame images, based on the displacement amount of the first component of the two first component images and the second component image. Here, in order to reduce the radiation exposure amount when capturing a radiation image of a moving image, the radiation amount to be irradiated is reduced. In this case, the signal-to-noise ratio (SN ratio) of the radiation image decreases, resulting in an image containing more noise. Therefore, by configuring as described above, the displacement amount of the first component is estimated based on the first component image with reduced noise acquired from the radiation image, so that the displacement amount of the first component can be estimated more accurately than a configuration in which the displacement amount of the first component is estimated based on a radiation image in which noise is not reduced. Therefore, it is possible to accurately estimate the displacement amount of the first component, and therefore it is possible to improve the displacement amount of the first component of two first component images and the accuracy of the radiation interpolation image generated based on the second component image. Therefore, it is possible to provide an image processing device that can reduce the radiation dose to reduce the radiation exposure amount and improve the accuracy of the radiation interpolation image by reducing the number of times radiation is irradiated and imaging is performed by thinning-out imaging, thereby reducing the radiation exposure amount. Note that thinning-out imaging means imaging in which imaging by irradiating radiation is not performed, and the time interval between capturing frame images is made longer than a predetermined frame rate. Therefore, when thinning-out imaging is performed, a moving image is captured at a frame rate lower than a predetermined frame rate. [Brief description of the drawings]
[0013] [Figure 1] 1 is a block diagram showing the configuration of an image processing device according to an embodiment, and the configuration of an X-ray imaging device that captures frame images to be processed by the image processing device. [Diagram 2] 1 is a schematic diagram showing a configuration of an X-ray imaging device that captures frame images to be processed by an image processing device according to an embodiment. [Diagram 3] FIG. 11 is a schematic diagram for explaining a configuration for capturing a moving image according to a comparative example. [Figure 4] FIG. 2 is a schematic diagram for explaining a configuration for capturing a moving image according to an embodiment. [Diagram 5] 1 is a schematic diagram for explaining a configuration in which an image processing unit according to an embodiment generates an interpolated image by using a motion vector; [Figure 6] FIG. 11 is a schematic diagram for explaining a configuration in which an image processing unit according to one embodiment generates an interpolated image using a first trained model. [Figure 7] 10 is a schematic diagram for explaining a configuration in which an image processing unit according to an embodiment generates an interpolated image and stores an interpolated moving image that is a moving image including the interpolated image. FIG. [Figure 8] 1 is a schematic diagram for explaining a configuration in which an image processing unit according to an embodiment generates a first component image and a second component image from a frame image. FIG. [Figure 9] 1 is a schematic diagram for explaining a configuration in which an image processing unit according to an embodiment generates an interpolated image by interpolation. FIG. [Figure 10] 1 is a schematic diagram for explaining a configuration in which an image processing device according to an embodiment generates an interpolated image by extrapolation. [Figure 11] 11 is a flowchart for explaining a process in which an image processing unit according to an embodiment stores an interpolated moving image, which is a moving image including an interpolated image. [Figure 12] FIG. 13 is a block diagram showing the configuration of an X-ray imaging apparatus including an image processing unit according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
[0015] As shown in FIG. 1, the image processing device 100 includes an image acquisition unit 1 and an image processing unit 2. The image processing device 100 also includes a control unit 3, a storage unit 4, a display unit 5, and an input receiving unit 6. The image processing device 100 is an image processing device that processes a plurality of frame images 10 constituting a moving image input from an X-ray imaging device 200. The frame images 10 include radiation images. In this embodiment, the frame images 10 are radiation images of a moving image captured at a predetermined frame rate. The predetermined frame rate is, for example, 7.5 fps (frames per second). The radiation images are, for example, X-ray images.
[0016] The image acquisition unit 1 is configured to acquire a frame image 10. The image acquisition unit 1 is, for example, an input / output interface.
[0017] The image processing unit 2 is configured to process the frame image 10 acquired by the image acquisition unit 1. The image processing unit 2 is configured with a processor such as a CPU (Central Processing Unit), an FPGA (field-programmable gate array), or a circuit, and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory).
[0018] The image processing unit 2 also includes functional blocks, namely, a component image acquisition unit 2a, a first component intermediate image generation unit 2b, a displacement amount estimation unit 2c, a converted image generation unit 2d, an interpolated image generation unit 2e, and a moving image generation unit 2f. The component image acquisition unit 2a, the first component intermediate image generation unit 2b, the displacement amount estimation unit 2c, the converted image generation unit 2d, the interpolated image generation unit 2e, and the moving image generation unit 2f are configured in terms of software as functional blocks realized by the image processing unit 2 executing an image processing program 20 stored in the storage unit 4. The component image acquisition unit 2a, the first component intermediate image generation unit 2b, the displacement amount estimation unit 2c, the converted image generation unit 2d, the interpolated image generation unit 2e, and the moving image generation unit 2f may be configured by hardware separate from each other by providing a dedicated processor (processing circuit). The functions of the component image acquisition unit 2a, the first component intermediate image generation unit 2b, the displacement estimation unit 2c, the converted image generation unit 2d, the interpolated image generation unit 2e, and the moving image generation unit 2f will be described in detail later.
[0019] The control unit 3 is configured to control each unit of the image processing device 100 by executing various programs (not shown) stored in the storage unit 4. The control unit 3 is configured, for example, by a processor such as a CPU or a circuit, and memories such as a ROM and a RAM.
[0020] The storage unit 4 is configured to store an image processing program 20. The storage unit 4 is also configured to store a frame image 10, a first component image 11, a second component image 12, an interpolated image 13, a first component intermediate image 14, and a converted image 15, which will be described later. The storage unit 4 is also configured to store a first trained model 21 and a second trained model 22, which will be described later. The storage unit 4 is also configured to store a filter 23 used in a filter process, which will be described later. The storage unit 4 also stores various programs executed by the control unit 3. The storage unit 4 includes, for example, a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The interpolated image 13 is an example of a "radiation interpolated image" in the claims.
[0021] The image processing program 20 is an image processing program executed in the image processing device 100 (image processing unit 2) which processes a plurality of frame images 10 constituting a moving image. Details of the image processing performed by the image processing unit 2 using the image processing program 20 will be described later.
[0022] The display unit 5 is configured to display the moving images generated by the image processing unit 2. The display unit 5 is, for example, a display device such as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor.
[0023] The input receiving unit 6 is configured to receive an operation input from an operator and includes input devices such as a mouse and a keyboard.
[0024] The X-ray imaging device 200 is configured to capture X-ray images, which are moving images to be presented to an operator such as a doctor during treatment of a subject 90 (see FIG. 2) using a catheter 91 (see FIG. 4). The X-ray imaging device 200 includes an X-ray source 201, an X-ray detector 202, and an X-ray imaging device control unit 203.
[0025] The X-ray source 201 is configured to irradiate the subject 90 (see FIG. 2) with X-rays. The X-ray source 201 includes, for example, an X-ray generating device such as an X-ray tube.
[0026] The X-ray detector 202 is configured to detect X-rays irradiated from the X-ray source 201. The X-ray detector 202 is, for example, a FPD (Flat Panel Detector), and detects X-rays that have passed through the subject 90.
[0027] The X-ray imaging apparatus control unit 203 is configured to control each unit of the X-ray imaging apparatus 200. Specifically, the X-ray imaging apparatus control unit 203 is configured to control X-ray irradiation by the X-ray source 201, such as starting and stopping X-ray irradiation, and control detection by the X-ray detector 202. The X-ray imaging apparatus control unit 203 is configured with a processor such as a CPU, FPGA, or circuitry, and memories such as a ROM and RAM.
[0028] (Configuration of X-ray equipment) As shown in FIG. 2, the X-ray source 201 is held on one side of the arm 204 so as to face the X-ray detector 202. Therefore, in this embodiment, the X-rays emitted from the X-ray source 201 are directed toward the X-ray detector 202. In addition, in this embodiment, the arm 204 is disposed so that the top plate 207a is disposed between the X-ray source 201 and the X-ray detector 202. Therefore, the X-ray source 201 is configured to irradiate the subject 90 with X-rays. In the example shown in FIG. 2, the up-down direction is the Z direction, the up direction is the Z1 direction, and the down direction is the Z2 direction. In addition, two directions perpendicular to each other in a horizontal plane perpendicular to the Z direction are the X direction and the Y direction. One side of the X direction is the X1 direction, and the other side is the X2 direction. In addition, one side of the Y direction is the Y1 direction, and the other side is the Y2 direction.
[0029] The X-ray detector 202 is held on the other side of the arm 204 so as to face the X-ray source 201. Therefore, the X-ray detector 202 is configured to detect X-rays irradiated from the X-ray source 201.
[0030] Moreover, the arm 204 is configured to hold the X-ray source 201 and the X-ray detector 202. In this embodiment, the arm 204 has an arc shape, and holds the X-ray source 201 and the X-ray detector 202 at one end and the other end. The arm 204 is a so-called C-arm.
[0031] The arm driving mechanism 205 is configured to drive the arm 204. The arm 204 is configured to be driven by the arm driving mechanism 205 so that the X-ray source 201 and the X-ray detector 202 can be disposed at various positions.
[0032] The X-ray detector moving mechanism 206 is provided on the arm 204, and is configured to move the X-ray detector 202 forward or backward in the direction of the X-ray irradiation axis 50. The X-ray detector moving mechanism 206 includes, for example, a linear motion mechanism.
[0033] The bed 207 is configured so that the subject 90 can be placed thereon. The bed 207 includes a top board 207a and a top board moving mechanism 207b. The subject 90 is placed on the top board 207a.
[0034] The top board moving mechanism 207b is configured to move the top board 207a in the Z direction. The top board moving mechanism 207b is also configured to translate the top board 207a in the XY plane. The top board moving mechanism 207b includes, for example, a linear motion mechanism that moves the top board 207a in the X direction, a linear motion mechanism that moves the top board 207a in the Y direction, and a linear motion mechanism that moves the top board 207a in the Z direction.
[0035] The arm position changing mechanism 208 is configured to move the arm driving mechanism 205 and move the arm 204 together with the arm driving mechanism 205 to a desired imaging position. This allows the X-ray imaging device 200 to image the subject 90 from various imaging positions.
[0036] (Video capture by comparative example) Next, with reference to FIG. 3, a description will be given of capturing moving images according to a comparative example. In the comparative example shown in FIG. 3, five images, a first frame image 10a to a fifth frame image 10e, are captured as frame images 10 according to a predetermined frame rate. The predetermined frame rate is, for example, 7.5 fps. In the example shown in FIG. 3, the frame images 10 are illustrated arranged according to the capturing time. The first frame image 10a is the oldest frame image 10, and the fifth frame image 10e is the newest frame image 10. Note that in the example shown in FIG. 3, a catheter 91 is captured in each frame image 10.
[0037] In the comparative example, when capturing the first frame image 10a, the second frame image 10b, the third frame image 10c, the fourth frame image 10d, and the fifth frame image 10e, X-rays are irradiated from the X-ray source 201. Therefore, the higher the frame rate at which the frame images 10 are captured, the more times X-rays are irradiated, and the radiation exposure of the subject 90 (see FIG. 2) increases.
[0038] (Capturing moving images according to this embodiment) Therefore, in this embodiment, in order to reduce the radiation exposure of the subject 90, thinning-out imaging is performed as shown in FIG. 4. Thinning-out imaging means that when the first frame image 10a to the fifth frame image 10e are captured according to a predetermined frame rate, X-rays are not irradiated every time, and imaging that irradiates X-rays is not performed, and the time interval for capturing the frame images 10 is made longer than the predetermined frame rate. Note that in the example shown in FIG. 4, the frame images 10 are also illustrated arranged according to the imaging time. Also in the example shown in FIG. 4, the first frame image 10a is the oldest frame image 10, and the fifth frame image 10e is the newest frame image 10.
[0039] In the example shown in FIG. 4, X-rays are irradiated when the first frame image 10a, the third frame image 10c, and the fifth frame image 10e are captured. In addition, X-rays are not irradiated at the timing corresponding to the first frame 10f between the first frame image 10a and the third frame image 10c, and the second frame 10g between the third frame image 10c and the fifth frame image 10e. This can reduce the amount of radiation exposure of the subject 90 (see FIG. 2). Note that in FIG. 4, the first frame 10f and the second frame 10g are illustrated by dashed lines for the sake of convenience, but since X-rays are not irradiated in these frames, the subject is not captured.
[0040] Here, when thinning-out imaging is performed, there are frames that are not irradiated with X-rays, and the captured moving images have a frame rate lower than the predetermined frame rate. In this case, the moving images displayed on the display unit 5 (see FIG. 1) become less smooth, which causes problems when the operator treats the subject 90 (see FIG. 2).
[0041] Therefore, in this embodiment, the image processing unit 2 (see FIG. 1) is configured to perform thinning-out shooting and to interpolate the frame images 10 so that the moving image to be output has a predetermined frame rate.
[0042] Specifically, the image processing unit 2 is configured to generate an interpolated image 13 (see FIG. 5) to interpolate the frame image 10. In this embodiment, the image processing unit 2 is configured to generate the interpolated image 13 using a motion vector 30a (see FIG. 5) or to generate the interpolated image 13 using a first trained model 21 (see FIG. 6).
[0043] (Generating an Interpolated Image Using Motion Vectors) First, with reference to Fig. 5, a configuration in which the image processing unit 2 (see Fig. 1) generates an interpolated image 13 using a motion vector 30a will be described. In the example shown in Fig. 5, the image processing unit 2 generates an interpolated image 13 corresponding to the image of a frame (first frame 10f (see Fig. 4)) between the first frame image 10a and the third frame image 10c from the first frame image 10a and the third frame image 10c.
[0044] The image processing unit 2 acquires a motion vector 30a of the catheter 91 from the first frame image 10a and the third frame image 10c. Specifically, the image processing unit 2 sets a plurality of feature points on the catheter 91, and acquires a motion vector 30a for each feature point based on the position of each of the plurality of feature points in each frame image 10. That is, the image processing unit 2 acquires the motion vector 30a by the Lucas-Kanade method. Note that the image processing unit 2 may be configured to acquire the motion vector 30a by template matching.
[0045] Then, the image processing unit 2 generates an interpolated image 13 based on the catheter 91 shown in the first frame image 10a or the catheter 91 shown in the third frame image 10c and the acquired motion vector 30a. For example, as shown in FIG. 5, when generating an interpolated image 13 as a frame image 10 between the first frame image 10a and the third frame image 10c, the interpolated image 13 is generated using a motion vector that has the same direction as the motion vector 30a acquired based on the first frame image 10a and the third frame image 10c and is half the size. The size of the motion vector 30a when generating the interpolated image 13 is set according to the position and number of the interpolated images 13 to be inserted. For example, when inserting two interpolated images 13 between the first frame image 10a and the third frame image 10c, the size of the motion vector 30a is set to 1 / 3 when generating the interpolated image 13 to be inserted on the first frame image 10a side. Moreover, when generating the interpolated image 13 to be inserted on the third frame image 10c side, the size of the motion vector 30a is made 2 / 3.
[0046] (Generating an interpolated image using a trained model) Next, with reference to Fig. 6, a configuration will be described in which the image processing unit 2 (see Fig. 1) generates an interpolated image 13 using the first trained model 21. In the example shown in Fig. 6, the image processing unit 2 also generates an interpolated image 13 corresponding to the image of the frame (first frame 10f (see Fig. 4)) between the first frame image 10a and the third frame image 10c from the first frame image 10a and the third frame image 10c.
[0047] The first trained model 21 is generated by training a learning model that is, for example, a convolutional neural network (CNN) or includes a convolutional neural network in part. Specifically, the first trained model 21 is generated by training to generate an image in which an image component is displaced based on a plurality of images. When generating the first trained model 21, an image with a frame rate sufficiently smaller than the respiration and / or heartbeat of the subject 90 (FIG. 2) is used as a moving image used for training so that the displacement of the image component due to the respiration and / or heartbeat of the subject 90 (FIG. 2) can be accurately estimated. Note that being sufficiently smaller than the respiration and / or heartbeat of the subject 90 means that one frame image does not include multiple phases of the respiration and / or heartbeat of the subject 90.
[0048] The image processing unit 2 generates an interpolated image 13, which is a frame image 10 between the first frame image 10a and the third frame image 10c, by inputting the first frame image 10a and the third frame image 10c, which show the catheter 91, into the first trained model 21.
[0049] Here, when imaging the subject 90 with X-rays, in order to reduce the amount of radiation exposure to the subject 90, the dose of X-rays irradiated from the X-ray source 201 (see FIG. 1) is set as small as possible within the range in which imaging is possible. In this case, the signal-to-noise ratio decreases, and the frame image 10 becomes an image containing noise. In both the configuration in which the image processing unit 2 generates the interpolated image 13 using the motion vector 30a (see FIG. 5) and the configuration in which the image processing unit 2 generates the interpolated image 13 using the first trained model 21, if noise is included in the frame image 10, the accuracy of the interpolated image 13 decreases. Specifically, when the image processing unit 2 generates the interpolated image 13 using the motion vector 30a, it becomes difficult to accurately obtain the displacement of the feature point of the catheter 91 due to noise, so the accuracy of the motion vector 30a decreases. Therefore, the accuracy of the generated interpolated image 13 decreases. In addition, when the image processing unit 2 generates the interpolated image 13 using the first trained model 21, artifacts due to noise occur in the generated interpolated image 13, so the accuracy of the interpolated image 13 decreases.
[0050] (Generation of Interpolated Images and Generation of Interpolated Moving Images) Therefore, in this embodiment, the image processing unit 2 is configured to be able to suppress a decrease in accuracy of the interpolated image 13 even when noise is contained in the frame image 10. The details of the configuration in which the image processing unit 2 (see FIG. 1) generates the interpolated image 13 based on the frame image 10 containing noise will be described with reference to FIGS. 7 to 10.
[0051] 7, the component image acquiring unit 2a acquires a frame image 10 from the image acquiring unit 1. The component image acquiring unit 2a acquires a first component image 11 and a second component image 12 based on the frame image 10. In this embodiment, the component image acquiring unit 2a is configured to acquire the first component image 11 and the second component image 12 based on a radiation image, which is a moving image.
[0052] (Acquisition of the first and second component images) Here, referring to Fig. 8, a detailed configuration will be described in which the component image acquisition unit 2a (see Fig. 7) acquires the first component image 11 and the second component image 12 based on the frame image 10. In the example shown in Fig. 8, the frame image 10 is an X-ray image showing a catheter 91, a blood vessel 92a, a rib 92b, a diaphragm 92c, and a spine 92d. In the example shown in Fig. 8, dots indicate that noise is included in the frame image 10.
[0053] In this embodiment, the component image acquisition unit 2a is configured to acquire a first component image 11 containing a first component in which noise has been reduced from each of two frame images 10 captured at different times among the multiple frame images 10. That is, the image processing program 20 (see FIG. 1) causes the component image acquisition unit 2a to execute a process of acquiring a first component image 11 containing a first component in which noise has been reduced from each of two frame images 10 captured at different times among the multiple frame images 10.
[0054] In addition, in this embodiment, the component image acquisition unit 2a is configured to reduce noise contained in each of the two frame images 10 by at least one of frequency component processing, filter processing, and processing using a learned model, and generate two first component images 11.
[0055] When reducing noise by frequency component processing, the component image acquisition unit 2a is configured to perform frequency component processing on two frame images 10 to separate them into at least high frequency components and low frequency components, and to generate an image containing the low frequency components from among the high frequency components and low frequency components as a first component image 11.
[0056] Furthermore, when noise is reduced by filtering, the component image acquisition unit 2a performs filtering using a filter 23 (see FIG. 1). The filter 23 includes, for example, a smoothing filter. That is, in this embodiment, the component image acquisition unit 2a is configured to generate the first component image 11 by performing filtering on the two frame images 10 using a smoothing filter. The smoothing filter includes, for example, a Gaussian filter or a bilateral filter.
[0057] When the contrast of the catheter 91 or the like in the frame image 10 is reduced by the filtering process using the smoothing filter, the contrast of the first component in the first component image 11 is also reduced. Therefore, the component image acquisition unit 2a may perform filtering process using a smoothing filter on the two frame images 10, and then perform filtering process using an enhancement filter such as an unsharp mask or an edge enhancement filter to generate the first component image 11. That is, the filter 23 may include an enhancement filter.
[0058] In addition, when reducing noise using a trained model, the component image acquisition unit 2a inputs the frame image 10 to a trained model that has been trained to reduce noise from an image, and acquires a first component image 11 in which noise has been reduced.
[0059] As shown in FIG. 8, the first component image 11 includes, as the first component, linear image components such as a catheter 91 and a blood vessel 92a, as well as linear image components such as the contours of a rib 92b, a diaphragm 92c, and a spine 92d.
[0060] 8, the component image acquiring unit 2a is configured to acquire a second component image 12 including a second component different from the first component from one of the two frame images 10. That is, the image processing program 20 causes the component image acquiring unit 2a to execute a process of acquiring a second component image 12 including a second component different from the first component from one of the two frame images 10.
[0061] In this embodiment, the component image acquisition unit 2a is configured to generate a second component image 12 by subtracting a corresponding image of the two first component images 11 from each of the two frame images 10. The second component image 12 includes, as the second component, image components of parts other than the contours of the ribs 92b, the diaphragm 92c, and the spine 92d, and noise. Note that the second component image 12 does not include the first component, and therefore does not include linear image components such as the catheter 91 and the blood vessel 92a, and linear image components such as the contours of the ribs 92b, the contour of the diaphragm 92c, and the contour of the spine 92d.
[0062] 7 again, the component image acquisition unit 2a outputs the acquired first component image 11 to the first component intermediate image generation unit 2b and the displacement amount estimation unit 2c. The component image acquisition unit 2a also outputs the acquired second component image 12 to the converted image generation unit 2d.
[0063] The first component intermediate image generating unit 2b is configured to generate a first component intermediate image 14 including a noise-reduced first component and at a time different from the two frame images 10 based on two first component images 11. Specifically, the first component intermediate image generating unit 2b is configured to generate the first component intermediate image 14 by inputting the two first component images 11 to a first trained model 21 (see FIG. 6) that has been trained to generate an image in which an image component is displaced based on a plurality of images. The first component intermediate image generating unit 2b may obtain a motion vector 30a (see FIG. 5) of the first component from the two first component images 11, and generate the first component intermediate image 14 based on the obtained motion vector 30a and the first component images 11.
[0064] The first component intermediate image generating unit 2b outputs the generated first component intermediate image 14 to the displacement amount estimating unit 2c. In addition, the first component intermediate image generating unit 2b outputs the generated first component intermediate image 14 to the interpolated image generating unit 2e.
[0065] The displacement amount estimation unit 2c estimates the displacement amount 30 of the first component based on the two first component images 11. Specifically, the displacement amount estimation unit 2c is configured to estimate the displacement amount 30 of the first component based on one of the two first component images 11 and the first component intermediate image 14. That is, the image processing program 20 is configured to cause the displacement amount estimation unit 2c to execute a process of estimating the displacement amount 30 of the first component based on the two first component images 11.
[0066] In this embodiment, the displacement amount estimation unit 2c is configured to acquire a first component motion vector 30a between frames of the two first component images 11 based on the two first component images 11, and to estimate the displacement amount 30 of the first component based on the acquired motion vector 30a. That is, the displacement amount estimation unit 2c estimates the first component displacement amount 30 by acquiring the first component motion vector 30a based on the first component image 11 and the first component intermediate image 14. Note that the displacement amount estimation unit 2c may be configured to estimate the first component displacement amount 30 by inputting the two first component images 11 to a second trained model 22 (see FIG. 1) that has been trained to estimate the image component displacement amount 30 from a plurality of images.
[0067] The second trained model 22 is generated by training a trained model that is, for example, a convolutional neural network or includes a convolutional neural network in part. In addition, when generating the second trained model 22, images with a frame rate that is sufficiently smaller than the respiration and / or heartbeat of the subject 90 are used as moving images used for training so that the displacement of the image component due to the respiration and / or heartbeat of the subject 90 can be accurately estimated. The displacement amount estimation unit 2c outputs the estimated displacement amount 30 of the first component to the conversion image generation unit 2d.
[0068] The converted image generating unit 2d is configured to convert the second component in the second component image 12 based on the displacement amount 30 of the first component estimated by the displacement amount estimating unit 2c, thereby acquiring the converted image 15. That is, the image processing program 20 is configured to cause the converted image generating unit 2d to execute a process of acquiring the converted image 15 by converting the second component in the second component image 12 based on the estimated displacement amount 30 of the first component.
[0069] In this embodiment, the transformed image generating unit 2d is configured to convert the second component by performing a geometric transformation process on the second component image 12 based on the estimated displacement amount 30 of the first component, thereby acquiring the transformed image 15. The transformed image generating unit 2d may also generate a coordinate transformation map based on the displacement amount 30 of the first component estimated by the displacement amount estimating unit 2c, and perform an affine transformation on the second component image 12 to generate the transformed image 15. Then, the transformed image generating unit 2d outputs the acquired transformed image 15 to the interpolated image generating unit 2e.
[0070] The interpolated image generating unit 2e is configured to generate an interpolated image 13 of a different time from the two frame images 10, based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12. That is, the image processing program 20 is configured to cause the interpolated image generating unit 2e to execute a process of generating an interpolated image 13 of a different time from the two frame images 10, based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12. In this embodiment, the interpolated image generating unit 2e is configured to generate the interpolated image 13, based on the first component intermediate image 14 and the converted image 15.
[0071] Here, the first component intermediate image 14 generated using the first trained model 21 may have a different contrast from the frame image 10. In this case, the interpolated image 13 generated based on the first component intermediate image 14 and the converted image 15 may also have a different contrast from the frame image 10. When the frame image 10 and the interpolated image 13 are displayed as a moving image in a state in which the contrast between the interpolated image 13 and the frame image 10 is different, flickering occurs in the moving image due to the difference in contrast.
[0072] Therefore, in this embodiment, the interpolated image generating unit 2e is configured to perform weighting based on the contrast of the first component intermediate image 14 and the contrast of the second component image 12 when generating the interpolated image 13, and then synthesize the first component intermediate image 14 and the converted image 15. For example, when the difference between the contrast of the first component intermediate image 14 and the contrast of the second component image 12 is large, the interpolated image generating unit 2e increases the weighting of the converted image 15, synthesizes the first component intermediate image 14 and the converted image 15, and generates the interpolated image 13. When the difference between the contrast of the first component intermediate image 14 and the contrast of the second component image 12 is small, the interpolated image generating unit 2e increases the weighting of the first component intermediate image 14, and then synthesizes the first component intermediate image 14 and the converted image 15, and generates the interpolated image 13. The interpolated image generating unit 2e outputs the generated interpolated image 13 to the moving image generating unit 2f.
[0073] The moving image generation unit 2f generates an interpolated moving image 16 based on the interpolated image 13 input from the interpolated image generation unit 2e and the frame images 10 acquired from the image acquisition unit 1. The interpolated moving image 16 has a higher frame rate than the moving image constituted by the frame images 10. In this embodiment, for example, when the frame rate of the moving image constituted by the frame images 10 is 7.5 fps, the frame rate of the interpolated moving image 16 is 15 fps. The moving image generation unit 2f is configured to generate the interpolated moving image 16 by interpolation. The moving image generation unit 2f outputs the generated interpolated moving image 16 to the storage unit 4.
[0074] (Generation of an interpolated image by interpolation) When the moving image generator 2f (see FIG. 7) generates the interpolated moving image 16 by interpolation, as shown in FIG 9, the moving image generator 2f inserts an interpolated image 13 into a frame between each frame image 10 (see FIG. 7). In the example shown in FIG 9, the moving image generator 2f inserts a first interpolated image 13a between the first frame image 10a and the third frame image 10c. The first interpolated image 13a is an interpolated image 13 generated based on the first frame image 10a and the third frame image 10c.
[0075] Furthermore, the moving image generating unit 2f inserts a second interpolated image 13b between the third frame image 10c and the fourth frame image 10d. The second interpolated image 13b is an interpolated image 13 generated based on the third frame image 10c and the fifth frame image 10e. The moving image generating unit 2f generates an interpolated moving image 16 by repeating the same process. When generating the interpolated moving image 16 by interpolation, the image processing unit 2 shoots frame images 10 at a predetermined frame rate, and then generates a plurality of interpolated images 13 based on the plurality of frame images 10, and generates the interpolated moving image 16 by interpolation. In other words, the generation of the interpolated moving image 16 by interpolation is performed in order to improve the frame rate of the moving image composed of the frame images 10 after shooting of the moving image composed of the frame images 10 is completed.
[0076] (Generation of interpolated images by extrapolation) Furthermore, when generating the first component image 11 based on the interpolated image 13, the image processing unit 2 (see FIG. 7) generates the second component image 12 and the first component intermediate image 14 based on the first component image 11 generated based on the interpolated image 13, and generates the interpolated image 13. In this case, the image processing unit 2 may be configured to generate the interpolated moving image 16 by extrapolation. When the image processing unit 2 generates the interpolated moving image 16 by extrapolation, as shown in FIG. 10, the image processing unit 2 (see FIG. 1) inserts a third interpolated image 13c generated from the first frame image 10a and a frame image 10 (not shown) one frame before the first frame image 10a, as the frame image 10 one frame after the first frame image 10a. Then, the image processing unit 2 generates a fourth interpolated image 13d based on the third interpolated image 13c and the third frame image 10c, and inserts it as the frame image 10 one frame after the third frame image 10c. Furthermore, the image processing unit 2 generates a fifth interpolated image 13e based on the fourth interpolated image 13d and the fifth frame image 10e, and inserts it as the frame image 10 one frame after the fifth frame image 10e. In this way, the image processing unit 2 generates the interpolated moving image 16. That is, the generation of the interpolated moving image 16 by extrapolation generates an interpolated image 13 and inserts it as a new frame image 10 every time one frame of a frame image 10 is shot. In other words, the generation of the interpolated moving image 16 by extrapolation is performed in order to improve the frame rate of the moving image made up of the frame images 10 in real time while the moving image made up of the frame images 10 is being shot.
[0077] In this embodiment, the image processing unit 2 (see FIG. 1) is configured to generate the first component image 11 based on any one of a plurality of frame images 10 and the interpolated image 13. That is, the image processing unit 2 (component image acquisition unit 2a) may be configured to generate the interpolated image 13 based only on the frame images 10. Also, the component image acquisition unit 2a may be configured to generate the interpolated image 13 as an interpolated image different from the already generated interpolated image 13 based on the first component image 11 generated based on the already generated interpolated image 13. In either case where the image processing unit 2 generates the first component image 11 using only the frame images 10, or where the image processing unit 2 generates the first component image 11 using the frame images 10 and the interpolated image 13, the image processing unit 2 can generate the interpolated moving image 16 (see FIG. 7) by interpolation or extrapolation.
[0078] (Interpolated image generation process) Next, a process in which the image processing unit 2 (see FIG. 7) generates the interpolated image 13 (see FIG. 7) will be described with reference to Fig. 11. Note that the process described in Fig. 11 is a process in which the image processing unit 2 generates the interpolated image 13 and generates the interpolated moving image 16 by interpolation.
[0079] In step 101, the image acquisition unit 1 (see FIG. 7) acquires a plurality of frame images 10 (see FIG. 7) from the X-ray imaging device 200 (see FIG. 1).
[0080] In step 102, the component image acquisition unit 2a (see Figure 7) acquires a first component image 11 (see Figure 7) containing a noise-reduced first component from each of two frame images 10 that were captured at different times among the multiple frame images 10.
[0081] In step 103, the component image acquisition section 2a acquires, from one of the two frame images 10, a second component image 12 (see FIG. 7) that includes a second component different from the first component.
[0082] In step 104, the displacement amount estimating unit 2c (see FIG. 7) estimates the first component displacement amount 30 (see FIG. 7) based on the two first component images 11.
[0083] In step 105, the first component intermediate image generating unit 2b (see FIG. 7) generates a first component intermediate image 14 (see FIG. 7) based on the two first component images 11.
[0084] In step 106, the converted image generating unit 2d (see FIG. 7) generates a converted image 15 (see FIG. 7) based on the displacement amount 30 of the first component and the second component image 12.
[0085] In step 107, the interpolated image generation unit 2e (see FIG. 7) generates an interpolated image 13 of a different time from the two frame images 10 based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12.
[0086] In step 108, the image processing unit 2 determines whether or not to end the generation of the interpolated image 13. The image processing unit 2 determines whether or not to end the generation of the interpolated image 13, for example, based on whether or not the interpolated image 13 corresponding to the multiple input frame images 10 has been generated. If the generation of the interpolated image 13 is to be ended, the process proceeds to step 109. If the generation of the interpolated image 13 is not to be ended, the process proceeds to step 111.
[0087] In step 109, the video generator 2f (see FIG. 7) generates an interpolated video 16 (see FIG. 7).
[0088] In step 110, the storage unit 4 (see FIG. 7) stores the interpolated video sequence 16. Then, the process ends.
[0089] Also, when the process proceeds from step to step 111, in step 111, the component image acquiring section 2a acquires the next frame image 10. After that, the process proceeds to step .
[0090] It should be noted that either the process in step 102 or the process in step 103 may be performed first. Also, it should be noted that either the process in step 104 or the process in step 105 may be performed first.
[0091] Furthermore, when the image processing unit 2 generates the interpolated video 16 by extrapolation, the process of step 109 is performed after step 107. Furthermore, the control unit 3 is configured to perform control to display the interpolated video 16 stored in the storage unit 4 on the display unit 5.
[0092] (Effects of this embodiment) In this embodiment, the following effects can be obtained.
[0093] In this embodiment, as described above, the image processing device 100 is an image processing device that processes a plurality of frame images 10 that constitute a moving image, and includes an image acquisition unit 1 that acquires the plurality of frame images 10, and an image processing unit 2 that processes the plurality of frame images 10 acquired by the image acquisition unit 1, and the image processing unit 2 is configured to acquire a first component image 11 containing a first component with reduced noise from each of two frame images 10 that have different shooting times among the plurality of frame images 10, acquire a second component image 12 containing a second component different from the first component from one of the two frame images 10, estimate a displacement amount 30 of the first component based on the two first component images 11, and generate an interpolated image 13 of a time different from the two frame images 10 based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12.
[0094] As a result, the first component displacement amount 30 is estimated based on the first component image 11 in which noise has been reduced, and therefore the first component displacement amount 30 can be estimated with higher accuracy than a configuration in which the first component displacement amount 30 is estimated based on an image in which noise has not been reduced. Therefore, since it is possible to accurately estimate the first component displacement amount 30, it is possible to improve the accuracy of the first component displacement amount 30 of the two first component images 11 and the interpolated image 13 generated based on the second component image 12. As a result, even when the interpolated image 13 is generated based on a plurality of frame images 10 containing noise, the interpolated image 13 can be generated with high accuracy.
[0095] Moreover, in this embodiment, the image processing device 100 is an image processing device that processes a plurality of frame images 10 that constitute a radiographic image of a moving image, and includes an image acquisition unit 1 that acquires the radiographic image that is a moving image, and an image processing unit 2 that processes the radiographic image acquired by the image acquisition unit 1. The image processing unit 2 is configured to acquire a first component image 11 containing a first component in which noise has been reduced from at least two frame images 10 of the plurality of frame images 10 included in the radiographic image that have different shooting times, acquire a second component image 12 containing a second component different from the first component from one of the two frame images 10, estimate a displacement amount 30 of the first component based on the two first component images 11, and generate a radiographic interpolated image (interpolated image 13) that is a radiographic image of a time different from the two frame images 10 based on the displacement amount 30 of the first component of the two first component images 11 and the second component image 12.
[0096] Here, in capturing a radiographic image of a moving image, the amount of radiation to be irradiated is reduced in order to reduce the amount of radiation exposure. In this case, the signal-to-noise ratio of the radiographic image is reduced, resulting in an image containing more noise. Therefore, by configuring as described above, the amount of displacement 30 of the first component is estimated based on the first component image 11 in which noise has been reduced and acquired from the radiographic image, so that the amount of displacement 30 of the first component can be estimated with high accuracy compared to a configuration in which the amount of displacement 30 of the first component is estimated based on a radiographic image in which noise has not been reduced. Therefore, since the amount of displacement 30 of the first component can be estimated with high accuracy, the accuracy of the amount of displacement 30 of the first component of the two first component images 11 and the interpolated image 13 generated based on the second component image 12 can be improved. Therefore, the amount of radiation is reduced to reduce the amount of radiation exposure, and the number of times radiation is irradiated and captured is reduced by thinning-out imaging, so that it is possible to provide an image processing device 100 that can reduce the amount of radiation exposure and improve the accuracy of the interpolated image 13.
[0097] In addition, in this embodiment, the image processing method is an image processing method for processing a plurality of frame images 10 that constitute a moving image, and includes the steps of obtaining a first component image 11 containing a first component with reduced noise from each of two frame images 10 that were captured at different times among the plurality of frame images 10, obtaining a second component image 12 containing a second component different from the first component from one of the two frame images 10, estimating a displacement amount 30 of the first component based on the two first component images 11, and generating an interpolated image 13 of a time different from the two frame images 10 based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12.
[0098] This makes it possible to provide an image processing method capable of generating an interpolated image 13 with high accuracy, even when the interpolated image 13 is generated based on a frame image 10 containing noise, similar to the image processing device 100 described above.
[0099] In addition, in this embodiment, the image processing program 20 is an image processing program executed in an image processing device 100 that processes a plurality of frame images 10 that constitute a moving image, and causes the image processing device 100 to execute the following processes: a process of acquiring a first component image 11 containing a first component with reduced noise from each of two frame images 10 that were captured at different times among the plurality of frame images 10; a process of acquiring a second component image 12 containing a second component different from the first component from one of the two frame images 10; a process of estimating a displacement amount 30 of the first component based on the two first component images 11; and a process of generating an interpolated image 13 of a time different from the two frame images 10 based on the displacement amount 30 of the first components of the two first component images 11 and the second component image 12.
[0100] This makes it possible to provide an image processing program 20 that can generate an interpolated image 13 with high accuracy, even when the interpolated image 13 is generated based on a frame image 10 that contains noise, similar to the image processing device 100 described above.
[0101] In addition, in the above embodiment, the following additional effects can be obtained by configuring as follows.
[0102] That is, in this embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to generate a first component intermediate image 14 containing a noise-reduced first component and at a time different from that of the frame image 10 based on two first component images 11, estimate a displacement amount 30 of the first component based on one of the two first component images 11 and the first component intermediate image 14, convert the second component in the second component image 12 based on the estimated displacement amount 30 of the first component, acquire a converted image 15, and generate an interpolated image 13 based on the first component intermediate image 14 and the converted image 15. As a result, the first component intermediate image 14 is generated based on the noise-reduced first component, and therefore the accuracy of the first component intermediate image 14 can be improved compared to a configuration in which the first component intermediate image 14 is generated based on a noise-containing first component. Furthermore, since the displacement amount 30 of the first component is estimated based on the first component intermediate image 14 generated based on the first component with reduced noise and the first component image 11, the displacement amount 30 of the first component can be estimated with high accuracy by comparing with a configuration in which the displacement amount 30 of the first component is estimated based on the first component intermediate image generated based on the first component including noise and the first component image. Therefore, the accuracy of the converted image 15 generated based on the displacement amount 30 of the first component can be improved. As a result, the interpolated image 13 is generated based on the first component intermediate image 14 and the converted image 15 with improved accuracy, so that the accuracy of the interpolated image 13 can be further improved.
[0103] In the present embodiment, as described above, the frame image 10 includes a radiographic image, and the component image acquisition unit 2a (image processing unit 2) is configured to acquire the first component image 11 and the second component image 12 based on the radiographic image, which is a moving image. As a result, the first component image 11 and the second component image 12 are acquired from the radiographic image, which is a moving image, so that the accuracy of the interpolated image 13, which is a radiographic image, can be improved. Therefore, even if the number of radiation (X-ray) irradiations is reduced to reduce the radiation exposure dose and the number of frames of the radiographic image, which is a moving image, is reduced, the interpolated image 13 with improved accuracy can be generated. Therefore, the number of frames of the radiographic image, which is a moving image, can be increased by the interpolated image 13 with improved accuracy, so that the accuracy of the radiographic image, which is a moving image, can be suppressed from decreasing. As a result, it is possible to achieve both a reduction in the radiation exposure dose and suppression of a decrease in the accuracy of the radiographic image, which is a moving image.
[0104] In addition, in this embodiment, as described above, the first component intermediate image generating unit 2b (image processing unit 2) is configured to generate the first component intermediate image 14 by inputting two first component images 11 to the first trained model 21 that has been trained to generate an image in which the image component is displaced based on a plurality of images. Here, for example, when estimating the displacement of the image component in a plurality of images by acquiring a motion vector, if the image component is deformed between the plurality of images, it becomes difficult to accurately estimate the displacement of the image component. On the other hand, when generating the first trained model 21 by training to generate an image in which the image component is displaced based on a plurality of images, the first trained model 21 that can acquire the displacement of the image component even if the image component is deformed between the images can be generated by training using teacher data including an image in which the image component is deformed between the images. Therefore, as described above, by configuring the first component intermediate image 14 to be generated by inputting two first component images 11 to the first trained model 21, even if the image component is deformed between the plurality of images, the first component intermediate image 14 that accurately reflects the displacement of the first component can be generated based on the first trained model 21. As a result, even if the first component is transformed between a plurality of first component images 11, the accuracy of the first component intermediate image 14 can be prevented from decreasing.
[0105] In the present embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to generate the first component image 11 based on any one of the multiple frame images 10 and the interpolated image 13. This allows the interpolated image 13 to be used as the frame image 10 for generating the first component image 11, so that the number of frame images 10 used for generating the first component image 11 can be increased without increasing the frame rate when capturing the multiple frame images 10, compared to a configuration in which the first component image 11 is generated based only on the multiple frame images 10. As a result, the number of generated interpolated images 13 can be increased, compared to a configuration in which the interpolated image 13 is generated based only on the frame images 10, so that a moving image capable of displaying the displacement of the image components in more detail can be output.
[0106] Furthermore, in this embodiment, as described above, the interpolated image generating unit 2e (image processing unit 2) is configured to perform weighting based on the contrast of the first component intermediate image 14 and the contrast of the second component image 12 when generating the interpolated image 13, and then synthesize the first component intermediate image 14 and the converted image 15. Here, for example, if there is a large difference between the contrast of the first component intermediate image 14 and the contrast of the multiple frame images 10, there will also be a large difference between the contrast of the interpolated image 13 generated by the first component intermediate image 14 and the converted image 15 and the contrast of the frame images 10. In this case, flickering may occur in the moving image due to the difference in contrast between the frame in which the interpolated image 13 is displayed and the frame in which the multiple frame images 10 are displayed. Therefore, as described above, by performing weighting based on the contrast of the first component intermediate image 14 and the contrast of the second component image 12, and then configuring to synthesize the first component intermediate image 14 and the converted image 15, when the difference between the contrast of the plurality of frame images 10 and the contrast of the interpolated image 13 is large, the weighting of the converted image 15 is increased, thereby making it possible to suppress the difference between the contrast of the interpolated image 13 and the contrast of the frame images 10 from becoming large. As a result, in a moving image (interpolated moving image 16) composed of the interpolated image 13 and the frame images 10, it is possible to suppress the occurrence of flickering due to the difference in contrast between the frame images 10. Furthermore, when the difference between the contrast of the frame images 10 and the contrast of the interpolated image 13 is small, the weighting of the first component intermediate image 14 is increased, thereby making it possible to generate an interpolated image 13 in which the visibility of the first component is improved while suppressing noise. As a result, in a moving image (interpolated moving image 16) generated based on the frame images 10 and the interpolated image 13, it is possible to improve the visibility of the first component.
[0107] In addition, in this embodiment, as described above, the displacement amount estimation unit 2c (image processing unit 2) is configured to obtain a motion vector 30a of the first component between the frames of the two first component images 11 based on the two first component images 11, and estimate the displacement amount 30 of the first component based on the obtained motion vector 30a, or to estimate the displacement amount 30 of the first component by inputting the two first component images 11 to the second trained model 22 that has been trained to estimate the displacement amount 30 of the image component from a plurality of images. Here, the motion vector 30a between the plurality of images is obtained based on the movement of the feature points included in each image. Therefore, when estimating the displacement amount 30 of the first component based on the motion vector 30a of the first component, it is possible to obtain the motion vector 30a by comparing the feature points of the images between the frames of the two first component images 11, and therefore it is possible to suppress an increase in the processing load of the image processing unit 2. In addition, when estimating the displacement amount 30 of the first component by inputting two first component images 11 to the second trained model 22 that has been trained to estimate the displacement amount 30 of the image component from multiple images, the displacement amount 30 of the first component can be estimated even if deformation occurs in the first component between the two first component images 11.
[0108] In addition, in this embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to reduce noise contained in each of the two frame images 10 by at least one of the processes of frequency component processing, filter processing, and processing using a trained model, and generate two first component images 11. Here, the noise contained in each of the two frame images 10 is, for example, a high-frequency component such as white noise. Therefore, by removing the high-frequency component by at least one of the processes of frequency component processing, filter processing, and processing using a trained model, noise can be easily reduced from each of the two frame images 10. Therefore, as described above, by reducing noise from each of the two frame images 10 by at least one of the processes of frequency component processing, filter processing, and processing using a trained model, the first component image 11 can be easily generated.
[0109] Also, in this embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to perform frequency component processing on the two frame images 10 to separate them into at least high frequency components and low frequency components, and to generate an image including the low frequency components of the high frequency components and low frequency components as the first component image 11. In this way, by separating each of the two frame images 10 into high frequency components and low frequency components and using the image including the low frequency components as the first component image 11, it is possible to easily prevent noise from being included in the first component image 11. As a result, it is possible to easily generate a first component image 11 with reduced noise by the frequency component processing.
[0110] Also, in this embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to perform filtering processing on the two frame images 10 using a smoothing filter to generate the first component image 11. This makes it possible to easily reduce noise from each of the two frame images 10 by filtering processing using the smoothing filter. As a result, it is possible to easily generate the first component image 11 with reduced noise from each of the two frame images 10 by filtering processing using the smoothing filter.
[0111] In the present embodiment, as described above, the component image acquisition unit 2a (image processing unit 2) is configured to generate the second component image 12 by subtracting the corresponding image of the two first component images 11 from each of the two frame images 10. This allows the second component image 12, which includes image components other than the first component among the image components included in the frame images 10, to be acquired by subtracting the first component from each of the two frame images 10. Therefore, unlike a configuration in which the second component image 12 is acquired by extracting a portion of the image components from each of the two frame images 10 as the second component, it is possible to generate the second component image 12, which includes image components other than the first component among the image components included in the frame image 10, which is the original image. As a result, it is possible to prevent the interpolation image 13 from becoming an unnatural image due to a large difference between the image components included in the interpolation image 13 generated based on the first component image 11 and the second component image 12 and the image components included in the frame image 10.
[0112] In the present embodiment, as described above, the converted image generating unit 2d (image processing unit 2) is configured to convert the second component by performing a geometric transformation process on the second component image 12 based on the estimated displacement amount 30 of the first component, thereby acquiring the converted image 15. Here, the displacement amount 30 of the first component is acquired based on the first component image 11 in which noise has been reduced, and therefore the estimation accuracy is higher than the displacement amount of the image component estimated based on an image in which noise has not been reduced. Therefore, by configuring as described above, the geometric transformation process is performed on the second component image 12 based on the displacement amount 30 of the first component with high estimation accuracy, so that the second component of the second component image 12 can be converted with high accuracy. As a result, it is possible to improve the accuracy of the converted image 15, and it is possible to further suppress the interpolated image 13 from becoming an unnatural image.
[0113] [Variations] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims, not by the description of the embodiments above, and further includes all modifications (variations) within the meaning and scope of the claims.
[0114] For example, in the above embodiment, an example has been shown in which the image processing device 100 and the X-ray imaging device 200 are configured as different devices, but the present invention is not limited to this. For example, as in an X-ray imaging device 300 according to a modified example shown in Fig. 12, an image processing unit 2 that executes an image processing program 20 may be provided in the X-ray imaging device 300. The X-ray imaging device 300 according to the modified example differs from the X-ray imaging device 200 according to the above embodiment in that it includes an image processing unit 209, a storage unit 210, a display unit 211, and an input receiving unit 212.
[0115] The image processing unit 209 has the same configuration as the image processing unit 2 according to the above embodiment. The storage unit 210, the display unit 211, and the input receiving unit 212 have the same configuration as the storage unit 4, the display unit 5, and the input receiving unit 6 according to the above embodiment, respectively.
[0116] That is, in the X-ray imaging apparatus 300 according to the modified example, an image processing unit 209 provided in the X-ray imaging apparatus 300 acquires a first component image 11 and a second component image 12 based on a frame image 10. Then, the image processing unit 209 generates an interpolated image 13 based on the first component image 11 and the second component image 12. The configuration in which the image processing unit 209 generates the interpolated image 13 is similar to that of the image processing unit 2 according to the above embodiment, and therefore a detailed description thereof will be omitted.
[0117] With this configuration, it is possible to provide an X-ray imaging device 300 that can generate an interpolated image 13 with high accuracy even when the interpolated image 13 is generated based on a frame image 10 that contains noise.
[0118] In the above embodiment, the image processing unit 2 generates the first component intermediate image 14 and the transformed image 15, and generates the interpolated image 13 based on the first component intermediate image 14 and the transformed image 15. However, the present invention is not limited to this. For example, the image processing unit 2 does not need to generate the first component intermediate image 14 and the transformed image 15. In this case, the image processing unit may be configured to generate the interpolated image 13 directly from the first component image 11 and the second component image 12, without generating the first component intermediate image 14 and the transformed image 15, by using a trained model that has been trained to generate the interpolated image 13 based on the first component image 11 and the second component image 12.
[0119] In the above embodiment, the frame image 10 is a radiation image, but the present invention is not limited to this. For example, the frame image may be an optical image captured by an optical camera of a subject that is degraded by light (illumination light) irradiated during shooting.
[0120] In addition, in the above embodiment, the radiation image is an X-ray image, but the present invention is not limited to this. For example, the radiation image may be a gamma ray image using gamma rays as radiation.
[0121] In the above embodiment, the image processing unit 2 (interpolated image generating unit 2e) performs weighting based on the contrast of the first component intermediate image 14 and the contrast of the second component image 12, and then synthesizes the first component intermediate image 14 and the converted image 15. However, the present invention is not limited to this. For example, the image processing unit may be configured to generate the interpolated image 13 by synthesizing the first component intermediate image 14 and the converted image 15 without performing weighting. However, when the image processing unit synthesizes the first component intermediate image 14 and the converted image 15 without performing weighting, a difference in contrast may occur between the interpolated image 13 and each frame image 10. In this case, flicker may occur in the moving image (interpolated moving image 16) to be output. Therefore, it is preferable that the image processing unit is configured to perform weighting based on the contrast of the first component intermediate image 14 and the contrast of the second component image 12, and then synthesizes the first component intermediate image 14 and the converted image 15.
[0122] In the above embodiment, the image processing unit 2 performs frequency component processing on the frame image 10 and separates the image into at least high-frequency components and low-frequency components, but the present invention is not limited to this. For example, the image processing unit may be configured to perform frequency component processing on the frame image 10 and separate the image into high-frequency components, medium-frequency components, and low-frequency components. In this case, the image processing unit may obtain an image that does not contain high-frequency components as the first component image 11. That is, the image processing unit may obtain an image that contains both medium-frequency components and low-frequency components as the first component image 11, or may obtain an image that contains only medium-frequency components as the first component image 11. Also, the image processing unit may obtain an image that contains at least high-frequency components as the second component image. That is, the image processing unit may obtain an image that contains both high-frequency components and low-frequency components as the second component image 12, or may obtain an image that contains only high-frequency components as the second component image 12. That is, the image processing unit may be configured to obtain the first component image 11 and the second component image 12 using a part of the image components contained in the frame image 10.
[0123] In the above embodiment, an example of a configuration in which the image processing unit 2 generates the second component image 12 by subtracting a corresponding image of the two first component images 11 from each of the two frame images 10 has been shown, but the present invention is not limited to this. For example, the image processing unit 2 may be configured to obtain the second component image 12 by obtaining the second component from each of the two frame images 10.
[0124] In the above embodiment, the frame rate of the moving image is 7.5 fps, but the present invention is not limited to this. For example, the frame rate of the moving image may be 15 fps. The frame rate of the moving image may be any frame rate as long as it is appropriate for the moving image to be displayed when an operator such as a doctor treats the subject 90 using the catheter 91.
[0125] In the above embodiment, for convenience of explanation, the process of generating the interpolated image 13 using the image processing device 100 of the present invention has been explained using a flow-driven flowchart in which the processes are performed in order according to a processing flow, but the present invention is not limited to this. In the present invention, the process of generating the interpolated image 13 using the image processing device 100 may be performed by event-driven processing in which processing is performed on an event-by-event basis. In this case, the process may be completely event-driven, or may be a combination of event-driven and flow-driven.
[0126] [Aspects] It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.
[0127] (Item 1) An image processing device that processes a plurality of frame images constituting a moving image, an image acquisition unit that acquires the plurality of frame images; an image processing unit that processes the plurality of frame images acquired by the image acquisition unit, The image processing unit includes: acquiring a plurality of first component images including a first component in which noise has been reduced from two frame images captured at different times among the plurality of frame images; obtaining a second component image including a second component different from the first component from one of the two frame images; Estimating a displacement amount of the first component based on the two first component images; an image processing device configured to generate an interpolated image of a time different from that of the two frame images based on the displacement amounts of the first components of the two first component images and the second component image.
[0128] (Item 2) The image processing unit includes: generating a first component intermediate image including the first component with reduced noise based on the two first component images and having a different time from the two frame images; estimating a displacement amount of the first component based on one of the two first component images and the first component intermediate image; obtaining a transformed image by transforming the second component in the second component image based on the estimated displacement amount of the first component; 2. The image processing device according to claim 1, configured to generate the interpolated image based on the first component intermediate image and the transformed image.
[0129] (Item 3) the frame image includes a radiological image, 3. The image processing device according to item 2, wherein the image processing unit is configured to obtain the first component image and the second component image based on the radiation image which is a moving image.
[0130] (Item 4) 4. The image processing device according to claim 3, wherein the image processing unit is configured to generate the first component intermediate image by inputting the two first component images to a first trained model that has been trained to generate an image in which image components are displaced based on a plurality of images.
[0131] (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the image processing unit is configured to generate the first component image based on any one of the plurality of frame images and the interpolated image.
[0132] (Item 6) 5. The image processing device according to any one of items 2 to 4, wherein, when generating the interpolated image, the image processing unit is configured to perform weighting based on a contrast of the first component intermediate image and a contrast of the second component image, and then combine the first component intermediate image and the converted image.
[0133] (Item 7) The image processing device according to any one of items 1 to 6, wherein the image processing unit is configured to acquire a motion vector of the first component between frames of the two first component images based on the two first component images, and estimate a displacement amount of the first component based on the acquired motion vector, or estimate the displacement amount of the first component by inputting the two first component images to a second trained model trained to estimate a displacement amount of an image component from a plurality of images.
[0134] (Item 8) The image processing device according to any one of items 1 to 7, wherein the image processing unit is configured to reduce noise contained in each of the two frame images by at least one of frequency component processing, filtering, and processing using a trained model, and generate the two first component images.
[0135] (Item 9) 9. The image processing device according to item 8, wherein the image processing unit is configured to perform frequency component processing on the two frame images to separate them into at least high frequency components and low frequency components, and to generate an image including the low frequency components among the high frequency components and the low frequency components as the first component image.
[0136] (Item 10) 9. The image processing device according to item 8, wherein the image processing unit is configured to generate the first component image by performing filtering on the two frame images using a smoothing filter.
[0137] (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the image processing unit is configured to generate the second component image by subtracting a corresponding image out of the two first component images from each of the two frame images.
[0138] (Item 12) 5. The image processing device according to any one of items 2 to 4, wherein the image processing unit is configured to convert the second component by performing a geometric transformation process on the second component image based on the estimated displacement amount of the first component, thereby acquiring the converted image.
[0139] (Item 13) 1. An image processing device that processes a plurality of frame images constituting a radiographic image of a moving image, an image acquisition unit for acquiring the radiographic image, which is a moving image; an image processing unit that processes the radiation image acquired by the image acquisition unit, The image processing unit includes: obtaining a first component image including a first component in which noise has been reduced from each of two frame images captured at different times among the plurality of frame images included in the radiation image; obtaining a second component image including a second component different from the first component from one of the two frame images; estimating a displacement amount of the first component based on the two first component images; an image processing device configured to generate a radiation interpolation image, which is a radiation image at a time different from that of the two frame images, based on the displacement amounts of the first components of the two first component images and the second component image.
[0140] (Item 14) 1. An image processing method for processing a plurality of frame images constituting a moving image, comprising the steps of: acquiring a plurality of first component images including a first component in which noise has been reduced from two frame images captured at different times among the plurality of frame images; acquiring a second component image including a second component different from the first component from one of the two frame images; estimating a displacement amount of the first component based on two of the first component images; generating an interpolated image for a time different from the two frame images based on the displacement amounts of the first components of the two first component images and the second component image.
[0141] (Item 15) An image processing program executed in an image processing device that processes a plurality of frame images constituting a moving image, A process of acquiring a plurality of first component images including a first component in which noise has been reduced from two frame images captured at different times among the plurality of frame images; A process of acquiring a second component image including a second component different from the first component from one of the two frame images; A process of estimating a displacement amount of the first component based on two of the plurality of first component images; and generating an interpolated image of a different time from the two frame images based on the displacement amount of the first components of the two first component images and the second component image. [Explanation of symbols]
[0142] 1 Image acquisition section 2, 209 Image processing section 10, 10a, 10b, 10c, 10d, 10e Frame images 11 First component image 12 Second component image 13 Interpolated image (radial interpolated image) 14 First component intermediate image 15 Converted Images 20 Image Processing Programs 21 First trained model 22 Second trained model 30 Displacement of the first component 30a Motion Vector 100 Image processing device
Claims
1. An image processing device that processes a plurality of frame images constituting a moving image, an image acquisition unit that acquires the plurality of frame images; an image processing unit that processes the plurality of frame images acquired by the image acquisition unit, The image processing unit includes: obtaining a first component image including a first component in which noise has been reduced from each of two frame images captured at different times among the plurality of frame images; obtaining a second component image including a second component different from the first component from one of the two frame images; Estimating a displacement amount of the first component based on the two first component images; an image processing device configured to generate an interpolated image of a time different from the two frame images based on the displacement amount of the first components of the two first component images and the second component image.
2. The image processing unit includes: generating a first component intermediate image including the first component with reduced noise based on the two first component images and having a different time from the two frame images; estimating a displacement amount of the first component based on one of the two first component images and the first component intermediate image; obtaining a transformed image by transforming the second component in the second component image based on the estimated displacement amount of the first component; The image processing apparatus of claim 1 , configured to generate the interpolated image based on the first component intermediate image and the transformed image.
3. the frame image includes a radiological image, The image processing device according to claim 2 , wherein the image processing unit is configured to obtain the first component image and the second component image based on the radiation image which is a moving image.
4. 4. The image processing device according to claim 3, wherein the image processing unit is configured to generate the first component intermediate image by inputting the two first component images to a first trained model that has been trained to generate an image in which image components are displaced based on a plurality of images.
5. The image processing device according to claim 1 , wherein the image processing unit is configured to generate the first component image based on any one of the plurality of frame images and the interpolated image.
6. 3. The image processing device according to claim 2, wherein, when generating the interpolated image, the image processing unit is configured to perform weighting based on a contrast of the first component intermediate image and a contrast of the second component image, and then combine the first component intermediate image and the converted image.
7. 2. The image processing device according to claim 1, wherein the image processing unit is configured to acquire a motion vector of the first component between frames of the two first component images based on the two first component images, and estimate a displacement amount of the first component based on the acquired motion vector, or estimate the displacement amount of the first component by inputting the two first component images to a second trained model that has learned to estimate a displacement amount of an image component from a plurality of images.
8. 2. The image processing device according to claim 1, wherein the image processing unit is configured to reduce noise contained in each of the two frame images by at least one of frequency component processing, filtering, and processing using a trained model, and generate the two first component images.
9. 9. The image processing device according to claim 8, wherein the image processing unit is configured to perform frequency component processing on the two frame images to separate them into at least high frequency components and low frequency components, and to generate an image including the low frequency components among the high frequency components and the low frequency components as the first component image.
10. The image processing device according to claim 8 , wherein the image processing unit is configured to generate the first component image by performing a filtering process on the two frame images using a smoothing filter.
11. The image processing device according to claim 1 , wherein the image processing unit is configured to generate the second component image by subtracting a corresponding one of the two first component images from each of the two frame images.
12. 3. The image processing device according to claim 2, wherein the image processing unit is configured to convert the second component by performing a geometric transformation process on the second component image based on the estimated displacement amount of the first component, thereby acquiring the transformed image.
13. 1. An image processing device that processes a plurality of frame images constituting a radiographic image of a moving image, an image acquisition unit for acquiring the radiographic image, which is a moving image; an image processing unit that processes the radiation image acquired by the image acquisition unit, The image processing unit includes: obtaining a first component image including a first component in which noise has been reduced from each of two frame images captured at different times among the plurality of frame images included in the radiation image; obtaining a second component image including a second component different from the first component from one of the two frame images; Estimating a displacement amount of the first component based on the two first component images; an image processing device configured to generate a radiation interpolation image, which is a radiation image at a different time than the two frame images, based on the displacement amounts of the first components of the two first component images and the second component image.
14. 1. An image processing method for processing a plurality of frame images constituting a moving image, comprising the steps of: acquiring a first component image including a first component in which noise has been reduced from each of two frame images captured at different times among the plurality of frame images; acquiring a second component image including a second component different from the first component from one of the two frame images; estimating a displacement amount of the first component based on the two first component images; generating an interpolated image for a time different from the two frame images based on the displacement amount of the first components of the two first component images and the second component image.
15. An image processing program executed in an image processing device that processes a plurality of frame images constituting a moving image, A process of acquiring a first component image including a first component in which noise has been reduced from each of two frame images captured at different times among the plurality of frame images; A process of acquiring a second component image including a second component different from the first component from one of the two frame images; A process of estimating a displacement amount of the first component based on two of the first component images; and generating an interpolated image of a different time from the two frame images based on the displacement amount of the first components of the two first component images and the second component image.
Citation Information
Patent Citations
Interpolation frame generation apparatus
JP2019057866A
