Radiographic imaging device, image processing device, and image processing method

Abstract

This radiographic imaging device comprises: a storage unit for storing a trained model (74a) that, by receiving as input a first image (81) and a second image (82) captured at different time points, generates a motion vector (9v) of an object in the first image (81) to the second image (82), and generates, on the basis of the motion vector (9v), an intermediate image (81a) at an intermediate time point between the time point at which the first image (81) was captured and the time point at which the second image (82) was captured; and a control unit. The control unit, after acquiring the motion vector (9v) by inputting the first image (81) and the second image (82) into the trained model (74a), sets tracking points (9a, 9b) in the second image (82) on the basis of the tracking points (9a, 9b) in the first image (81) and the acquired motion vector (9v).

Description

Radiography apparatus, image processing apparatus, and image processing method

[0001] The present invention relates to a radiographic apparatus, an image processing apparatus, and an image processing method.

[0002] BACKGROUND ART Conventionally, a radiation imaging apparatus has been known. Such a radiation imaging apparatus is disclosed, for example, in Japanese Patent Application Laid-Open No. 2018-15263.

[0003] The above-mentioned Japanese Patent Application Laid-Open No. 2018-15263 discloses an X-ray imaging diagnostic apparatus (radiography apparatus). This X-ray imaging diagnostic apparatus is used to observe the swallowing movement of a subject. The X-ray imaging diagnostic apparatus includes an imaging unit and an image processing unit. The imaging unit is configured to image the subject by detecting X-rays irradiated from an X-ray source and transmitted through the subject using an X-ray receiver. The image processing unit is configured to create an X-ray fluoroscopic image based on information about the transmitted X-rays.

[0004] The image processing unit in JP 2018-15263 A is configured to detect movement of the hyoid bone in an X-ray fluoroscopic image. Specifically, the image processing unit is configured to determine that the hyoid bone has moved based on the movement of the hyoid bone from inside a region of interest to outside the region of interest.

[0005] Furthermore, Japanese Patent Laid-Open Publication No. 2020-80949 discloses an image processing device that performs image processing on moving images acquired during a swallowing test. This image processing device calculates a correlation value between each of multiple swallowing test images included in the acquired moving images and a template image including the trachea and esophagus, and performs processing to recognize the region with the highest correlation value in each of the multiple swallowing test images as a region of interest.

[0006] JP 2018-15263 A JP 2020-80949 A

[0007] In the X-ray imaging diagnostic apparatus of JP 2018-15263 A, in order to detect the movement of the hyoid bone in an X-ray fluoroscopic image, it is conceivable to perform processing similar to the processing of the image processing apparatus of JP 2020-80949 A, in which a correlation value between the X-ray fluoroscopic image and a template image including the hyoid bone is calculated, and the region with the highest correlation value is recognized as a region of interest including the hyoid bone. However, when template matching is used to calculate a correlation value with a template image to track the hyoid bone (tracking point) in a swallowing test, if multiple regions similar to the template image exist in the X-ray fluoroscopic image, it is conceivable that the tracking point will be erroneously detected. For this reason, there is a demand for a tracking method that can suppress erroneous detection of the subject's hyoid bone (tracking point) and track the tracking point with high accuracy.

[0008] The present invention has been made to solve the above-mentioned problems, and one object of the present invention is to provide a radiographic imaging device, an image processing device, and an image processing method that are capable of suppressing erroneous detection of tracking points on a subject and tracking points with high accuracy.

[0009] In order to achieve the above object, the inventors of the present application conducted extensive research and discovered that by utilizing optical flow generated by a trained model that generates intermediate images (intermediate image generation AI (Artificial Intelligence)), tracking with fewer false positives of tracking points becomes possible. Based on this finding, a first aspect of the present invention provides a radiographic apparatus including a radiation irradiation unit that irradiates radiation onto a subject and a radiation detection unit that detects radiation that has passed through the subject, and is equipped with: a radiographic imaging unit that images the subject; a storage unit that stores a trained model that receives a first image and a second image captured at different times to generate a motion vector of an object in the first image to a second image, and generates an intermediate image at an intermediate point in time between the time the first image was captured and the time the second image was captured based on the motion vector; and a control unit, wherein the radiographic imaging unit captures the first radiographic image as the first image and the second radiographic image as the second image, the images captured at different times; and the control unit sets tracking points in the first radiographic image and inputs the first radiographic image and the second radiographic image into the trained model to obtain a motion vector of the object in the first radiographic image to the second radiographic image, and then sets tracking points in the second radiographic image based on the tracking points in the first radiographic image and the acquired motion vector.

[0010] In addition, an image processing device according to a second aspect of the present invention includes a trained model that receives a first image and a second image captured at different times, thereby generating a motion vector of the object in the first image to the second image, and generates an intermediate image at a time midway between the times at which the first image and the second image were captured, based on the motion vector; and a tracking point setting unit that sets a tracking point in a first radiographic image (a first image captured by a radiographic unit that captures images of the subject), and inputs the first radiographic image and a second radiographic image (a second image captured by the radiographic unit at a different time from the first radiographic image) to the trained model to obtain a motion vector of the object in the first radiographic image to the second radiographic image, and then sets a tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired motion vector.

[0011] In addition, an image processing method according to a third aspect of the present invention includes the steps of capturing a first radiographic image and a second radiographic image of the subject captured at different times using a radiographic imaging unit that images the subject by detecting radiation irradiated from a radiation irradiation unit and transmitted through the subject using a radiation detection unit; setting a tracking point in the first radiographic image; acquiring a movement vector of the object in the first radiographic image to the second radiographic image by inputting the first radiographic image as the first image and the second radiographic image as the second image into a trained model that inputs the first image and the second image captured at different times to generate a movement vector of the object in the first image to the second image, and generating an intermediate image at an intermediate time between the time points of the first image and the time points of the second image based on the movement vector; and, after acquiring the movement vector of the object in the first radiographic image to the second radiographic image, setting a tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired movement vector.

[0012] In the radiographic imaging apparatus according to the first aspect, the image processing apparatus according to the second aspect, and the image processing method according to the third aspect, a tracking point is set in a first radiographic image, and the first radiographic image and the second radiographic image are input into a trained model to obtain a motion vector of the object in the first radiographic image to the second radiographic image. Then, a tracking point is set in the second radiographic image based on the tracking point in the first radiographic image and the obtained motion vector. This allows a tracking point corresponding to the tracking point set in the first radiographic image to be set in the second radiographic image using the motion vector generated by the trained model that generates intermediate images. Therefore, unlike a case where template matching is used to calculate a correlation value with a template image to track a tracking point, even if multiple regions similar to the template image are present in the second radiographic image, erroneous detection of a corresponding tracking point in the second radiographic image can be suppressed. As a result, erroneous detection of a tracking point on the subject can be suppressed, enabling tracking points to be tracked with high accuracy.

[0013] FIG. 1 is a front view of an X-ray imaging apparatus according to an embodiment; FIG. 2 is a side view of an X-ray imaging apparatus according to an embodiment; FIG. 3 is a block diagram showing the control configuration of an X-ray imaging apparatus according to an embodiment; FIG. 4 is a schematic diagram showing a plurality of consecutive X-ray images in a moving image captured by an X-ray imaging apparatus according to an embodiment; FIG. 5 is a schematic diagram showing a first X-ray image in a moving image captured by an X-ray imaging apparatus according to an embodiment; FIG. 6 is a schematic diagram showing acquisition of a movement vector and an intermediate image by a movement vector acquisition unit of a control unit of an X-ray imaging apparatus according to an embodiment; FIG. 7 is a schematic diagram showing a movement vector map acquired by a movement vector acquisition unit of a control unit of an X-ray imaging apparatus according to an embodiment; FIG. 8 is a schematic diagram showing a moving image displayed on a display unit of an X-ray imaging apparatus according to an embodiment, and intermediate images and movement vectors acquired by a movement vector acquisition unit. FIG. 9 is a flowchart showing an image processing method performed in an X-ray imaging apparatus according to an embodiment.

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific embodiments of the present invention will be described with reference to the accompanying drawings.

[0015] The configuration of an X-ray imaging apparatus 100 according to an embodiment of the present invention will be described with reference to Figures 1 to 9. The X-ray imaging apparatus 100 is an example of the "radiation imaging apparatus" in the claims.

[0016] As shown in Figures 1 and 2, the X-ray imaging device 100 is an apparatus that captures X-ray images of organs such as bones and internal organs of a subject 200. The following describes a case in which the X-ray imaging device 100 captures a moving image 8 (see Figure 4) showing the movements of a hyoid bone 201 (see Figure 4), vertebrae 202 (see Figure 4), and mandible 203 (see Figure 4) during a swallowing test of the subject 200. The X-ray image is an example of a "radiographic image" in the claims. The hyoid bone 201, vertebrae 202, and mandible 203 are also examples of an "object" in the claims.

[0017] The X-ray imaging device 100 includes a tabletop 1, an X-ray imaging unit 2, a rotation mechanism unit 3, a support column 4, a base plate 5, a display unit 6 (see FIG. 3), and a control unit 7 (see FIG. 3). The tabletop 1 is a platform on which a subject 200 in a supine position is placed. The tabletop 1 is also a platform on which a subject 200 in an upright position comes into contact. The X-ray imaging unit 2 is configured to image the subject 200. Specifically, the X-ray imaging unit 2 includes an X-ray irradiator 21, an X-ray detector 22, and an attachment member 23. The X-ray imaging unit 2 is an example of a "radiation imaging unit" in the claims. The X-ray irradiator 21 is an example of a "radiation irradiator" in the claims. The X-ray detector 22 is an example of a "radiation detector" in the claims. The control unit 7 is an example of an "image processing device" in the claims.

[0018] The X-ray irradiator 21 is configured to irradiate the subject 200 with X-rays. The X-ray irradiator 21 has an X-ray source (not shown). The X-ray detector 22 is configured to detect X-rays that have passed through the subject 200. The X-ray detector 22 is a flat panel detector or the like. The X-ray detector 22 is configured to transmit a detection signal based on the detected X-rays to the controller 7. An X-ray image is thereby created in the controller 7 based on the detection signal. The mounting member 23 extends in a radial direction perpendicular to the rotation axis of the rotation mechanism 3. The top plate 1 and the X-ray detector 22 are attached to one end of the mounting member 23. The X-ray irradiator 21 is attached to the other end of the mounting member 23.

[0019] The rotation mechanism 3 is configured to rotate the mounting member 23. The rotation mechanism 3 is configured to rotate the mounting member 23 using the driving force of a drive source such as a motor to move the tabletop 1 to a predetermined position, such as a supine position or an upright position. The support column 4 is a member that supports the rotation mechanism 3 so that it can move up and down. The base plate 5 is installed on the floor with the support column 4 installed. The display unit 6 is configured to display X-ray images and the like created by the control unit 7. The display unit 6 is, for example, a liquid crystal display.

[0020] As shown in FIG. 3 , the control unit 7 is configured to control each unit of the X-ray imaging apparatus 100. The control unit 7 is communicably connected to each of the X-ray imaging unit 2, the rotation mechanism unit 3, the support column 4, and the display unit 6. The control unit 7 includes an X-ray image creation unit 71, an X-ray image storage unit 72, a processing unit 73, and a storage unit 74. The X-ray image creation unit 71 includes a GPU (Graphics Processing Unit) and memory. The X-ray image storage unit 72 includes a non-volatile storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The processing unit 73 includes a CPU (Central Processing Unit) and memory. The storage unit 74 includes a non-volatile storage medium such as an HDD or an SSD. A trained model 74a and a tracking point setting unit 74b are stored in the storage unit 74. Each of the trained model 74a and the tracking point setting unit 74b is a functional block of a control program that controls each unit of the X-ray imaging apparatus 100.

[0021] As shown in Fig. 4, the trained model 74a is configured to acquire movement vectors 9v (see Fig. 6) of the hyoid bone 201, vertebrae 202, mandible 203, and the like that move in the moving image 8. That is, the trained model 74a is a trained model that has learned to calculate the optical flow of a moving object from two X-ray images. Here, in the optical flow of the object, a movement vector 9v having the amount and direction of movement of the object is calculated.

[0022] Specifically, the trained model 74a is a trained model that uses an intermediate image generation AI called FILM (Frame Interpolation for Large Motion) provided by Google Research. FILM inputs two images captured at different times and performs resolution conversion processing and feature extraction processing for each image. FILM generates optical flow in the flow estimation process. FILM generates intermediate images in the fusion process. The optical flow, which is an intermediate product when generating this intermediate image, is extracted and used to track the tracking points.

[0023] In this way, the trained model 74a is configured to input a first image and a second image captured at different times, generate a movement vector 9v (see Figure 6) of the object in the first image to the second image, and generate an intermediate image at a point in time between the time the first image was captured and the time the second image was captured based on the movement vector 9v.

[0024] This enables the trained model 74a to generate intermediate images (see Figure 6) that interpolate between a series of X-ray images in the moving image 8 based on the movement vector 9v.

[0025] 4 and 5 , the tracking point setting unit 74b is configured to set each of the tracking points 9a and 9b set by the user in the first X-ray image 81 in the moving image 8 in the second X-ray image 82, the third X-ray image 83, the fourth X-ray image 84, etc. in the moving image 8. Here, the first X-ray image 81, the second X-ray image 82, the third X-ray image 83, and the fourth X-ray image 84 are X-ray images captured at different times. That is, in the following description, it is assumed that the first X-ray image 81 is the first image in the moving image 8. It is also assumed that the second X-ray image 82 is the image captured next after the first X-ray image 81 in the moving image 8, the third X-ray image 83 is the image captured next after the second X-ray image 82 in the moving image 8, and the fourth X-ray image 84 is the image captured next after the third X-ray image 83 in the moving image 8.

[0026] The processing of the control unit 7 by the trained model 74a and the tracking point setting unit 74b will be described in detail below.

[0027] (Tracking Process) As shown in FIG. 6 , the control unit 7 of this embodiment controls the setting of tracking points 9a and 9b in the second X-ray image 82, each of which corresponds to the tracking points 9a and 9b set in the first X-ray image 81, based on each of the tracking points 9a and 9b set in the first X-ray image 81 and a movement vector 9v acquired by the trained model 74a. The movement vector 9v is information acquired for generating an intermediate image by the trained model 74a. The control unit 7 controls the setting of tracking points 9a and 9b in the second X-ray image 82, each of which corresponds to the tracking points 9a and 9b set in the first X-ray image 81, using the movement vector 9v, which is information acquired for generating the intermediate image. This enables the tracking process of the tracking points 9a and 9b set in the first X-ray image 81.

[0028] Here, as an example, the tracking process is a process performed in a swallowing test performed by the X-ray imaging device 100. The swallowing test is, for example, a test for quantitatively evaluating pharyngeal function by fluoroscopically imaging the state of the pharynx when the subject 200 swallows a pseudo food product 300 containing a contrast agent. As a quantitative test, tests are performed on the amount of movement of the hyoid bone 201, the swallowing pressure, the swallowing speed, and the like during the swallowing movement of the subject 200. The tracking process is performed after acquiring a moving image 8 by fluoroscopically imaging the state of the pharynx when the subject 200 swallows the pseudo food product 300 using the X-ray imaging device 100.

[0029] In order to perform the quantitative examination described above, tracking points 9a and 9b are set. In the following description, it is assumed that the tracking points 9a are five points set on the hyoid bone 201 and in the vicinity of the hyoid bone 201 in the first X-ray image 81. It is also assumed that the tracking points 9b are points set on the left ends of the third and fifth vertebrae from the top of the head among the multiple vertebrae 202. Each of these tracking points 9a and 9b is set by the user. Here, the control unit 7 acquires the positions of pixels 81p in the first X-ray image 81 of the tracking points 9a and 9b set by the user.

[0030] The control unit 7 inputs the first X-ray image 81 and the second X-ray image 82 of the subject 200 captured using the X-ray imaging unit 2 into the learned model 74a, and controls the acquisition of the movement vectors 9v of each of the tracking points 9a and 9b that have moved from their positions in the first X-ray image 81 in the second X-ray image 82.

[0031] 7, the control unit 7 performs control to acquire, using the trained model 74a, a movement vector map 81m having a plurality of movement vectors 9v corresponding to the positions of pixels 81p in the first X-ray image 81. That is, the control unit 7 acquires, using the trained model 74a, all of the movement vectors 9v corresponding to the positions of the plurality of pixels 81p arranged in a matrix in the first X-ray image 81.

[0032] 7 , the hyoid bone 201 and the lower jaw 203 in the first X-ray image 81 have moved in the second X-ray image 82 from their positions in the first X-ray image 81. Therefore, the control unit 7 uses the trained model 74a to acquire a movement vector 9v having a movement amount and a movement direction, which is an angle, at the positions of pixels 81p corresponding to the hyoid bone 201 and the lower jaw 203 in the first X-ray image 81. The control unit 7 also acquires a movement vector 9v with a movement amount of 0 (zero vector) at the positions of pixels 81p corresponding to portions in the first X-ray image 81 that have not moved.

[0033] The motion vector map 81m is information including the motion vectors 9v of the moved portions and the motion vectors 9v of the non-moved portions, i.e., all the motion vectors 9v corresponding to the positions of the plurality of pixels 81p arranged in a matrix in the first X-ray image 81.

[0034] 8 , the control unit 7 uses the trained model 74a to extract movement vectors 9v corresponding to the tracking points 9a and 9b from the acquired movement vector map 81m, and then uses the tracking point setting unit 74b to set each of the tracking points 9a and 9b in the second X-ray image 82 based on the extracted movement vectors 9v. That is, the control unit 7 uses the trained model 74a to extract movement vectors 9v corresponding to each of the tracking points 9a and 9b from the multiple movement vectors 9v in the first X-ray image 81, and then uses the tracking point setting unit 74b to set each of the tracking points 9a and 9b in the second X-ray image 82 based on the extracted movement vectors 9v.

[0035] Specifically, the control unit 7 uses the trained model 74a to extract, from the acquired movement vector map 81m, movement vectors 9v corresponding to the positions of pixels 81p of each of the tracking points 9a and 9b in the first X-ray image 81. The control unit 7 also uses the tracking point setting unit 74b to calculate positions obtained by moving the positions of the pixels 81p of each of the tracking points 9a and 9b in the first X-ray image 81 by the movement amounts in the movement directions of the movement vectors 9v corresponding to each of the tracking points 9a and 9b, as the positions of the pixels 81p of the tracking points 9a and 9b in the second X-ray image 82 after movement.

[0036] That is, in the example shown in Fig. 8, five movement vectors 9v corresponding to tracking point 9a are extracted from the movement vector map 81m among the multiple movement vectors 9v acquired by the trained model 74a. Also, in the example shown in Fig. 8, two movement vectors 9v (movement amount 0 and movement direction 0 degrees) corresponding to tracking point 9b are extracted from the multiple movement vectors 9v acquired by the trained model 74a in the movement vector map 81m. As a result, in the example shown in Fig. 8, the tracking point setting unit 74b displays the calculated tracking points 9a and 9b after movement in the second X-ray image 82 in the moving image 8.

[0037] Furthermore, after calculating the positions of each of the tracking points 9a and 9b in the second X-ray image 82, the control unit 7 extracts, from the movement vector map 82m, movement vectors 9v corresponding to the positions of pixels in the second X-ray image 82 of each of the tracking points 9a and 9b in the second X-ray image 82, among the multiple movement vectors 9v from the second X-ray image 82 to the third X-ray image 83 acquired by the trained model 74a. The control unit 7 calculates, using the tracking point setting unit 74b, positions obtained by moving the positions of each of the pixels of the tracking points 9a and 9b in the second X-ray image 82 by the movement amounts in the movement directions of the movement vectors 9v corresponding to each of the tracking points 9a and 9b, as the positions of pixels 81p of the tracking points 9a and 9b in the third X-ray image 83 after movement.

[0038] Similarly, after calculating the positions of each of the tracking points 9a and 9b in the third X-ray image 83, the control unit 7 extracts, from the movement vector map 83m, movement vectors 9v corresponding to the positions of pixels in the third X-ray image 83 of each of the tracking points 9a and 9b in the third X-ray image 83, among the multiple movement vectors 9v from the third X-ray image 83 to the fourth X-ray image 84 acquired by the trained model 74a. The control unit 7 calculates, by the tracking point setting unit 74b, positions obtained by moving the positions of the pixels of the tracking points 9a and 9b in the third X-ray image 83 by the movement amounts in the movement directions of the movement vectors 9v corresponding to each of the tracking points 9a and 9b, as the positions of pixels 81p of the tracking points 9a and 9b in the fourth X-ray image 84 after movement.

[0039] The control unit 7 sequentially repeats the above-described processing, so that the tracking point setting unit 74b sets each of the tracking points 9a and 9b in the captured moving image 8.

[0040] In this way, the control unit 7 extracts the movement vectors 9v corresponding to each of the tracking points 9a and 9b set by the user in the first X-ray image 81 from among the multiple movement vectors 9v in the first X-ray image 81 acquired by the trained model 74a, and sets each of the tracking points 9a and 9b in the second X-ray image 82 by the tracking point setting unit 74b based on the extracted movement vectors 9v.

[0041] The control unit 7 also displays the acquired tracking points 9 a and 9 b in the moving image 8 on the display unit 6 in sequence.

[0042] 8 , the control unit 7 acquires a plurality of movement vectors 9v in the first X-ray image 81 using the trained model 74a, and generates a first intermediate image 81a that interpolates the movements of the hyoid bone 201, vertebrae 202, and mandible 203 in the moving image 8 based on the acquired plurality of movement vectors 9v. The first intermediate image 81a is an image taken at an intermediate point in time between the time when the first X-ray image 81 was captured and the time when the second X-ray image 82 was captured. The control unit 7 also acquires a plurality of movement vectors 9v in the second X-ray image 82 using the trained model 74a, and generates a second intermediate image 82a that interpolates the movements of the hyoid bone 201, vertebrae 202, and mandible 203 in the moving image 8 based on the acquired plurality of movement vectors 9v. The second intermediate image 82a is an image taken at an intermediate point in time between the time when the second X-ray image 82 was captured and the time when the third X-ray image 83 was captured. The control unit 7 also acquires a plurality of movement vectors 9v in the third X-ray image 83 using the trained model 74a, and generates a third intermediate image 83a that interpolates the movements of the hyoid bone 201 and the vertebrae 202 in the moving image 8 based on the acquired plurality of movement vectors 9v. The third intermediate image 83a is an image taken at an intermediate point in time between the capture of the third X-ray image 83 and the capture of the fourth X-ray image 84.

[0043] However, the control unit 7 does not display the generated first intermediate image 81a, second intermediate image 82a, and third intermediate image 83a, but instead sequentially displays each of the tracking points 9a and 9b in the moving image 8 on the display unit 6. This is because each of the first intermediate image 81a, second intermediate image 82a, and third intermediate image 83a is not an image actually captured by the X-ray imaging unit 2. Each of the first intermediate image 81a, second intermediate image 82a, and third intermediate image 83a is generated in order to obtain a motion vector map 81m (82m, 83m). In this way, the control unit 7 generates intermediate images including a first intermediate image 81a, a second intermediate image 82a, and a third intermediate image 83a that interpolate the movement of the hyoid bone 201 and the vertebrae 202 in the moving image 8 based on the acquired multiple movement vectors 9v, and sequentially displays tracking points 9a and 9b in the moving image 8 on the display unit 6 without displaying the generated intermediate images including the first intermediate image 81a, the second intermediate image 82a, and the third intermediate image 83a.

[0044] (Image Processing Method) With reference to FIG. 9, an image processing method performed by the X-ray imaging apparatus 100 will be described.

[0045] 9 , in step 401S, a moving image 8 of the subject 200 is captured. This results in the acquisition of a moving image 8 capturing the movements of objects such as the hyoid bone 201 and vertebrae 202 of the subject 200. This results in the acquisition of X-ray images captured at different times (for example, a first X-ray image 81 and a second X-ray image 82).

[0046] In step 402S, the setting of tracking points 9a and 9b by the user in the first X-ray image 81 in the moving image 8 is accepted. In step 403S, X-ray images (e.g., the first X-ray image 81 and the second X-ray image 82) captured before and after (at different capture times) in the moving image 8 are input to the trained model 74a, thereby acquiring a plurality of movement vectors 9v in the first X-ray image 81. In step 403S, a process of acquiring movement vectors 9v is performed for all X-ray images captured before and after in the moving image 8. In step 404S, movement vectors 9v for tracking points 9a and 9b are acquired from the plurality of movement vectors 9v. In step 405S, tracking points 9a and 9b in the moving image 8 are set based on the acquired movement vectors 9v, and the set tracking points 9a and 9b are displayed on the display unit 6. After step 405S, the image processing method ends.

[0047] (Effects of this embodiment) In this embodiment, the following effects can be obtained.

[0048] In the present embodiment, as described above, the control unit 7 sets the tracking point 9 a (tracking point 9 b) in the first X-ray image 81. The control unit 7 inputs the first X-ray image 81 and the second X-ray image 82 to the trained model 74 a to acquire a movement vector 9 v of an object, such as the hyoid bone 201 and the vertebrae 202, in the first X-ray image 81 to the second X-ray image 82, and then sets the tracking point 9 a (tracking point 9 b) in the second X-ray image 82 based on the tracking point 9 a (tracking point 9 b) in the first X-ray image 81 and the acquired movement vector 9 v. This makes it possible to set, in the second X-ray image 82, the tracking point 9 a (tracking point 9 b) corresponding to the tracking point 9 a (tracking point 9 b) set in the first X-ray image 81, using the movement vector 9 v generated by the trained model 74 a that generates intermediate images. Therefore, unlike the case where template matching is used to calculate a correlation value with a template image for tracking the tracking point 9 a (tracking point 9 b), even if a plurality of regions similar to the template image exist in the second X-ray image 82, it is possible to suppress erroneous detection of the corresponding tracking point 9 a (tracking point 9 b) in the second X-ray image 82. As a result, it is possible to suppress erroneous detection of the tracking point 9 a (tracking point 9 b) of the subject 200 and track the tracking point 9 a (tracking point 9 b) with high accuracy.

[0049] Furthermore, in this embodiment, as described above, the X-ray imaging unit 2 is configured to capture a moving image 8 of the subject 200. The control unit 7 sets the tracking point 9a (tracking point 9b) in the second X-ray image 82 based on the tracking point 9a (tracking point 9b) in the first X-ray image 81 and the movement vector 9v from the first X-ray image 81 to the second X-ray image 82, and then sets the tracking point 9a (tracking point 9b) in the third X-ray image 83 based on the tracking point 9a (tracking point 9b) and the movement vector 9v from the second X-ray image 82 to the third X-ray image 83, thereby tracking the movement of the tracking point 9a (tracking point 9b) in the captured moving image 8. As a result, since the movement vector 9v is obtained individually for each of the previous and next X-ray images (for example, the first X-ray image 81 and the second X-ray image 82) in the moving image 8, it is possible to accurately set the positions of the tracking points 9a and 9b in the subsequent X-ray image (for example, the second X-ray image 82). As a result, it is possible to accurately track the positions of the tracking points 9a and 9b in the moving image 8.

[0050] Furthermore, in this embodiment, as described above, the control unit 7 extracts the movement vectors 9v corresponding to each of the tracking points 9a and 9b from the plurality of movement vectors 9v in the first X-ray image 81 acquired by the trained model 74a, and sets the tracking point 9a (tracking point 9b) in the second X-ray image 82 based on the extracted movement vectors 9v. This makes it possible to acquire the movement vectors 9v that match the positions of the tracking points 9a and 9b from the plurality of movement vectors 9v, and therefore to accurately set the positions of the tracking points 9a and 9b in a subsequent X-ray image (for example, the second X-ray image 82).

[0051] Furthermore, in this embodiment, as described above, the control unit 7 uses the trained model 74a to acquire a motion vector map 81m having a plurality of motion vectors 9v corresponding to the positions of pixels 81p in the first X-ray image 81. The control unit 7 also extracts motion vectors 9v corresponding to each of the tracking points 9a and 9b from the motion vector map 81m, and sets the tracking point 9a (tracking point 9b) in the second X-ray image 82 based on the extracted motion vectors 9v. In this way, by using the motion vector map 81m, it is possible to acquire motion vectors 9v that match the positions of the tracking points 9a and 9b from among all motion vectors 9v that correspond to the positions of the pixels 81p in the first X-ray image 81. Therefore, tracking points can be tracked no matter where they are set in the first X-ray image 81.

[0052] Furthermore, in this embodiment, as described above, the control unit 7 sequentially displays each of the tracking points 9 a and 9 b in the video 8 on the display unit 6 based on the movement vector 9 v acquired by the trained model 74 a. This allows the user to visually recognize the movement of each of the tracking points 9 a and 9 b in the video 8, and therefore the user can determine the amount of movement of each of the tracking points 9 a and 9 b, as well as the movement speed of each of the tracking points 9 a and 9 b.

[0053] Furthermore, in this embodiment, as described above, the control unit 7 acquires a plurality of movement vectors 9v in the first X-ray image 81 using the trained model 74a. The control unit 7 generates a first intermediate image 81a, a second intermediate image 82a, and a third intermediate image 83a (intermediate images) that interpolate the movements of the hyoid bone 201, the vertebrae 202, and the mandible 203 (object) in the moving image 8 based on the acquired plurality of movement vectors 9v. The control unit 7 sequentially displays each of the tracking points 9a and 9b in the moving image 8 on the display unit 6 without displaying the generated first intermediate image 81a, the second intermediate image 82a, and the third intermediate image 83a (intermediate images). This allows only the X-ray images actually captured by the X-ray imaging unit 2 to be displayed on the display unit 6, allowing the user to visually recognize only the actual movements of the tracking points 9a and 9b.

[0054] Furthermore, in this embodiment, as described above, the control unit 7 extracts movement vectors 9v corresponding to each of the tracking points 9a and 9b set by the user in the first X-ray image 81, and sets the tracking points 9a (tracking points 9b) in the second X-ray image 82 based on the extracted movement vectors 9v. This makes it possible to obtain, from among the multiple movement vectors 9v, movement vectors 9v that match the positions of the tracking points 9a and 9b set by the user, and therefore to track the movement of a part where it is desired to know how the user moves.

[0055] Furthermore, in this embodiment, as described above, the tracking points 9 a and 9 b are points set on the hyoid bone 201 and the vertebrae 202, respectively, which are objects in the first X-ray image 81. This allows the movements of the hyoid bone 201 and the vertebrae 202 to be continuously acquired during the swallowing test, making it possible to easily perform quantitative analysis of the amount of movement of each of the hyoid bone 201 and the vertebrae 202 and the speed of movement of each of the hyoid bone 201 and the vertebrae 202 during the swallowing movement of the subject 200.

[0056] [Modifications] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the description of the above-mentioned embodiments, and further includes all modifications (modifications) within the meaning and scope of the claims.

[0057] For example, in the above embodiment, the tracking point 9 a is a point set on the hyoid bone 201 and its vicinity in the first X-ray image 81, and the tracking point 9 b is a point set on multiple vertebrae 202. However, the present invention is not limited to this. In the present invention, the tracking point may include at least one of a predetermined marker, a medical device inserted into the subject, and a treatment target region of the subject. In this case, the corresponding positions of the marker, the medical device inserted into the subject, and the treatment target region of the subject in the first radiographic image are automatically acquired using a method such as pattern matching. The control unit then extracts a movement vector corresponding to the predetermined tracking point in the first radiographic image, and tracks the movement of the tracking point based on the extracted movement vector.

[0058] In this manner, the tracking points include at least one of a marker, a medical device inserted into the subject, and a treatment target region of the subject, which are preset. Furthermore, the control unit extracts a movement vector corresponding to the preset tracking point in the first radiographic image, and then sets the tracking point in the second radiographic image based on the extracted movement vector. This allows the tracking point to be automatically acquired and set, thereby minimizing the workload on the user for tracking the tracking point.

[0059] In the above embodiment, the X-ray imaging device 100 (radiography device) captures moving images 8 showing the movements of the hyoid bone 201, vertebrae 202, and mandible 203 during a swallowing test of the subject 200. However, the present invention is not limited to this. In the present invention, tracking points may be tracked on moving images captured by the radiography device for quantitative analysis other than a swallowing test of the subject. For example, tracking points may be used to track a stent as a medical device, or to track a treatment target site (tumor) based on a metal marker as a marker. Furthermore, tracking points may be tracked on multiple images rather than on moving images captured by the radiography device.

[0060] In the above embodiment, the control unit 7 generates intermediate images that interpolate the movement of the hyoid bone 201 and vertebrae 202 (objects) in the moving image 8 based on the acquired plurality of movement vectors 9v, and sequentially displays the tracking points 9a and 9b in the moving image 8 on the display unit 6 without displaying the generated intermediate images, but the present invention is not limited to this. In the present invention, the control unit may generate intermediate images that interpolate the movement of the object in the moving image based on the acquired plurality of movement vectors, and display both the captured X-ray image and the generated intermediate images, and sequentially display the tracking points in the moving image on the display unit.

[0061] In the above embodiment, the first X-ray image 81 (first radiographic image) is described as the first image in the moving image sequence 8, but the present invention is not limited to this. In the present invention, if the first radiographic image is not the first image in the moving image sequence but an intermediate image, the image following the first radiographic image becomes the second radiographic image, and the image following the second radiographic image becomes the third radiographic image.

[0062] In the above embodiment, for convenience of explanation, the control processing of the control unit 7 is explained using a flow-driven flowchart in which processing is performed sequentially according to a processing flow, but the present invention is not limited to this. In the present invention, the control processing of the control unit may be performed by event-driven processing in which processing is performed on an event-by-event basis. In this case, the control processing may be performed completely event-driven, or may be performed in a combination of event-driven and flow-driven processing.

[0063] Aspects It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0064] a storage unit that stores a trained model that receives a first image and a second image captured at different times to generate a motion vector of an object in the first image to the second image, and generates an intermediate image at an intermediate point in time between the time of capturing the first image and the time of capturing the second image based on the motion vector; and a control unit, wherein the radiation imaging unit captures a first radiographic image as the first image and a second radiographic image as the second image captured at different times, and the control unit sets a tracking point in the first radiographic image and inputs the first radiographic image and the second radiographic image to the trained model to obtain the motion vector of the object in the first radiographic image to the second radiographic image, and then sets the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the obtained motion vector.

[0065] (Item 2) The radiation imaging device according to item 1, wherein the radiation imaging unit is configured to capture a moving image of the subject, and the control unit is configured to set the tracking point in the captured moving image by sequentially repeating a process of setting the tracking point in the second radiation image based on the tracking point in the first radiation image and the movement vector from the first radiation image to the second radiation image, and then setting the tracking point in the third radiation image based on the tracking point in the second radiation image and the movement vector from the second radiation image to the third radiation image.

[0066] (Item 3) In the radiographic apparatus according to Item 1, the control unit is configured to extract the movement vector corresponding to the tracking point from a plurality of movement vectors in the first radiographic image acquired by the trained model, and to set the tracking point in the second radiographic image based on the extracted movement vector.

[0067] (Item 4) In the radiographic apparatus according to Item 3, the control unit is configured to obtain, using the trained model, a motion vector map having a plurality of motion vectors corresponding to positions of pixels in the first radiographic image, extract the motion vector corresponding to the tracking point from the motion vector map, and set the tracking point in the second radiographic image based on the extracted motion vector.

[0068] (Item 5) The radiation imaging apparatus according to Item 2, wherein the control unit is configured to sequentially display the tracking points in the moving image on a display unit based on the movement vector acquired by the trained model.

[0069] (Item 6) The radiographic imaging device according to Item 5, wherein the control unit is configured to acquire a plurality of the movement vectors in the first radiographic image using the trained model, generate the intermediate image that interpolates the movement of the object in the moving image based on the acquired plurality of movement vectors, and sequentially display the tracking points in the moving image on the display unit without displaying the generated intermediate image.

[0070] (Item 7) In the radiographic imaging device according to Item 3, the control unit is configured to extract the movement vector corresponding to the tracking point set by a user in the first radiographic image, and to set the tracking point in the second radiographic image based on the extracted movement vector.

[0071] (Item 8) The radiographic imaging device described in Item 3, wherein the tracking points include at least one of a marker, a medical device inserted into the subject, and a treatment target region of the subject that is set in advance, and the control unit is configured to extract the movement vector corresponding to the tracking points set in advance in the first radiographic image, and then set the tracking points in the second radiographic image based on the extracted movement vector.

[0072] (Item 9) The radiographic imaging apparatus according to item 1, wherein the tracking points are points set on each of the vertebrae and the hyoid bone as the object in the first radiographic image.

[0073] (Item 10) An image processing device comprising: a trained model that, by inputting a first image and a second image captured at different times, generates a movement vector of an object in the first image to the second image, and generates an intermediate image at an intermediate time between the time of capturing the first image and the time of capturing the second image based on the movement vector; and a tracking point setting unit that sets a tracking point in a first radiographic image as the first image captured by a radiographic unit that captures images of a subject, and inputs the first radiographic image and a second radiographic image as the second image captured by the radiographic unit at a different time from the first radiographic image to the trained model to obtain the movement vector of the object in the first radiographic image to the second radiographic image, and then sets the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired movement vector.

[0074] (Item 11) An image processing method comprising the steps of capturing a first radiographic image and a second radiographic image of the subject captured at different times using a radiographic imaging unit that images the subject by detecting radiation irradiated from a radiation irradiation unit and transmitted through the subject using a radiation detection unit; setting a tracking point in the first radiographic image; acquiring the movement vector of the object in the first radiographic image to the second radiographic image by inputting the first image and the second image as the first image and the second image into a trained model that generates an intermediate image at an intermediate time between the time of capturing the first image and the time of capturing the second image based on the movement vector; and after acquiring the movement vector of the object in the first radiographic image to the second radiographic image, setting the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired movement vector.

[0075] DESCRIPTION OF SYMBOLS 2 X-ray imaging unit (radiation imaging unit) 6 Display unit 7 Control unit (image processing device) 8 Moving image 9a, 9b Tracking point 9v Movement vector 21 X-ray irradiation unit (radiation irradiation unit) 22 X-ray detection unit (radiation detection unit) 74 Memory unit 74a Learned model 74b Tracking point setting unit 81 First X-ray image (first radiation image) 81a First intermediate image 81m, 82m, 83m Movement vector map 81p Pixel 82 Second X-ray image (second radiation image) 82a Second intermediate image (intermediate image) 83 Third X-ray image (third radiation image) 83a Third intermediate image (intermediate image) 100 X-ray imaging device (radiation imaging device) 200 Subject 201 Hyoid bone (object) 202 Vertebra (object) 203 Mandible (object)

Claims

1. A radiographic apparatus comprising: a radiographic imaging unit that images the subject, including a radiation irradiation unit that irradiates radiation onto the subject and a radiation detection unit that detects radiation that has passed through the subject; a memory unit that stores a trained model that receives a first image and a second image captured at different times to generate a movement vector of an object in the first image to the second image, and generates an intermediate image at an intermediate point in time between the time the first image was captured and the time the second image was captured based on the movement vector; and a control unit, wherein the radiographic imaging unit captures a first radiographic image as the first image and a second radiographic image as the second image, the images captured at different times; and the control unit sets a tracking point in the first radiographic image and inputs the first radiographic image and the second radiographic image into the trained model to obtain the movement vector of the object in the first radiographic image to the second radiographic image, and then sets the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the obtained movement vector.

2. The radiographic imaging device according to claim 1, wherein the radiographic imaging unit is configured to capture moving images of the subject, and the control unit is configured to set the tracking point in the captured moving images by sequentially repeating a process of setting the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the movement vector from the first radiographic image to the second radiographic image, and then setting the tracking point in the third radiographic image based on the tracking point in the second radiographic image and the movement vector from the second radiographic image to the third radiographic image.

3. The radiographic imaging device of claim 1, wherein the control unit is configured to extract the movement vector corresponding to the tracking point from the plurality of movement vectors in the first radiographic image acquired by the trained model, and to set the tracking point in the second radiographic image based on the extracted movement vector.

4. The radiographic imaging device of claim 3, wherein the control unit is configured to obtain a motion vector map having a plurality of motion vectors corresponding to pixel positions in the first radiographic image using the trained model, extract the motion vector corresponding to the tracking point from the motion vector map, and set the tracking point in the second radiographic image based on the extracted motion vector.

5. A radiographic imaging device as described in claim 2, wherein the control unit is configured to sequentially display the tracking points in the moving image on a display unit based on the movement vector obtained by the trained model.

6. The radiographic imaging device of claim 5, wherein the control unit is configured to acquire a plurality of the movement vectors in the first radiographic image using the trained model, generate an intermediate image that interpolates the movement of the object in the moving image based on the acquired plurality of movement vectors, and sequentially display the tracking points in the moving image on the display unit without displaying the generated intermediate image.

7. The radiographic imaging device of claim 3, wherein the control unit is configured to extract the movement vector corresponding to the tracking point set by the user in the first radiographic image, and to set the tracking point in the second radiographic image based on the extracted movement vector.

8. The radiographic imaging device of claim 3, wherein the tracking points include at least one of a marker, a medical device inserted into the subject, and a region of the subject to be treated, which are set in advance; and the control unit is configured to extract the movement vector corresponding to the tracking points set in advance in the first radiographic image, and then set the tracking points in the second radiographic image based on the extracted movement vector.

9. The radiographic imaging device according to claim 1, wherein the tracking points are points set on each of the vertebrae and the hyoid bone as the object in the first radiographic image.

10. An image processing device comprising: a trained model that receives a first image and a second image captured at different times, and generates a movement vector of an object in the first image to the second image, and generates an intermediate image at a time midway between the time the first image was captured and the time the second image was captured based on the movement vector; and a tracking point setting unit that sets a tracking point in a first radiographic image as the first image captured by a radiographic unit that captures images of a subject, and inputs the first radiographic image and a second radiographic image as the second image captured by the radiographic unit at a different time from the first radiographic image to the trained model, thereby obtaining the movement vector of the object in the first radiographic image to the second radiographic image, and then sets the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired movement vector.

11. An image processing method comprising the steps of: capturing a first radiographic image and a second radiographic image of the subject captured at different times using a radiographic imaging unit that images the subject by detecting radiation irradiated from a radiation irradiation unit and transmitted through the subject using a radiation detection unit; setting a tracking point in the first radiographic image; acquiring the movement vector of the object in the first radiographic image to the second radiographic image by inputting the first image and the second image as the first image into a trained model that inputs the first image and the second image captured at different times, and generates an intermediate image at an intermediate time between the times when the first image and the second image were captured based on the movement vector; and after acquiring the movement vector of the object in the first radiographic image to the second radiographic image, setting the tracking point in the second radiographic image based on the tracking point in the first radiographic image and the acquired movement vector.

Images