Image processing device, radiography system, image processing method, and program

Abstract

Regardless of whether the subject's movements are periodic or aperiodic, the system is designed to enable radiography at a time when the subject is in a state suitable for radiography. [Solution] The system includes an optical image acquisition unit 210 that acquires an optical image obtained by optically photographing a subject to be subjected to radiography using radiation, and a body motion prediction unit 220 that predicts a specific state of the subject suitable for radiography by inputting the optical image acquired by the optical image acquisition unit 210 into a trained model 221, the trained model 221 being a trained model that has been trained based on a state prior to the specific state of the subject suitable for radiography.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, a radiation imaging system, an image processing method, and a program.

Background Art

[0002] In radiation imaging for medical examinations, various parts of a subject may be imaged. Among these various parts, there are parts that are difficult to image. One of these difficult-to-image parts is chest imaging of newborns, infants, or unconscious subjects.

[0003] In chest imaging, in the case of a general subject, the subject is asked to hold their breath in an inhalation state for imaging. However, in the case of the above-mentioned newborns, infants, or unconscious subjects, since communication is impossible, it is not possible to ask them to hold their breath, and the imager visually recognizes the breathing state of the subject and performs imaging at the timing of inhalation. Therefore, in the case of the above-mentioned newborns, infants, or unconscious subjects, it is necessary to match the subject state and imaging timing, which are not the same as in the imaging of general subjects, making the imaging difficult. As a conventional technique, Patent Document 1 proposes a technique of synchronizing imaging so that imaging can be performed at a specific phase based on the periodic movement of a subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, if the subject is a newborn or infant, for example, they may cry due to anxiety from being forced into a specific imaging position or from being in an unfamiliar environment, causing their movements to become irregular. In such cases, techniques that utilize periodicity, as described in Patent Document 1, have the challenge of being unable to synchronize the imaging timing with the subject's state, which is suitable for radiography.

[0006] This disclosure has been made in view of these challenges and aims to enable radiography to be performed at a time when the subject is in a state suitable for radiography, regardless of whether the subject's movement is periodic or aperiodic. [Means for solving the problem]

[0007] The image processing apparatus of the present disclosure comprises an acquisition means for acquiring an optical image obtained by optically photographing a subject to be subjected to radiography using radiation, and a prediction means for predicting a specific state of the subject suitable for radiography by inputting the optical image acquired by the acquisition means into a trained model, wherein the trained model is a trained model that has been trained based on states prior to the specific state. [Effects of the Invention]

[0008] According to this disclosure, regardless of whether the subject's movement is periodic or aperiodic, radiography can be performed at a timing that is suitable for the subject's state. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of a schematic configuration of a radiography system according to the first embodiment. [Figure 2] This figure shows an example of a schematic configuration of an image processing device according to the first embodiment. [Figure 3] This flowchart shows an example of the processing procedure for an image processing method using an image processing apparatus according to the first embodiment. [Figure 4]This flowchart shows a first embodiment and is an example of a detailed processing procedure for the intake timing detection process in step S104 of Figure 3. [Figure 5] This figure shows a first embodiment and provides an example of displaying the result of deriving the intake timing by step S203 in Figure 4 and the result of determining whether or not it is the intake timing by step S204 in Figure 4. [Figure 6] This flowchart shows a first embodiment and is an example of a detailed processing procedure for the intake timing derivation process in step S203 of Figure 4. [Figure 7] This figure shows a first embodiment and is a first example of training data used when training a trained model. [Figure 8] This figure shows the first embodiment and a second example of training data used when training a trained model. [Figure 9] This figure shows a first embodiment and an example of a machine learning model that can be applied to a pre-trained model. [Figure 10] This flowchart shows an example of the processing procedure for the image processing method using the image processing apparatus 150 according to the second embodiment. [Modes for carrying out the invention]

[0010] The embodiments for implementing this disclosure will be described below with reference to the drawings. However, in the embodiments of this disclosure described below, the relative positions of each component are arbitrary and can be changed as needed. Furthermore, in this specification, X-rays are preferred as the radiation relating to this disclosure, but the invention is not limited to X-rays, and includes, for example, particle radiation such as alpha rays, beta rays, gamma rays, particle beams, proton beams, heavy ion beams, and meson beams.

[0011] Furthermore, in this specification, a machine learning model refers to a learning model based on a machine learning algorithm. Specific machine learning algorithms include nearest neighbors, naive Bayes, decision trees, and support vector machines. Another specific machine learning algorithm is deep learning, which uses neural networks to generate features and connection weights for learning. Algorithms using decision trees include methods employing gradient boosting, such as LightGBM and XGBoost. Appropriately, any of the above algorithms can be used and applied to the embodiments of this disclosure described below.

[0012] In this specification, "training data" refers to "training data," which consists of pairs of input and output data. The output data of the training data is also called "ground truth data." Furthermore, in this specification, "trained model" refers to a machine learning model that follows any machine learning algorithm, such as deep learning, and has been trained (learned) in advance using appropriate training data. However, while a trained model has been trained in advance using appropriate training data, this does not mean that it cannot be trained further; additional training is possible. Additional training can be performed even after the device that processes using the trained model has been installed at the user's site.

[0013] (First embodiment) First, let me describe the first embodiment.

[0014] In the first embodiment, a radiation imaging system will be described that can predict the movement (body movement) of a subject and perform radiation irradiation on the subject at a specific state suitable for radiation imaging. In the first embodiment, as an example, the specific state of the subject suitable for radiation imaging is the inhalation state of the lungs of a subject who is a newborn or an infant. The specific state of the subject suitable for radiation imaging is preferably the maximum inhalation state in a predetermined period in the lungs of the subject, but the present disclosure is not limited to this. For example, an inhalation state that is not the maximum inhalation state but is close to the maximum inhalation state can also be included. Further, the specific state of the subject suitable for radiation imaging is preferably a state that occurs aperiodically in the subject, but the present disclosure is not limited to this. For example, a state that occurs periodically in the subject can also be included.

[0015] FIG. 1 is a diagram showing an example of the schematic configuration of a radiation imaging system 100 according to the first embodiment. In FIG. 1, the state where the subject H is in a lying body position is shown, but the subject H may be, for example, in a standing body position or a sitting body position. Also, there may be an imaging table used to support the subject H.

[0016] As shown in FIG. 1, the radiation imaging system 100 includes a radiation control device 110, a radiation generation device 120, a radiation flat detector 130, an optical camera device 140, an image processing device 150, a user input device 160, and a user output device 170. As shown in FIG. 1, the image processing device 150 is communicably connected to the radiation control device 110, the radiation flat detector 130, the optical camera device 140, the user input device 160, and the user output device 170. That is, the image processing device 150 is configured to be able to control each of the communicably connected devices and communicate with each of these devices.

[0017] The radiation control device 110 controls the radiation generation device 120 based on the control of the image processing device 150. Specifically, the radiation control device 110 controls the irradiation of the radiation R by the radiation generation device 120.

[0018] The radiation generating device 120 includes, for example, not only an X-ray tube that generates radiation R, but also a collimator, a collimator lamp, etc., and can irradiate radiation (radiation beam) R based on the control of the radiation control device 110. The radiation (radiation beam) R irradiated from the radiation generating device 120 passes through the subject H while being attenuated and enters the radiation flat panel detector 130.

[0019] The radiation flat panel detector 130 detects the incident radiation R, generates an electrical signal corresponding to the dose of the detected radiation R as radiation image data (hereinafter simply referred to as "radiation image"), and transmits the generated radiation image to the image processing device 150. The radiation flat panel detector 130 may be any radiation detector that detects radiation R and outputs a radiation image related to the corresponding electrical signal. For example, it can be configured using an FPD (Flat Panel Detector), etc. The radiation flat panel detector 130 may be an indirect conversion type detector that once converts radiation R into visible light using a scintillator or the like and then converts the visible light into an electrical signal by a photoelectric conversion element. Further, the radiation flat panel detector 130 may be a direct conversion type detector that directly converts the incident radiation R into an electrical signal.

[0020] In the present embodiment, the radiation control device 110, the radiation generating device 120, and the radiation flat panel detector 130 are communicably connected to the image processing device 150, and constitute a radiation imaging device that performs radiation imaging of the subject H using radiation R.

[0021] The optical camera device 140 is an example of an optical device that, based on the control of the image processing device 150, optically photographs the subject H that is subjected to radiation imaging using radiation R and generates optical image data of the subject H (hereinafter simply referred to as "optical image"). In this embodiment, the range in which the optical camera device 140 optically photographs the subject H includes the range in which the subject H is subjected to radiation imaging and is wider than the range in which the subject H is subjected to radiation imaging. The optical camera device 140 then transmits the optical image obtained by optical photography to the image processing device 150. The optical camera device 140 may have any known configuration, and may be configured as a camera device capable of shooting video, such as a video camera, or as a camera device that only takes still images. Furthermore, the optical camera device 140 may be configured to perform optical photography using visible light, or it may be configured to perform optical photography using invisible light other than visible light, such as infrared light.

[0022] The image processing device 150 acquires various images, including radiation images generated by the radiation plane detector 130 and optical images generated by the optical camera device 140, and performs image processing and analysis on the acquired images. The image processing device 150 also comprehensively controls the operation of the radiation imaging system 100 by controlling each device of the radiation imaging system 100 based on information input from, for example, the user input device 160. For example, the image processing device 150 can send a signal to the radiation control device 110 regarding the timing of the start of radiation R irradiation. The image processing device 150 can also communicate with users, such as the person performing the imaging, via the user input device 160 and the user output device 170.

[0023] The user input device 160 is a device for the user to input operations to the image processing device 150, and includes, for example, a keyboard, mouse, touch panel, console monitor, and radiation irradiation switch.

[0024] The user output device 170 is a device for outputting the processing results and status of the image processing device 150 to the user, and includes, for example, monitors including a console monitor, display devices such as indicator lamps, and audio output devices such as speakers.

[0025] Figure 2 is a diagram showing an example of the schematic configuration of the image processing apparatus 150 according to the first embodiment. That is, it is a diagram showing an example of the schematic configuration of the image processing apparatus 150 shown in Figure 1.

[0026] As shown in Figure 2, the image processing device 150 includes a user input device communication unit 201, a user output device communication unit 202, a processor 203, a calculation memory unit 204, a memory unit 205, and a bus 206. Furthermore, as shown in Figure 2, the image processing device 150 also includes an optical image acquisition unit 210, a body motion prediction unit 220, a radiation control device communication unit 230, and a radiation image acquisition unit 240. Each component of the image processing device 150 is connected via the bus 206 so as to be able to communicate with each other and send and receive data.

[0027] The user input device communication unit 201 communicates with the user input device 160, which is used by the user to input operations to the image processing device 150. The user can switch whether or not to take images using the motion prediction unit 220 via the user input device communication unit 201.

[0028] The user output device communication unit 202 communicates with the user output device 170, which outputs (displays, etc.) the processing results and status of the image processing device 150 to the user. The user output device communication unit 202 can display information related to shooting, various images, buttons, sliders, and other arbitrary displays or GUIs to the user output device 170 to accept user operation input.

[0029] The processor 203 is a processor that comprehensively controls the operation of the image processing device 150. The processor 203 uses the arithmetic memory unit 204 to perform overall control of the image processing device 150 based on operation input information from the user output device communication unit 202 and programs and various information stored in the memory unit 205. The processor 203 may include, for example, a CPU (Central Processing Unit), an MPU (Microprocessing Unit), and / or a GPU (Graphics Processing Unit). The processor 203 may also include, for example, a DSP (Digital Signal Processor), a DFP (Dataflow Processor), and / or an NPU (Neural Processing Unit).

[0030] The arithmetic memory unit 204 is composed of memory and is used for storing temporary information (including data and images).

[0031] The storage unit 205 stores programs and various information (including data and images) necessary for various control and processing operations performed by the image processing device 150 (especially the processor 203). The storage unit 205 also stores various information (including data and images) obtained through various control and processing operations performed by the image processing device 150 (especially the processor 203). Furthermore, the storage unit 205 stores patient information related to subject H, imaging condition information, and various parameters set by the user. The storage unit 205 can be configured using any storage medium, such as a magnetic disk, optical disk, or memory.

[0032] The image processing device 150 can be configured as a computer equipped with a processor 203 and memory. In this case, the image processing device 150 may be configured as a general-purpose computer, or as a computer dedicated to the radiography system 100. Furthermore, the image processing device 150 may be configured as a personal computer (PC), and the PC may be a desktop PC, a notebook PC, a tablet PC (portable information terminal device), etc. In addition, the image processing device 150 may be configured as a cloud-type computer in which some components are located on external devices.

[0033] Next, the optical image acquisition unit 210, the motion prediction unit 220, the radiation control device communication unit 230, and the radiation image acquisition unit 240 will be described. The optical image acquisition unit 210, the motion prediction unit 220, the radiation control device communication unit 230, and the radiation image acquisition unit 240 may be composed of software modules obtained by the processor 203 executing a program stored in the memory unit 205. Furthermore, the optical image acquisition unit 210, the motion prediction unit 220, the radiation control device communication unit 230, and the radiation image acquisition unit 240 may be composed of circuits or independent devices that perform specific functions, such as ASICs.

[0034] The optical image acquisition unit 210 is an optical image acquisition means that controls the optical camera device 140 and acquires optical images obtained when the optical camera device 140 optically photographs the subject H from the optical camera device 140. Alternatively, the optical image acquisition unit 210 may acquire optical images obtained by optically photographing the subject H, which is subjected to radiation imaging using radiation R, from an external storage device or optical equipment indirectly connected to the image processing device 150 via any network. Alternatively, the optical image acquisition unit 210 may acquire optical images stored in the storage unit 205.

[0035] The motion prediction unit 220 is a prediction means that predicts a specific state of subject H suitable for radiography by inputting the optical image acquired by the optical image acquisition unit 210 into a trained model 221. Here, the trained model 221 is a trained model that has been trained based on a state of subject H prior to the above-mentioned specific state (where the state prior to the above-mentioned specific state can also be interpreted as a state that is a precursor to the above-mentioned specific state). Here, as an example of a state that is a precursor to the above-mentioned specific state, it has been given that it can be interpreted as a state prior to the maximum inspiratory state in the lungs of subject H over a predetermined period of time, which is an example of the specific state, but this disclosure is not limited to this. For example, a state that is a precursor to the above-mentioned specific state may also include a state prior to the maximum inspiratory state in the lungs of subject H over a predetermined period of time, and which is also a state in which subject H has entered inhalation.

[0036] The radiation control device communication unit 230 communicates with the radiation control device 110, which controls the irradiation of radiation R by the radiation generator 120. For example, the radiation control device communication unit 230 may be configured as an output means that outputs (transmits) a signal to the radiation control device 110 regarding the start of irradiation of radiation R based on the prediction result of a specific state of the subject H by the body motion prediction unit 220.

[0037] The radiation image acquisition unit 240 communicates with the radiation control device 110 and controls the radiation plane detector 130 to control the radiation imaging of the subject H and acquires a radiation image of the subject H from the radiation plane detector 130. Furthermore, if it is necessary to predict the motion of the subject H from the imaging information stored in the storage unit 205 when performing radiation imaging, the radiation image acquisition unit 240 may instruct the radiation control device 110 on the timing of the imaging based on the motion prediction result of the subject H by the motion prediction unit 220.

[0038] Figure 3 is a flowchart showing an example of the processing procedure of the image processing method by the image processing device 150 according to the first embodiment. Specifically, Figure 3 is a flowchart showing the processing procedure of the image processing device 150 up to the point of performing radiographic imaging of the subject H.

[0039] First, in step S101 of Figure 3, the processor 203 acquires imaging information related to radiography, for example, via the user input device communication unit 201. Here, the imaging information includes, for example, information on the imaging site and direction, as well as the age of the subject H and setting information on whether or not to synchronize with the inhalation timing.

[0040] Next, in step S102 of Figure 3, the motion prediction unit 220 determines whether or not to synchronize the radiography of subject H with the inhalation timing, based on the imaging information acquired in step S101. Here, an example has been described in which the motion prediction unit 220 makes the decision in step S102 based on setting information included in the imaging information regarding whether or not to synchronize with the inhalation timing. However, it is also possible to have a correspondence table in advance that indicates whether or not to synchronize with the inhalation timing based on the imaging information and make the decision from there.

[0041] In step S102 of Figure 3, if the motion prediction unit 220 determines that the radiography of subject H should not be synchronized with the inhalation timing (S102 / No), the process proceeds to step S103.

[0042] When the process proceeds to step S103 in Figure 3, the user input device communication unit 201 receives input from the user to the radiation irradiation switch of the user input device 160. Then, when input is received from the user to the radiation irradiation switch of the user input device 160 in step S103, the process proceeds to step S106. Subsequently, proceeding to step S106 in Figure 3, the radiation control device communication unit 230 outputs a signal to the radiation control device 110 regarding the start of radiation irradiation R in order to irradiate subject H with radiation R in order to perform radiography of subject H.

[0043] Furthermore, in step S102 of Figure 3, if the body motion prediction unit 220 determines that the radiography of subject H should be synchronized with the inhalation timing (S102 / Yes), the process proceeds to step S104.

[0044] When the process proceeds to step S104 in Figure 3, the motion prediction unit 220 inputs the optical image acquired by the optical image acquisition unit 210 into the trained model 221 to predict and detect the timing of the inhalation state, which is a specific state of the subject H suitable for radiography.

[0045] Next, in step S105 of Figure 3, the body motion prediction unit 220 determines whether the user synchronized the operation input to the radiation irradiation switch of the user input device 160 with the timing of the intake state detected in step S104.

[0046] In step S105 of Figure 3, if the motion prediction unit 220 determines that the user did not synchronize their operation input to the radiation irradiation switch with the timing of the intake state detected in step S104 (S105 / No), the process returns to step S104.

[0047] Furthermore, in step S105 of Figure 3, if the body motion prediction unit 220 determines that the user has synchronized their operation input to the radiation irradiation switch with the timing of the intake state detected in step S104 (S105 / Yes), the process proceeds to step S106. Subsequently, proceeding to step S106 in Figure 3, the radiation control device communication unit 230 outputs a signal to the radiation control device 110 regarding the start of radiation irradiation R in order to irradiate subject H with radiation R in order to perform radiography of subject H.

[0048] Once the process in step S106 of Figure 3 is completed, the process shown in the flowchart in Figure 3 is terminated.

[0049] In the flowchart shown in Figure 3, the sequence of steps S104 and S105 is just one example; the processing procedure is not limited as long as the objective of synchronizing the timing of the detected intake state with the user's radiation R input is achieved.

[0050] Figure 4 shows a first embodiment and is a flowchart illustrating an example of a detailed processing procedure for the intake timing detection process in step S104 of Figure 3.

[0051] When the process in step S104 in Figure 3 begins, first, in step S201 in Figure 4, the optical image acquisition unit 210 acquires an optical image from the optical camera device 140 obtained by the optical camera device 140 optically photographing the subject H.

[0052] Next, in step S202 of Figure 4, the optical image acquisition unit 210 stores one or more optical images acquired in step S201 in the storage unit 205. Here, the number of optical images to be stored is at least equal to or greater than the number required for the intake timing derivation process in the subsequent step S203.

[0053] Next, in step S203 of Figure 4, the motion prediction unit 220 inputs the optical images accumulated in step S202 into the trained model 221 to derive the timing of the inhalation state (inhalation timing), which is a specific state of the subject H suitable for radiography.

[0054] Next, in step S204 of Figure 4, the body motion prediction unit 220 determines whether or not the intake timing was derived in step S203.

[0055] In step S204 of Figure 4, if the body motion prediction unit 220 determines that the intake timing is not as derived in step S203 (S204 / No), the process returns to step S201 and repeats the processing from step S201 onwards.

[0056] Furthermore, in step S204 of Figure 4, if the body motion prediction unit 220 determines that the intake timing was derived in step S203 (S204 / Yes), the process of the flowchart shown in Figure 4 is terminated.

[0057] Furthermore, when deriving the inhalation timing in S203 of Figure 4, if the inhalation timing flag or numerical value is output as ON / OFF considering the time lag between the transmission of the signal regarding the start of radiation R irradiation and the actual irradiation of radiation R, the following processing is performed, for example. Specifically, if the body motion prediction unit 220 outputs a flag or numerical value indicating inhalation timing, it makes an affirmative judgment (S204 / Yes) in step S204 of Figure 4 and terminates the processing of the flowchart shown in Figure 4. If the body motion prediction unit 220 does not output a flag or numerical value indicating inhalation timing, it makes a negative judgment (S204 / No) in step S204 of Figure 4 and returns to step S201. Here, when deriving the inhalation timing in S203 of Figure 4, if the confidence level is output along with the inhalation timing flag or numerical value, the system determines that it is inhalation timing if the confidence level falls within an acceptable range. The acceptable range of confidence level here can be specified by a pre-set threshold, etc.

[0058] Furthermore, when deriving the intake timing in S203 of Figure 4, if an output that does not take into account the time lag mentioned above, such as the time or number of frames until the intake timing, is output, the following processing is performed, for example. Specifically, the motion prediction unit 220 determines that it is the intake timing if the output result and the time lag mentioned above are within an acceptable range, and makes an affirmative judgment (S204 / Yes) in step S204 of Figure 4, ending the processing of the flowchart shown in Figure 4. Also, the motion prediction unit 220 determines that it is not the intake timing if the output result and the time lag mentioned above are not within an acceptable range, and makes a negative judgment (S204 / No) in step S204 of Figure 4, returning to step S201. Here, the acceptable range may be less than ±1 frame or an equivalent time based on the frame rate, or it may be a pre-set threshold.

[0059] Furthermore, in step S204, a negative judgment (S204 / No) is made, which includes not only cases where the inhalation timing is not as derived in step S203, but also cases where the inhalation timing could not be derived in step S203. Therefore, in step S204, a negative judgment (S204 / No) is made, for example, when only a part of subject H is visible in the optical image during the positioning of subject H, or when subject H is not present.

[0060] Furthermore, the processor 203 may control the user output device 170 via the user output device communication unit 202 to output the result of the intake timing derivation by S203 in Figure 4 or the result of the determination of whether or not it is the intake timing by S204 in Figure 4. In this case, the user output device 170 may, for example, display a GUI showing these results on a monitor, output pre-set audio from a speaker based on these results, or output a combination of these.

[0061] Figure 5 shows the first embodiment and illustrates an example of displaying the result of deriving the intake timing by step S203 in Figure 4 and the result of determining whether or not it is the intake timing by step S204 in Figure 4.

[0062] Specifically, Figure 5(a) shows an example of displaying the result of the inhalation timing derivation in step S203 of Figure 4. Figure 5(a) shows an example of when the processor 203 controls the display of the result of the inhalation timing derivation in step S203 of Figure 4 on the monitor of the user output device 170. In Figure 5(a), a bar display area 510 indicating the time until the inhalation timing is displayed on the monitor of the user output device 170, and the inhalation timing is shown as the max at the right end of the bar display area 510. In the bar display area 510 shown in Figure 5(a), the bar is displayed such that it approaches the max as the number of frames or time until the inhalation timing decreases. The processor 203 that controls the display of the bar display area 510 shown in Figure 5(a) constitutes a display control means that controls the display related to the state of the subject H based on the prediction result of a specific state of the subject H by the body motion prediction unit 220.

[0063] Figure 5(b) is a diagram showing an example of the display of the determination result of whether or not it is the inhalation timing in step S204 of Figure 4. Specifically, Figure 5(b) shows an example of displaying the result of the determination that it is the inhalation timing in step S204 of Figure 4 on the monitor of the user output device 170. Figure 5(a) shows an example of lighting up the icon 520 to inform the user that it is the inhalation timing (i.e., the start timing of radiation irradiation R to the subject H (start timing of radiography)). The processor 203 that controls the lighting up of the icon 520 shown in Figure 5(b) constitutes a display control means that controls the display related to the start of radiation irradiation H based on the prediction result of a specific state of the subject H by the body motion prediction unit 220.

[0064] Figure 6 shows a first embodiment and is a flowchart illustrating an example of a detailed processing procedure for the intake timing derivation process in step S203 of Figure 4.

[0065] First, in step S301 of Figure 6, the body motion prediction unit 220 detects from the optical image accumulated in step S202 whether or not there is a part of the subject H used to predict the inhalation timing, and if there is a part, the target region.

[0066] Next, in step S302 of Figure 6, the body motion prediction unit 220 determines whether or not the target area of ​​the subject H was detected in step S201.

[0067] In step S302 of Figure 6, if the body motion prediction unit 220 determines that it was able to detect the target area of ​​the subject H in step S201 (S302 / Yes), the process proceeds to step S303. When the process proceeds to step S303 in Figure 6, the body motion prediction unit 220 extracts the target region from the optical image accumulated in step S202, based on the detection result of the target region of the subject H in step S201. In this extraction process in step S303, for example, the portion other than the target region of the subject H may be cut from the optical image, or the portion of the optical image other than the target region of the subject H may be set to a predetermined uniform value. Furthermore, known methods can be used as the specific method for the extraction process in step S303, for example, any of the various machine learning methods such as deep learning may be used, or template matching or rule-based algorithms may be used.

[0068] Next, in step S304 of Figure 6, the body movement prediction unit 220 outputs a prediction of the timing of the inhalation state by inputting the target region of the subject H extracted in step S303 into the trained model 221. An example of the training data used when training this trained model 221 is described below.

[0069] Figure 7 shows a first embodiment and is a diagram illustrating a first example of training data used when training the trained model 221. Figure 7 shows an example in which a numerical value 702 indicating how many frames before the maximum intake timing the optical image is is applied as training data to each optical image 701 in a time series. Here, in Figure 7, the optical image at the maximum intake timing, which is an example of a specific state of subject H, is shown with hatched lines. As a variation of Figure 7, a numerical value indicating how many times before the maximum intake timing the image is is also applied as training data. That is, in the form of Figure 7, the trained model 221 is trained using the number of frames (numerical value 702) or time up to the maximum intake timing, which is an example of a specific state of subject H, as training data.

[0070] Figure 8 shows the first embodiment and is a diagram illustrating a second example of training data used when training the trained model 221. Figure 8 shows an example in which a numerical value 802, indicated by a flag, is applied as training data to each optical image 801 in a series of time-series images, based on the time lag from when the radiation R irradiation start signal is sent to the radiography device until the radiation R is actually irradiated, and the optical image at maximum intake. In Figure 8, a flag is set (the numerical value 802 is set to "1") only for the optical image 801 corresponding to the timing when the radiation R irradiation start signal should actually be sent, and no flag is set (the numerical value 802 is set to "0") for the other optical images 801. In this case, the flag can be anything that can distinguish between the timing when the radiation R irradiation start signal should be sent and the timing when it should not be sent, for example, it could be "1" and "0" as shown as the numerical value 802 in Figure 8, or it could be a specific number and a different number. In other words, in the configuration shown in Figure 8, the trained model 221 has learned, using the time lag until the radiation R is actually irradiated, whether or not the frame is one that is a time lag earlier than the point at which the maximum intake timing (a specific state of the subject H) occurs.

[0071] As shown in Figures 7 and 8, the trained model 221 used in the motion prediction unit 220 is a trained model that has been trained based on states prior to a specific state of subject H suitable for radiography (in the examples shown in Figures 7 and 8, the maximum inspiratory state). By using the trained model 221 trained as shown in Figures 7 and 8, it becomes possible to predict a specific state of subject H (in the examples shown in Figures 7 and 8, the maximum inspiratory state) regardless of whether the subject H's respiratory state is periodic or aperiodic.

[0072] Figure 9 shows a first embodiment and illustrates an example of a machine learning model applicable to the trained model 221. Specifically, Figure 9 shows an example in which a convolutional neural network (CNN) is applied as a machine learning model applicable to the trained model 221. In the CNN shown in Figure 9, an arbitrary captured image and several previous captured images on the time axis are input as input images 901, and the prediction result of a specific state of subject H is output 902. Note that, in addition to the CNN shown in Figure 9, a recurrent network (RNN) that is strong with time series data or a hybrid model of CNN and RNN may also be used as a machine learning model applicable to the trained model 221.

[0073] The image processing apparatus 150 according to the first embodiment described above includes an optical image acquisition unit 210 that acquires an optical image obtained by optically photographing a subject H that is to be subjected to radiography using radiation R. The image processing apparatus 150 according to the first embodiment also includes a body motion prediction unit 220 that predicts a specific state of the subject H suitable for radiography by inputting the optical image acquired by the optical image acquisition unit 210 into a trained model 221. In this case, the trained model 221 is a trained model that has been trained based on a state of the subject H prior to the specific state suitable for radiography. With this configuration, regardless of whether the subject's movement is periodic or aperiodic, radiography can be performed at a timing that is suitable for the subject's state during radiography.

[0074] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment below, matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0075] The schematic configuration of the radiography system according to the second embodiment is the same as the schematic configuration of the radiography system 100 according to the first embodiment shown in Figure 1. Furthermore, the schematic configuration of the image processing device 150 according to the second embodiment is the same as the schematic configuration of the image processing device 150 according to the first embodiment shown in Figure 2.

[0076] The second embodiment describes a configuration in which guidance is output to the user so that the movement of subject H can be predicted and radiation R can be irradiated when subject H is in a specific state. In the second embodiment as well, similar to the first embodiment described above, the specific state of a subject suitable for radiography is, for example, the inspiratory state of the lungs of a newborn or infant.

[0077] Figure 10 is a flowchart showing an example of the processing procedure of the image processing method using the image processing device 150 according to the second embodiment. Specifically, Figure 10 is a flowchart showing the processing procedure of the image processing device 150 up to the point of performing radiography of the subject H. In Figure 10, the same processing steps as those shown in Figure 3 are given the same step numbers, and their detailed explanations are omitted.

[0078] Once the process in step S104 in Figure 10, which is the same as step S104 in Figure 3, is completed, the process proceeds to step S401. When the process proceeds to step S401 in Figure 10, the body motion prediction unit 220 outputs (for example, displays) information indicating the timing of the inhalation state (inhalation timing) detected in step S104 to the user output device 170 via the user output device communication unit 202. This allows the user to confirm the inhalation timing of the subject H detected in step S104.

[0079] Next, in step S402 of Figure 10, the motion prediction unit 220 determines whether or not the user has made an operational input to the radiation irradiation switch of the user input device 160 in order to start the irradiation of radiation R.

[0080] In step S402 of Figure 10, if the motion prediction unit 220 determines that the user did not input an operation to the radiation irradiation switch of the user input device 160 (S402 / No), the process returns to step S104.

[0081] Furthermore, in step S402 of Figure 10, if the motion prediction unit 220 determines that the user has made an operation input to the radiation irradiation switch of the user input device 160 (S402 / Yes), the process proceeds to step S106. In step S106 of Figure 10, the same processing as in step S106 of Figure 3 is performed.

[0082] Here, step S401 in Figure 10 is similar to the explanation of the output example (Figure 5) of the inhalation timing derivation result by step S203 in Figure 4 or the determination result of whether or not it is the inhalation timing by step S204 in Figure 4 in the first embodiment. As explained in the same way as in the first embodiment, the output of these results may be, for example, a GUI showing these results displayed on a monitor, or pre-set sound output from a speaker based on these results, or a combination of these outputs. Here, as an example of displaying the output on a monitor, if the time until the inhalation timing is shown, a bar may be displayed as shown in Figure 5(a), which approaches the max as the number of frames or time until the inhalation timing decreases. Also, if the flag for the start timing of radiography is shown, the icon 520 may be lit up as shown in Figure 5(b).

[0083] Furthermore, while step S401 in Figure 10 describes an example where the user performs an operation input to the radiation irradiation switch of the user input device 160, the operation input could also be performed using buttons displayed on a touch panel GUI, or by voice input.

[0084] According to the second embodiment, in addition to the effects of the first embodiment described above, guidance can be output to the user so that the movement of the subject H can be predicted and radiation R can be irradiated when the subject H is in a specific state.

[0085] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. This program and a computer-readable storage medium on which the program is stored are included in this disclosure.

[0086] Furthermore, the embodiments of this disclosure described above are merely examples of concrete implementations of this disclosure, and the technical scope of this disclosure should not be interpreted as being limited by them. In other words, this disclosure can be implemented in various ways without departing from its technical concept or its main features.

[0087] Embodiments of this disclosure include the following configurations, methods, and programs. [Configuration 1] An acquisition means for acquiring an optical image obtained by optically photographing a subject undergoing radiography using radiation, A prediction means that predicts a specific state of the subject suitable for radiography by inputting the optical image acquired by the acquisition means into a trained model, Equipped with, The image processing device is an image processing device in which the trained model is a trained model that has been trained based on states prior to the specific state. [Configuration 2] The aforementioned specific state is a state that occurs aperiodically in the subject. The image processing apparatus described in Configuration 1. [Configuration 3] The aforementioned specific state is the state of maximum inhalation in the subject over a predetermined period of time. The image processing apparatus described in Configuration 1. [Structure 4] The subject is a newborn or infant. An image processing apparatus according to any one of configurations 1 to 3. [Composition 5] The aforementioned trained model is one that has been trained using the number of frames or time until the specific state is reached as training data. An image processing apparatus according to any one of configurations 1 to 4. [Composition 6] The aforementioned trained model has been trained to determine, using as training data whether a frame is earlier than the point in time when the specific state is reached, taking into account the time lag until the radiation is actually irradiated. An image processing apparatus according to any one of configurations 1 to 4. [Composition 7] The system further comprises an output means that outputs a signal relating to the start of radiation irradiation based on the prediction result of the specific state by the prediction means. An image processing apparatus according to any one of configurations 1 to 6. [Structure 8] The system further includes a display control means that controls the display of the subject's state based on the prediction result of the prediction means for the specific state. An image processing apparatus according to any one of configurations 1 to 7. [Composition 9] The system further includes a display control means that controls the display related to the start of radiation irradiation based on the prediction result of the specific state by the prediction means. An image processing apparatus according to any one of configurations 1 to 7. [Configuration 10] An image processing apparatus as described in any one of items 1 to 9, A radiography apparatus that is communicatively connected to the image processing apparatus and performs radiographic imaging of the subject using the radiation, A radiography system equipped with [specific features / equipment]. [Method 1] An acquisition step involves obtaining an optical image obtained by optically photographing a subject undergoing radiography using radiation, A prediction step in which the optical image acquired in the acquisition step is input into a trained model to predict a specific state of the subject suitable for radiography, Equipped with, An image processing method in which the trained model is a trained model that has been trained based on states prior to the specific state. [Program 1] A program for causing a computer to perform each step of the image processing method described in Method 1. [Explanation of symbols]

[0088] 100: Radiation imaging system, 110: Radiation control device, 120: Radiation generator, 130: Radiation plane detector, 140: Optical camera device, 150: Image processing device, 160: User input device, 170: User output device, 201: User input device communication unit, 202: User output device communication unit, 203: Processor, 204: Calculation memory unit, 205: Memory unit, 206: Bus, 210: Optical image acquisition unit, 220: Motion prediction unit, 221: Trained model, 230: Radiation control device communication unit, 240: Radiation image acquisition unit, H: Subject, R: Radiation

Claims

1. An acquisition means for acquiring an optical image obtained by optically photographing a subject undergoing radiography using radiation, A prediction means that predicts a specific state of the subject suitable for radiography by inputting the optical image acquired by the acquisition means into a trained model, Equipped with, The image processing device is an image processing device in which the trained model is a trained model that has been trained based on states prior to the specific state.

2. The aforementioned specific state is a state that occurs aperiodically in the subject. The image processing apparatus according to claim 1.

3. The aforementioned specific state is the state of maximum inhalation in the subject over a predetermined period of time. The image processing apparatus according to claim 1.

4. The subject is a newborn or infant. The image processing apparatus according to claim 1.

5. The aforementioned trained model is one that has been trained using the number of frames or time until the specific state is reached as training data. The image processing apparatus according to claim 1.

6. The aforementioned trained model has been trained to determine, using as training data whether or not a frame is earlier than the point in time when the specific state is reached, taking into account the time lag until the radiation is actually irradiated. The image processing apparatus according to claim 1.

7. The system further comprises an output means that outputs a signal relating to the start of radiation irradiation based on the prediction result of the specific state by the prediction means. The image processing apparatus according to claim 1.

8. The system further includes a display control means that controls the display of the subject's state based on the prediction result of the prediction means for the specific state. The image processing apparatus according to claim 1.

9. The system further includes a display control means that controls the display related to the start of radiation irradiation based on the prediction result of the specific state by the prediction means. The image processing apparatus according to claim 1.

10. An image processing apparatus according to any one of claims 1 to 9, A radiography apparatus that is communicatively connected to the image processing apparatus and performs radiographic imaging of the subject using the radiation, A radiography system equipped with [specific features / equipment].

11. An acquisition step involves obtaining an optical image obtained by optically photographing a subject undergoing radiography using radiation, and A prediction step in which the optical image acquired in the acquisition step is input into a trained model to predict a specific state of the subject suitable for radiography, Equipped with, An image processing method in which the trained model is a trained model that has been trained based on states prior to the specific state.

12. A program for causing a computer to perform each step of the image processing method described in claim 11.

Images