Image processing device, radiation imaging system, operation method of image processing device, and program

Abstract

To provide an image processing device capable of applying image processing to a moving image suitably in real-time even immediately after photography.SOLUTION: An image processing device for applying image processing to a moving image containing multiple frames of radiographic images, comprises: a selection unit that selects a learned model to be used for image processing of the frame being the processing target from among the multiple learned models with mutually-different input frame numbers according to the acquired frame numbers; and an inference processing unit that performs inference processing using the selected learned model in the image processing of the frame being the processing target.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a radiation imaging system, an operation method for an image processing device, and a program. [Background technology]

[0002] In recent years, radiography systems equipped with detectors for detecting radiation such as X-rays have been widely used in fields such as industry and medicine. In particular, in the field of X-ray videography, digital radiography systems have become widespread, converting incident X-rays into visible light using a phosphor and obtaining a video image using a semiconductor sensor. Here, a video image refers to a set of multiple still images acquired continuously, and each still image in the video image is hereinafter referred to as a frame.

[0003] In such radiographic imaging systems, various types of image processing are applied to images acquired by semiconductor sensors to enhance diagnostic value. One example is noise reduction processing. During the imaging process, various types of noise, such as quantum noise due to fluctuations in X-ray quanta and system noise generated by detectors and circuits, are known to occur and be superimposed on the image. This phenomenon can worsen the graininess of the resulting moving image, reducing diagnostic performance. In particular, in medical X-ray imaging, imaging with a low X-ray dose is recommended to minimize patient exposure. Therefore, it is important to apply image processing to the captured images to appropriately reduce noise and improve image quality in order to enhance diagnostic performance.

[0004] In response to this, Patent Document 1 proposes a rule-based technology that creates rules for accurately determining motion from video by taking the influence of noise into account, and performs appropriate noise reduction by weighting and adding multiple frames of video in a time series according to the determination results. Furthermore, more advanced noise reduction processes that apply machine learning-based technologies such as deep learning have recently been put to practical use. For example, Non-Patent Document 1 proposes a technology that inputs frames before and after a frame to be subjected to noise reduction and uses a trained neural network to obtain a noise-reduced image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-48782 [Non-patent literature]

[0006] [Non-Patent Document 1] “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”, M Tassano,et.al, IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1351~1360 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional technologies may have the following problems. According to Patent Document 1, by combining rule-based motion detection with a recursive filter, it is possible to use images of frames prior to the current frame (hereinafter referred to as past frames) and perform weighted addition that combines temporal information and spatial information. However, with rule-based motion detection processing, it is difficult to create appropriate rules for all cases of various subject structures contained in captured images, and afterimages may occur due to noise reduction.

[0008] Furthermore, according to Non-Patent Document 1, a good noise reduction effect can be achieved by processing that applies machine learning-based technology. However, the configuration described in Non-Patent Document 1 requires multiple frames to be input to the neural network, which poses a problem in that suitable processing cannot be performed until all frames are acquired. Furthermore, the configuration described in Non-Patent Document 1 requires input of past frames and future frames after the current frame in addition to the current frame. Therefore, it is difficult to perform processing that displays the results of processing the current frame after acquiring the current frame and before acquiring the next frame (hereinafter referred to as real-time processing).

[0009] In medical X-ray video imaging, from the viewpoint of radiation exposure to the subject, it is desirable that all captured frames be output as images, and a configuration is required that does not cause unnecessary radiation exposure. Also, in order to perform prompt medical treatment, it is required that an image that has been suitably subjected to image processing such as noise reduction and provided in real time even when only one image frame is available immediately after imaging.

[0010] One embodiment of the present disclosure has been made in consideration of the above-mentioned problems, and one of its objectives is to provide an image processing device that can apply image processing to moving images appropriately and in real time, even immediately after shooting. [Means for solving the problem]

[0011] An image processing device according to an embodiment of the present disclosure is an image processing device that applies image processing to a moving image including a plurality of frames of radiographic images, After the start of shooting the video The system includes a selection unit that selects a trained model to be used for the image processing of the frame to be processed from a plurality of trained models that have different numbers of input frames depending on the number of frames being acquired, and an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed. [Effects of the Invention]

[0012] According to an embodiment of the present disclosure, image processing can be applied to moving images suitably and in real time even immediately after shooting. [Brief explanation of the drawings]

[0013] [Figure 1] 1 shows an example of a schematic configuration of a radiation imaging system and a radiation detector according to a first embodiment. [Figure 2] 2 illustrates an example of a schematic configuration of a control unit and a noise reduction processing unit according to the first embodiment. [Figure 3A] 1 illustrates an example of a schematic configuration of a trained model according to a first embodiment. [Figure 3B] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Figure 3C] FIG. 4 is a diagram illustrating an example of the operation of a learning process according to the first embodiment. [Figure 4] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Figure 5] 4 is a table for explaining the operation of the radiation imaging system and the noise reduction processing unit according to the first embodiment. [Figure 6] 4A to 4C show examples of images before and after image processing according to the first embodiment. [Figure 7] 4 is a flowchart of the operation of the radiation imaging system according to the first embodiment. [Figure 8] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Figure 9A]FIG. 10 is a schematic diagram illustrating the operation of the radiation imaging system according to the second embodiment. [Figure 9B] FIG. 10 is a schematic diagram illustrating the operation of the radiation imaging system according to the second embodiment. [Figure 9C] FIG. 10 is a schematic diagram illustrating the operation of the radiation imaging system according to the second embodiment. [Figure 9D] FIG. 10 is a schematic diagram illustrating the operation of the radiation imaging system according to the second embodiment. [Figure 9E] FIG. 10 is a schematic diagram illustrating the operation of the radiation imaging system according to the second embodiment. [Figure 10] 10 illustrates an example of a schematic configuration of a CNN according to a second embodiment. [Figure 11] 10 illustrates an example of a schematic configuration of an image processing unit and a super-resolution processing unit according to a third embodiment. [Figure 12] 10 illustrates an example of a schematic configuration of a trained model according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, exemplary embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed depending on the configuration of the device to which the present disclosure is applied or various conditions. In addition, the same reference numerals are used in the drawings to indicate identical or functionally similar elements.

[0015] In the following, a radiography system using X-rays as an example of radiation will be described. However, the radiation may be X-rays or other radiation. In the following examples, the term radiation may include, for example, electromagnetic radiation such as X-rays and gamma rays, as well as particle radiation such as alpha rays, beta rays, particle beams, proton beams, heavy ion beams, and meson beams.

[0016] In the following, a machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor algorithms, naive Bayes algorithms, decision trees, and support vector machines. Neural networks and deep learning may also be used. Any available algorithm among the above can be used as appropriate and applied to the following embodiments and modifications. Furthermore, learning data refers to a dataset used for learning a machine learning model, and is composed of a pair of input data input to the machine learning model and correct answer data (teaching data) that is the correct answer for the output result of the machine learning model.

[0017] A trained model is a machine learning model that follows any machine learning algorithm, such as deep learning, and that has been trained in advance using appropriate training data. However, although a trained model is obtained by training in advance using appropriate training data, it does not mean that it does not undergo further training and that additional training can be performed. Additional training can also be performed after the device is installed at the site of use.

[0018] Example 1 (Configuration of Radiography System) A radiography system, an image processing device, and an operation method of the image processing device according to a first embodiment of the present disclosure will be described below with reference to Figures 1(a) and 1(b). Figure 1(a) shows an example of a schematic configuration of a radiography system 1 according to this embodiment. In the following description, the object to be inspected O will be described as a human body, but the object to be inspected O imaged by the radiography system according to the present disclosure is not limited to a human body and may be another animal, a plant, an object to be subjected to non-destructive inspection, or the like.

[0019] The radiation imaging system 1 according to this embodiment includes a radiation detector 10, a control unit 20, a radiation generator 30, an input unit 40, and a display unit 50. The radiation imaging system 1 may also include an external storage device 70 such as a server connected to the control unit 20 via a network 60 such as the Internet or an intranet.

[0020] The radiation generator 30 includes a radiation generating source such as an X-ray tube and is capable of emitting radiation. The radiation detector 10 detects the radiation emitted by the radiation generator 30 and generates a radiation image corresponding to the detected radiation. Therefore, the radiation detector 10 can generate a radiation image of the object O under inspection by detecting the radiation that is emitted by the radiation generator 30 and has passed through the object O under inspection.

[0021] FIG. 1(b) shows an example of the schematic configuration of a radiation detector 10 according to this embodiment. The radiation detector 10 is provided with a phosphor 11 and an imaging sensor 12. The phosphor 11 converts radiation incident on the radiation detector 10 into light of a wavelength detectable by the imaging sensor 12. The phosphor 11 may include, for example, CsI or GOS (Gd2O2S). The imaging sensor 12 includes a photoelectric conversion element made of, for example, a-Si or crystalline Si, and is capable of detecting light corresponding to the radiation converted by the phosphor 11 and outputting a signal corresponding to the detected light. The radiation detector 10 is able to generate a radiation image by performing A / D conversion or the like on the signal output by the imaging sensor 12.

[0022] 1(b), the radiation detector 10 may include a calculation unit, an A / D conversion unit, etc. Also, a grid may be installed between the radiation detector 10 and the inspection object O to reduce scattered rays that are generated when radiation passes through the inspection object O and reach the radiation detector 10.

[0023] The control unit 20 is connected to the radiation detector 10, the radiation generator 30, the input unit 40, and the display unit 50. The control unit 20 can acquire radiation images output from the radiation detector 10 and perform image processing on the radiation images, and control the driving of the radiation detector 10 and the radiation generator 30. This allows the control unit 20 to control the radiation generator 30 to generate radiation under predetermined imaging conditions at appropriate timing, and to perform video imaging at any frame rate. The control unit 20 can also function as an example of an image processing device.

[0024] The control unit 20 may be connected to an external storage device 70 via any network 60 such as the Internet or an intranet, and may acquire radiological images and the like from the external storage device 70. Furthermore, the control unit 20 may be connected to other radiation detectors, radiation generators, and the like via the network 60. The control unit 20 may be connected to the external storage device 70 and the like via a wired or wireless connection.

[0025] The input unit 40 includes input devices such as a mouse, keyboard, trackball, touch panel, etc., and can be operated by an operator to input instructions to the control unit 20. The display unit 50 includes, for example, any monitor, and can display information and images output from the control unit 20, information input by the input unit 40, etc.

[0026] In this embodiment, the control unit 20, input unit 40, display unit 50, etc. are configured as separate devices, but they may also be configured as an integrated unit. For example, the input unit 40 and display unit 50 may be configured as a touch panel display. In addition, in this embodiment, the control unit 20 configures an image processing device, but the image processing device only needs to be able to acquire a radiographic image and perform image processing on the radiographic image, and does not need to control the driving of the radiation detector 10 or the radiation generator 30.

[0027] The control unit 20 may be connected to the radiation detector 10, the radiation generator 30, etc. by wire or wirelessly. Furthermore, the external storage device 70 may constitute an image system such as a PACS within a hospital, or may be a server outside the hospital.

[0028] (Configuration of control unit) Next, a more specific configuration of the control unit 20 will be described with reference to Figures 2(a) and 2(b). Figure 2(a) shows an example of a schematic configuration of the control unit 20 according to this embodiment, and Figure 2(b) shows an example of a schematic configuration of the noise reduction processing unit 26 according to this embodiment. The control unit 20 is provided with an acquisition unit 21, an image processing unit 22, a display control unit 23, a drive control unit 24, and a storage unit 25.

[0029] The acquisition unit 21 can acquire radiographic images output by the radiation detector 10 and various information input by the input unit 40. The acquisition unit 21 can also acquire radiographic images, patient information, and the like from an external storage device 70 or the like.

[0030] The image processing unit 22 is provided with a noise reduction processing unit 26 and a diagnostic image processing unit 27, and can perform image processing according to the present disclosure on the radiographic image acquired by the acquisition unit 21. In this example, noise reduction processing will be described as an example of image processing performed by the image processing unit 22.

[0031] 2(b), the noise reduction processing unit 26 includes a learning processing unit 261 and an inference processing unit 262. In addition to the configuration of the inference processing unit 262 and the trained model selection unit 263, the learning processing unit 261 also includes a training data generation unit 264 and a parameter update unit 265. With this configuration, the noise reduction processing unit 26 can train a machine learning model for performing noise reduction processing, and apply noise reduction processing suitable for radiological images using the machine learning model.

[0032] Furthermore, the diagnostic image processor 27 can perform diagnostic image processing on the image that has undergone noise reduction by the noise reduction processor 26 to convert it into an image suitable for diagnosis. Diagnostic image processing includes, for example, gradation processing for adjusting the gradation of the image, enhancement processing for emphasizing specific pixels in the image, and grid stripe reduction processing for reducing grid stripes in the image. Note that the diagnostic image processor 27 may perform gradation processing, enhancement processing, grid stripe reduction processing, etc., according to a region of interest (ROI) set in the radiographic image. For example, the gradation processing may be performed to broaden the gradation of the region of interest, and the enhancement processing may be performed to emphasize the region of interest. Here, the region of interest may be set according to an instruction from the operator, or may be set based on the imaged body part, disease name information, finding information, etc.

[0033] Next, we will explain the configuration of the learning processing unit 261. The learning processing unit 261 performs learning processing that is applied when training a machine learning model, and includes an inference processing unit 262 and a trained model selection unit 263, as well as a training data generation unit 264 and a parameter update unit 265.

[0034] When performing the learning process, an image is input to the learning processing unit 261, and learning data is created by the learning data generation unit 264. Here, an example configuration is shown in which an image with artificial noise added is used as input data and an image without artificial noise added is used as correct answer data, as a set of learning data for learning noise reduction processing. The learning data generation unit 264 performs processing to create a set of learning data by adding artificial noise created by simulating the characteristics of a radiological image to the input image. Here, the noise added by the learning data generation unit 264 reflects the amount of noise calculated by the learning data generation unit 264 that may vary due to manufacturing variations. Details of the artificial noise to be added will be described later.

[0035] The parameter update unit 265 performs processing to update the parameters of the machine learning model held by the inference processing unit 262 based on the calculation results of the inference processing unit 262 on the input data and the correct answer data.

[0036] The inference processing unit 262 uses the radiographic image as input for a trained model that has been trained using the training data described above, thereby inferring and generating an image in which image processing has been applied to the radiographic image. The trained model selection unit 263 selects a trained model to be used by the inference processing unit 262. Details of the selection of a trained model by the trained model selection unit 263 will be described later.

[0037] Here, the learning processing unit 261 does not need to be included in the control unit 20. For example, the components of the learning processing unit 261 other than the inference processing unit 262 and the trained model selection unit 263 may be configured on hardware separate from the control unit 20, such as a server, and a trained model may be created by previously performing training using appropriate training data. In this case, the control unit 20 may access the separate hardware via the inference processing unit 262 and perform only processing using the trained model. Alternatively, a trained model created in advance may be provided in the noise reduction processing unit 26, and the trained model provided in the inference processing unit 262 may be used. Alternatively, the learning processing unit 261 may be included in the control unit 20, allowing additional training to be performed using training data acquired after installation.

[0038] The display control unit 23 can control the display on the display unit 50, and can display radiographic images before and after image processing by the image processing unit 22, patient information, etc. on the display unit 50. The drive control unit 24 can control the driving of the radiation detector 10, the radiation generator 30, etc. Therefore, the control unit 20 can control the capture of radiographic images by controlling the driving of the radiation detector 10 and the radiation generator 30 using the drive control unit 24.

[0039] The storage unit 25 can store an operating system (OS), device drivers for peripheral devices, and programs for implementing various application software including programs for performing the processes described below. The storage unit 25 can also store information acquired by the acquisition unit 21, radiographic images processed by the image processing unit 22, etc. For example, the storage unit 25 can store radiographic images acquired by the acquisition unit 21, and radiographic images that have been subjected to noise reduction processing described below.

[0040] The control unit 20 can be configured using a general computer including a processor, memory, etc., but may also be configured as a computer dedicated to the radiation imaging system 1. Here, the control unit 20 functions as an example of an image processing device according to this embodiment, but the image processing device according to this embodiment may be a separate (external) computer communicatively connected to the control unit 20. The control unit 20 and the image processing device may be, for example, a personal computer, or a desktop PC, notebook PC, or tablet PC (portable information terminal). The processor may be a CPU (Central Processing Unit). The processor may also be, for example, an MPU (Micro Processing Unit), GPU (Graphical Processing Unit), FPGA (Field-Programmable Gate Array), etc.

[0041] Each function of the control unit 20 may be realized by a processor such as a CPU or MPU executing software modules stored in the storage unit 25. The processor may be, for example, a GPU or FPGA. Each function may also be configured by a circuit that performs a specific function, such as an ASIC. For example, the image processing unit 22 may be realized by dedicated hardware such as an ASIC, and the display control unit 23 may be realized by a dedicated processor such as a GPU that is different from the CPU. The storage unit 25 may be configured by any storage medium, such as an optical disk such as a hard disk, or a memory.

[0042] (Machine learning model configuration) Next, an example of a machine learning model constituting the trained model according to this embodiment will be described with reference to Figures 3A to 3C. An example of the machine learning model used by the inference processing unit 262 according to this embodiment is a multi-layer neural network.

[0043] FIG. 3A shows a schematic configuration example of a neural network model according to this embodiment. The neural network model configuration 33 shown in FIG. 3A is designed to output noise-reduced inference data 32 for input data 31 according to a previously learned trend. The output noise-reduced inference data 32 is based on learning content in a machine learning process, and the neural network according to this embodiment learns features for distinguishing between signals and noise contained in an input radiographic image. In the example shown in FIG. 3A, the input data 31 is the current frame and one or more frames prior to the current frame, and the noise-reduced inference data 32 is a frame in which the noise of the current frame has been reduced. It is also possible to configure a trained model in which the input data 31 is the current frame and the number of input frames is one.

[0044] At least a part of the multi-layered neural network may be, for example, a convolutional neural network (CNN), or may be based on an autoencoder.

[0045] Here, a case where CNN is used as a machine learning model for noise reduction processing of a radiographic image will be described. Fig. 3B shows an example of a schematic configuration 33 of a CNN constituting a neural network model according to this embodiment. In the example of the trained model according to this embodiment, when input data 31, which is a radiographic image, is input, inference data 32 can be output as a radiographic image with reduced noise.

[0046] The CNN shown in FIG. 3B is composed of multiple layers that process input values ​​and output them. The types of layers included in the CNN configuration 33 include a convolution layer, a downsampling layer, an upsampling layer, and a merge layer. The CNN configuration 33 may further include an addition layer 34, which serves as a shortcut for adding input data before output. This allows the CNN to learn the difference between input data and output data, making it suitable for handling systems that target noise.

[0047] The convolution layer is a layer that performs convolution processing on a group of input values ​​according to parameters such as the set filter kernel size, the number of filters, the stride value, the dilation value, etc. Note that the number of dimensions of the filter kernel size may also be changed depending on the number of dimensions of the input image.

[0048] The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning out or combining input value groups. Specifically, such processing includes, for example, max pooling processing.

[0049] An upsampling layer is a layer that performs processing to make the number of output values ​​larger than the number of input values ​​by duplicating input values ​​or adding values ​​interpolated from the input values, for example, upsampling by deconvolution.

[0050] A synthesis layer is a layer that inputs a group of values, such as a group of output values ​​from a layer or a group of pixel values ​​that make up an image, from multiple sources and performs processing to combine them by concatenating or adding them.

[0051] It should be noted that different parameter settings for the layers and nodes that make up the neural network may affect the degree to which trends trained from learning data can be reproduced during inference. In other words, in many cases, appropriate parameters differ depending on the implementation form, so they can be changed to preferred values ​​as needed.

[0052] In addition to changing the parameters as described above, better CNN characteristics may be obtained by changing the CNN configuration 33. Better characteristics include, for example, outputting radiological images with more accurate noise reduction, shortening processing time, shortening the time required to train a machine learning model, etc.

[0053] The CNN configuration 33 used in this embodiment is a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers. The U-net type machine learning model is configured (for example, by using skip connections) so that position information (spatial information) obscured in multiple layers configured as encoders can be used in layers of the same dimension (layers corresponding to each other) in multiple layers configured as decoders.

[0054] Although not shown, as an example of a modification of the CNN configuration, for example, layers of activation functions (for example, ReLu: Rectifier Linear Unit) may be incorporated before and after the convolution layer.

[0055] Through these steps of CNN, noise features can be extracted from the input radiological image.

[0056] Here, the learning processing unit 261 includes a parameter updating unit 265. As shown in FIG. 3C , the parameter updating unit 265 calculates a loss function from inference data 32 obtained by applying the neural network model of the inference processing unit 262 to input data 31 in the learning data, and from supervised data 35 in the learning data. Thereafter, the parameter updating unit 265 performs processing to update the parameters of the neural network model based on the calculated loss function. Here, the loss function indicates the error between the inference data 32 and the supervised data 35.

[0057] The parameter update unit 265 can update the filter coefficients of the convolution layer, for example, using backpropagation so as to reduce the error between the inference data 32 represented by the loss function and the ground truth data 35. The backpropagation is a method of adjusting parameters between the nodes of a neural network so as to reduce the error. Note that the learning may also use a method called dropout, which randomly inactivates units (neurons or nodes) that make up the CNN.

[0058] Furthermore, the trained model used by the inference processing unit 262 may be generated using transfer learning. In this case, for example, transfer learning may be performed on a machine learning model trained on radiographic images of inspection objects O of different types, etc., to generate a trained model to be used in noise reduction processing. By performing such transfer learning, it is possible to efficiently generate trained models even for inspection objects O for which it is difficult to obtain a large amount of training data. The inspection objects O of different types, etc., referred to here may be, for example, animals, plants, objects to be subjected to non-destructive testing, etc.

[0059] Here, a GPU can perform efficient calculations by processing a larger amount of data in parallel. Therefore, when performing learning multiple times using a machine learning model that uses CNN as described above, it is effective to use a GPU for the processing. Therefore, the learning processing unit 261 according to this embodiment uses a GPU in addition to a CPU. Specifically, when executing a learning program including a machine learning model, the CPU and GPU work together to perform calculations to perform learning. Note that calculations may be performed only by the CPU or the GPU in the learning process. Furthermore, each process of the inference processing unit 262 may also be realized using a GPU, as in the learning processing unit 261.

[0060] The configuration of the machine learning model has been described above, but the machine learning model is not limited to the model using CNN as shown above. The learning of the machine learning model used in this embodiment may be any model similar to machine learning that uses a model that can extract (express) features of learning data such as images by itself through learning.

[0061] Here, the learning processing unit 261 according to this embodiment can use any set of learning data for learning the noise reduction processing. The learning processing unit 261 can use, for example, learning data in which an image to which artificial noise has been added is used as input data and an image to which no artificial noise has been added is used as correct answer data. In addition, learning may be performed using, for example, an image before averaging as input data and an image after averaging as correct answer data, or an image before statistical processing such as MAP (maximum a posteriori probability) estimation processing as input data and an image after statistical processing as correct answer data.

[0062] (Operation of image processing unit) Next, detailed operations of the image processing unit 22 during video shooting will be described using Figures 4(a) to 8. In video shooting, past frames near the current frame often have similar structures. Therefore, when performing noise reduction on a target pixel in the current frame, not only spatial information such as similar structures in the same frame but also temporal information (time information) such as similar structures in past frames can be used for noise reduction.

[0063] From this perspective, by inputting multiple frames, the inference processing unit 262 can perform processing while utilizing more time information. In this case, in order to perform real-time processing, the inference processing unit 262 needs to be configured to input a total of N frames, including the current frame and a predetermined number of past frames. The past frames to be used vary depending on the frame rate used during shooting and the required noise reduction performance, but the following will explain, as a suitable example, the case where N=10 frames are input (the current frame plus the past 9 frames).

[0064] FIG. 4(a) is a schematic diagram of the configuration of a neural network when N=10. In the example shown in FIG. 4(a), the number of the current frame is t, and the current frame and past frames numbered t-1 to t-9 are input sequentially to a trained CNN 41. Using such a CNN 41, the inference processing unit 262 can obtain a noise-reduced image F(t) that has been subjected to noise reduction processing using spatial information of the current frame and temporal information of nine past frames. The trained CNN 41 has been trained using a set of training data in which, similar to the input described above, consecutive N=10 frames are used as input data and the correct image corresponding to the frame numbered t is used as correct data.

[0065] 4(b), the inference processing unit 262 inputs the frame numbered t+1, which corresponds to the current frame, and the past frames numbered t to t-8 into the trained CNN 41, thereby obtaining a noise-reduced image F(t+1). By performing this process sequentially, the inference processing unit 262 can obtain a suitable noise-reduced image in real time.

[0066] The problem here occurs when t=0 to 8 immediately after the start of shooting, when N (10) input frames are not available. Figures 5(a) and 5(b) are tables showing the relationship between CNN input frames and current frame numbers when the number of frames is not available immediately after shooting. As shown in Figure 5(a), until shooting at t=9 is completed, the input frames originally expected by CNN 41 are not available, and a situation occurs in which the inference processing unit 262 cannot operate CNN 41 normally.

[0067] As mentioned above, in medical X-ray video imaging, it is desirable to output all captured frames as images from the viewpoint of radiation exposure to the subject, and a configuration is required to prevent unnecessary radiation exposure. Furthermore, in order to perform prompt medical treatment, it is desirable to provide an image with suitably reduced noise through real-time processing even when there is only one frame image immediately after imaging, so it is desirable to handle imaging where the current frame number is t=0 to 8.

[0068] 6 shows an example of a portion of an image in a certain frame. Image 61 is an example of the image before noise reduction, and images 62 and 63 are examples of the image after noise reduction processing.

[0069] Now, we will explain an example in which, as in case 1 in Figure 5(b), when the number of input frames originally expected by the CNN 41 is not present, processing is performed using an acquired frame (e.g., frame number 0) as the frame that has not been acquired. Note that in the example shown in Figure 5(b), the current frame number t is assumed to be 5. In this case, as in image 62 in Figure 6, not only is the noise reduction effect low, but artifacts occur in the image, resulting in an image that is not suitable for diagnosis. On the other hand, as in case 2 in Figure 5(b), when all the input frames originally expected by the CNN 41 are present (assuming that frames numbered -1 to -4 have been obtained), an appropriate noise reduction effect can be obtained, as in image 63 in Figure 6.

[0070] In view of this situation, the configuration of the radiography system 1 according to this embodiment will be described with reference to Fig. 7 to Fig. 8(c). Fig. 7 shows an example of the flow of the noise reduction processing unit 26 when capturing a moving image. Fig. 8(a) and Fig. 8(b) show an example of the schematic configuration of the CNN used in the inference processing unit 262.

[0071] In this embodiment, three types of CNNs are prepared for use in inference processing, and the trained model selection unit 263 can select an appropriate trained model as the trained model to be used by the inference processing unit 262. Furthermore, the three types of CNNs, the first CNN 81 (CCN1), the second CNN 82 (CCN2), and the third CNN 83 (CCN3), use different numbers of input frames, such as N1=1, N2=5, and N3=10, respectively.

[0072] Each CNN is trained by the training processing unit 261 so as to obtain optimal performance with a predetermined number of input frames. Specifically, the first CNN 81 can train using, for example, a pair of a frame with number t to which noise has been added and a frame with number t to which noise has not been added as training data. The second CNN 82 can train using, for example, a pair of a frame with number t to which noise has been added, past frames numbered t-1 to t-4, and a frame with number t to which noise has not been added as training data. Similarly, the third CNN 83 can train using, for example, a pair of a frame with number t to which noise has been added, past frames numbered t-1 to t-9, and a frame with number t to which noise has not been added as training data.

[0073] The operation of the noise reduction processing unit 26 will be described below with reference to Fig. 7. First, in step S701, when a moving image shooting sequence is started, the noise reduction processing unit 26 sets the initial frame number to t=0.

[0074] In step S702, the noise reduction processing unit 26 acquires the t-th frame image via the acquisition unit 21. Initially, the noise reduction processing unit 26 is in a state of acquiring one frame at t=0.

[0075] In step S703, the noise reduction processor 26 performs preprocessing on the image acquired in step S702 to perform appropriate inference processing. While the preprocessing method is not limited, for example, in noise reduction processing, quantum noise following a Poisson distribution can be made substantially constant regardless of the intensity of the input radiation by performing, for example, square root transformation or logarithmic transformation. Preprocessing may also include conversion processing to handle additive noise or processing to set the average value to 0. Furthermore, in addition to these processes, the noise reduction processor 26 may perform appropriate preprocessing depending on the content of image processing, such as normalizing the data between 0 and 1, or standardizing the data to have an average value of 0 and a standard deviation of 1. The preprocessed frame will be used in the inference processing of the subsequent frame, and can therefore be temporarily stored in memory until use is complete.

[0076] In step S704, the trained model selection unit 263 determines the number of frames currently acquired, and in particular determines whether t is less than 4. Here, if t is less than 4, the process proceeds to step S705. In step S705, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used by the inference processing unit 262. The inference processing unit 262 performs inference processing using the selected first CNN 81 with a single input of the current frame. Under such circumstances, since there are few past frames acquired, it is difficult to use temporal information, and therefore the noise reduction processing unit 26 is configured to perform noise reduction using only spatial information.

[0077] On the other hand, if t is 4 or greater, the process proceeds to step S706. In step S706, the trained model selection unit 263 determines whether t is 4 or greater and less than 9. If t is 4 or greater and less than 9, the process proceeds to step S707. In step S707, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b) as the trained model to be used by the inference processing unit 262. The inference processing unit 262 performs inference processing using the selected second CNN 82 with a total of five input frames: one current frame and four past frames. In the situation where the process proceeds to step S707, although the number of past frames is not sufficient, a certain number of past frames have been obtained, and temporal information is available. For this reason, in step S707, the CNN's computational resources are configured to be used so that noise reduction can be performed using both spatial information and temporal information. In this case, the noise reduction processing unit 26 cannot use as much temporal information as the third CNN 83, but it can at least prevent the situation that generates artifacts such as those shown in image 62 of FIG. 6, and can also appropriately reduce noise.

[0078] On the other hand, if t is 9 or greater, the process proceeds to step S708. In step S708, since t is 9 or greater, a sufficient number of past frames are available to make use of the temporal information. Therefore, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c) as the trained model to be used by the inference processing unit 262. Using the selected third CNN 83, the inference processing unit 262 performs processing using a total of 10 input frames, including one current frame and nine past frames, thereby preventing the occurrence of artifacts and enabling noise reduction that makes maximum use of the temporal information.

[0079] In step S709, the noise reduction processing unit 26 performs post-processing on the inference processing result. The post-processing is the inverse processing of the various normalizations and smoothings performed in the pre-processing in step S703.

[0080] In step S710, the noise reduction processing unit 26 determines whether or not to end image acquisition. Note that the noise reduction processing unit 26 may determine whether to end image acquisition based on, for example, the set shooting conditions or an instruction from the operator. If image acquisition is to be continued, the process proceeds to step S711. In step S711, the noise reduction processing unit 26 adds 1 to the frame number t, and the process proceeds to step S702, where the noise reduction processing unit 26 repeats the processes of steps S702 to S710.

[0081] By performing the processes in steps S701 to S711, the noise reduction processing unit 26 can perform real-time processing to obtain an image with suitably reduced noise even when there is only one frame of image immediately after shooting.

[0082] As described above, the radiation imaging system 1 according to this embodiment includes the control unit 20, the radiation generator 30, and the radiation detector 10. The radiation generator 30 functions as an example of a radiation generating device that irradiates radiation, and the radiation detector 10 functions as an example of a radiation detecting device that detects the irradiated radiation. The control unit 20 functions as an example of an image processing device that applies image processing to a moving image including a plurality of radiation image frames.

[0083] The control unit 20 includes a trained model selection unit 263 and an inference processing unit 262. The trained model selection unit 263 functions as an example of a selection unit that selects a trained model to be used for image processing of a frame to be processed from multiple trained models that have different numbers of input frames, depending on the number of frames acquired. The inference processing unit 262 functions as an example of an inference processing unit that performs inference processing using the selected trained model in image processing of the frame to be processed. Note that the image processing may include noise reduction processing that reduces noise in the image. With this configuration, the radiation imaging system 1 according to this embodiment can apply image processing to moving images appropriately and in real time, even immediately after imaging.

[0084] The multiple trained models may include a trained model with one input frame and a trained model with more than 1 input frame. Therefore, the control unit 20 according to this embodiment can apply image processing to moving images appropriately and in real time not only in a situation where multiple frames are acquired but also in a situation where only one frame is acquired.

[0085] Furthermore, the inference processing unit 262 can input a first frame, which is the frame to be processed, and zero or more second frames acquired before the first frame, to the selected trained model according to the number of frames input to the selected trained model, and infer an image obtained by applying image processing to the first frame. With this configuration, the control unit 20 can perform image processing using frames prior to the frame to be processed, and can apply image processing to moving images appropriately and in real time.

[0086] The trained model selection unit 263 selects, from among multiple trained models, a trained model whose number of input frames is equal to or less than the number of acquired frames and whose number of input frames is greater than the number of acquired frames of other trained models whose number of input frames is equal to or less than the number of acquired frames, according to the number of acquired frames. With this configuration, the control unit 20 can perform image processing using a trained model that can perform more appropriate image processing according to the number of acquired frames.

[0087] Each of the multiple trained models is obtained by training using training data including a number of images corresponding to the number of input frames and an image obtained by applying image processing to a processing target image among the input frames. With this configuration, a trained model to which multiple frames are input can use not only spatial information, such as similar structures around the same frame, but also temporal information, such as similar structures in the multiple input frames, for image processing. Therefore, the control unit 20 according to this embodiment can more appropriately apply image processing to moving images in real time, even immediately after capture. Regarding the training data described above, the images to which artificial noise has been added correspond to a number of images corresponding to the number of input frames before noise reduction processing, and the images before adding artificial noise correspond to images obtained by applying noise reduction processing to the processing target image.

[0088] If the image processing unit 22 is unable to process the entire image at once due to its memory capacity or other performance, the image may be divided into small regions of an appropriate size (for example, 256×256 pixels) and processed.

[0089] Furthermore, the number of frames to be input to the trained model is not limited to N=10, and may be any number equal to or greater than 2. Note that if the influence of the movement of the object to be inspected O becomes large, it becomes difficult to obtain a structure similar to the target pixel of the current frame in the temporal information. Therefore, the number of frames to be input can be set so as to be completed within a certain real time period, taking into account the frame rate.

[0090] In addition, the above example shows the use of three trained models: a first CNN 81 with an input of one frame, a second CNN 82 with an input of five frames, and a third CNN 83 with an input of ten frames. However, the configuration of the trained models used is not limited to this, and the type of trained model used and the number of input frames for each may be set according to a desired configuration. For example, the type of trained model and the number of input frames may be freely changed depending on the sensitivity, bias voltage, noise characteristics, readout amplification factor, frame rate, image size, accumulation time when receiving signals, and imaging technique used for imaging.

[0091] Example 2 An image processing unit according to a second embodiment of the present disclosure will be described with reference to FIGS. 9A to 10(c). FIGS. 9A to 9E are diagrams illustrating an example of an operation flow of a radiography system according to this embodiment. In FIGS. 9A to 9E, the fluoroscopic imaging mode refers to an imaging mode for capturing moving images by radiography, and the general imaging mode refers to an imaging mode for capturing still images by radiography. Here, the imaging mode refers to a set of settings, for example, of the sensitivity and bias voltage of a radiation detector in the radiography system, noise characteristics, amplification factor during readout, frame rate, image size, accumulation time during signal reception, and imaging technique. Note that the configuration of the radiography system according to this embodiment, other than the image processing unit 22, is the same as the configuration of the radiography system 1 according to the first embodiment, and therefore the same reference numerals will be used and a description thereof will be omitted.

[0092] 9A shows the flow when fluoroscopic imaging (first fluoroscopic imaging) is performed to acquire n+1 frame images from t=0 to n (n≧9), then one still image is acquired by general imaging, and fluoroscopic imaging is performed again (second fluoroscopic imaging). Here, the first fluoroscopic imaging and the second fluoroscopic imaging are performed in the same mode (fluoroscopic imaging mode 1) of the radiation imaging system. In FIG. 9A, in the second fluoroscopic imaging, the frame number is set to t' and is reassigned from 0.

[0093] Here, after imaging starts, in fluoroscopic imaging mode 1, from t=0 to 3, the trained model selection unit 263 selects the first CNN 81 (CNN1) shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing for one input frame using the selected first CNN 81. Furthermore, from t=4 to 8, the trained model selection unit 263 selects the second CNN 82 (CNN2) shown in FIG. 8(b), and the inference processing unit 262 performs inference processing for five input frames using the second CNN 82. Furthermore, from t=9 to n, the trained model selection unit 263 selects the third CNN 83 (CNN3) shown in FIG. 8(c), and the inference processing unit 262 performs inference processing for ten input frames using the third CNN 83.

[0094] Thereafter, when shooting in general shooting mode, the trained model selection unit 263 selects a fourth CNN (CNN4), which is a CNN different from the first CNN 81 to the third CNN 83 and trained specifically for general shooting mode, as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing using the selected fourth CNN. Here, it is preferable to use a CNN trained to suitably perform noise reduction processing according to the characteristics of general shooting mode when a single frame is input as the fourth CNN.

[0095] As another configuration, if the fluoroscopic imaging mode used before the general imaging mode is limited, it is also possible to use a CNN that inputs multiple frames by adding frames from fluoroscopic imaging performed before the general imaging mode to the input of the fourth CNN. In this case, the learning data may be images captured in the general imaging mode and frames from fluoroscopic imaging performed before the general imaging mode, and images captured in the general imaging mode that have been subjected to noise reduction processing may be used as the correct answer data. Note that, as in Example 1, learning may be performed using learning data that uses images to which artificial noise has been added.

[0096] Next, imaging is resumed in fluoroscopic imaging mode 1, and the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) from t'=0 to 3. The inference processing unit 262 performs inference processing for one input frame using the selected first CNN 81. Furthermore, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b) from t'=4 to 8, and the inference processing unit 262 performs inference processing for five input frames using the second CNN 82. Furthermore, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c) from t'=9 to n, and the inference processing unit 262 performs inference processing for ten input frames using the third CNN 83.

[0097] In the flow shown in Figure 9B, the imaging procedure is the same as in Figure 9A, but the difference is that the third CNN 83 is used from t' = 0 in the second fluoroscopic imaging. As in this example, if the first and second imaging are in the same mode and the time interval between the two fluoroscopic imaging is short, such as when a general imaging mode or other short imaging is sandwiched in between, a configuration may be adopted in which the current frame t' = 0 and past frames t = n to n-8 are used. In this case, the trained model selection unit 263 selects the third CNN 83 from t' = 0 in the second fluoroscopic imaging, and the inference processing unit 262 can perform inference processing on the 10 input frames using the third CNN 83.

[0098] FIG. 9C shows a flow when fluoroscopic imaging is performed to acquire n+1 frame images from t=0 to n (where n≧9), followed by a pause in imaging and short-term imaging (here, t'=0 to 3), followed by further fluoroscopic imaging (t"=0 onward). Note that the pause in imaging and short-term imaging may be performed by repeatedly momentarily turning on and off a switch such as a foot pedal for irradiating radiation.

[0099] In this case, after imaging starts, in fluoroscopic imaging mode 1, from t=0 to 3, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing for one input frame using the selected first CNN 81. Furthermore, from t=4 to 8, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b), and the inference processing unit 262 performs inference processing for five input frames using the second CNN 82. Furthermore, from t=9 to n, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c), and the inference processing unit 262 performs inference processing for ten input frames using the third CNN 83. Thereafter, the trained model selection unit 263 selects the first CNN 81 as the trained model to be used for inference processing between t'=0 and 3, and the inference processing unit 262 performs inference processing for one frame using the first CNN 81. Further thereafter, the radiography system again switches to radiography in fluoroscopic radiography mode 1, and after t"=0, the system behaves in the same way as it did after the initial t=0.

[0100] Also, as shown in FIG. 9D, the shooting procedure is the same as in FIG. 9C, but between t'=0 and 3 and after t"=0, it is possible to operate the third CNN 83 by using frames going back from t=n as past frames as necessary.

[0101] FIG. 9E shows an example in which fluoroscopic imaging is performed in fluoroscopic imaging mode 1 to acquire n+1 frame images from t=0 to n (n≧9), and then the mode is switched to another fluoroscopic imaging mode 2 to acquire frames from t′=0 onward. In this case, after imaging begins, during the period from t=0 to 3 in fluoroscopic imaging mode 1, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing on a single input frame using the selected first CNN 81. During the period from t=4 to t=8, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b), and the inference processing unit 262 performs inference processing on a single input frame using the second CNN 82. Furthermore, between t=9 and n, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c), and the inference processing unit 262 performs inference processing on the input of 10 frames using the third CNN 83.

[0102] Thereafter, the radiography system switches to fluoroscopic imaging mode 2 and processes frames after t' = 0. Figures 10(a) to 10(c) show schematic diagrams of CNNs used in another mode (here, fluoroscopic imaging mode 2) by the inference processing unit 262, and each CNN is optimized for and trained on the characteristics of images in fluoroscopic imaging mode 2.

[0103] As mentioned above, the type of CNN used and the number of input frames for each can be freely changed depending on the image resolution of the system used, the frame rate during imaging, the imaging technique, etc. For example, in an imaging mode with a low frame rate, if the number of input frames to the CNN is too high, it may take too long to prepare the number of frames required for maximum performance, or the subject may move too much, making it impossible to effectively utilize the time information. Therefore, when the frame rate is low, it is effective to configure the CNN to have a smaller number of input frames.

[0104] 10(a) to 10(c) show an example of a configuration in which the number of input frames is different from that of the CNN shown in FIG. 8(a) to 8(c). In the example shown in FIG. 9D, when t'=0 to 1, the trained model selection unit 263 selects the fifth CNN 101 (CNN5) shown in FIG. 10(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing for a single input frame using the selected fifth CNN 101. Furthermore, when t'=2 to 4, the trained model selection unit 263 selects the sixth CNN 102 (CNN6) shown in FIG. 10(b), and the inference processing unit 262 performs inference processing for a three input frame using the sixth CNN 102. Furthermore, from t'=5 onwards, the trained model selection unit 263 selects the seventh CNN 103 (CNN7) shown in Figure 10(c), and the inference processing unit 262 performs inference processing on the input of six frames using the seventh CNN 103.

[0105] Here, the first to seventh CNNs exemplified above may be CNNs with the same network structure except for the number of input frames, with only the trained parameters changed, or may each have a different network structure.

[0106] As described above, in the control unit 20 according to this embodiment, the multiple trained models can include a group of trained models corresponding to the imaging mode. Furthermore, the trained model selection unit 263 can select a trained model to be used for image processing of the frames to be processed from the group of trained models corresponding to the imaging mode of the moving image to be processed, depending on the number of frames acquired. This configuration allows the control unit 20 to more appropriately apply image processing to the moving image in real time, depending on the imaging mode. The imaging mode can be set based on at least one of the sensitivity of the detector used for imaging, bias voltage, noise characteristics, amplification factor during readout, frame rate, image size, accumulation time during signal reception, and imaging technique.

[0107] Furthermore, the trained model selection unit 263 can exclude frames acquired in a shooting mode different from the shooting mode of the moving image to be processed from the number of acquired frames. In this case, the control unit 20 can prevent a situation in which image processing is not performed appropriately due to the use of frames acquired in a different shooting mode as input.

[0108] Furthermore, in a second image capture performed in a predetermined image capture mode after the first image capture in the predetermined image capture mode, the trained model selection unit 263 can count the number of frames acquired in the first image capture as the number of frames acquired. With this configuration, the control unit 20 can use frames acquired in the first image capture for image processing in the second image capture, for example, when the first and second image captures are performed in the same mode and the time interval between the two images is short. Therefore, even immediately after the start of the second image capture, time information can be used for image processing, enabling more appropriate and real-time image processing to be applied to moving images.

[0109] According to the configuration of this embodiment, the radiation imaging system of this embodiment can perform real-time processing to obtain images with suitably reduced noise for all frames, even if the imaging mode is changed during imaging.

[0110] Example 3 Up to now, noise reduction processing by the noise reduction processing unit 26 has been described as an example of image processing performed by the image processing unit 22. However, the present disclosure is not limited to this, and the above configuration can be adopted for any image processing using a machine learning model on a moving image. Here, an example of super-resolution processing for improving image resolution as image processing on a moving image for the image processing unit according to Example 3 of the present disclosure will be described. Note that the configuration of the radiation imaging system according to this example, other than the image processing unit 22, is the same as the configuration of the radiation imaging system 1 according to Example 1, so the same reference numerals will be used and description will be omitted.

[0111] 11A shows an example of a schematic configuration of the image processing unit 22 according to this embodiment. The image processing unit 22 according to this embodiment is provided with a super-resolution processing unit 116 instead of the noise reduction processing unit 26.

[0112] 11(b) shows an example of a schematic configuration of the super-resolution processing unit 116. The configuration of the super-resolution processing unit 116 is similar to that of the noise reduction processing unit 26 according to the first embodiment, and the super-resolution processing unit 116 includes a learning processing unit 261 and an inference processing unit 262. The learning processing unit 261 includes a learning data generating unit 264 and a parameter updating unit 265 in addition to the inference processing unit 262 and the trained model selecting unit 263.

[0113] FIG. 12 shows a schematic configuration example of a neural network model used in the super-resolution processing in this embodiment. The configuration 123 of the neural network model shown in FIG. 12 is designed to output inferred data 122 with improved resolution for input data 121 according to a previously learned trend. The input data 121 includes one or more frames with a resolution lower than a desired resolution, and may be, for example, a current frame and one or more frames past the current frame. The neural network model has a configuration 123 that has been trained using a set of training data in which a low-resolution image is used as input data and an image with a desired resolution is used as ground truth data. When input data 121, which is a radiographic image, is input, the neural network model is configured to output inferred data 122, which is an image after super-resolution processing, according to the configuration 123. It is also possible to configure a trained model in which the input data 121 is a single current frame and the number of input frames is one.

[0114] The training data can be generated by, for example, using a low-resolution image as input data and a reference image of a desired resolution generated by applying a known super-resolution process to the input data. The training data can also be generated by using an image acquired using a radiation detector capable of acquiring an image of a desired resolution as reference data and an image obtained by lowering the resolution of the image as the input image. The training data can also be generated by using an image acquired by setting a low resolution as the input data and an image acquired by setting a high resolution (desired resolution) as the reference image. When the input data consists of multiple frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the training data, as in the first embodiment.

[0115] An example of the machine learning model used by the inference processing unit 262 according to this embodiment may be a multi-layered neural network, and at least a portion of the multi-layered neural network may be, for example, a CNN. Furthermore, at least a portion of the multi-layered neural network may use autoencoder technology. The trained model used by the inference processing unit 262 may be generated using transfer learning. In this case, for example, transfer learning may be performed on a machine learning model trained on radiographic images of different types of inspection objects O to generate a trained model to be used in super-resolution processing. By performing such transfer learning, trained models can be efficiently generated even for inspection objects O for which it is difficult to obtain a large amount of training data. The inspection objects O of different types may be, for example, animals, plants, or objects subjected to non-destructive testing.

[0116] Even in such a system, multiple trained models with different numbers of input frames can be prepared, as shown in Figures 8(a) to 8(c), and the system can operate according to the flow shown in Figure 7.

[0117] In the control unit 20 according to this embodiment, image processing can include super-resolution processing for improving the resolution of an image. In this configuration, the super-resolution processing unit 116 performs processing using a trained model, making it possible to perform real-time processing for obtaining an image with an improved resolution even immediately after capturing an image.

[0118] In the first to third embodiments, the image processing unit 22 performs noise reduction processing or super-resolution processing on a moving image of radiographic images using a trained model corresponding to the number of acquired frames. However, the image processing unit 22 may perform other image processing on the moving image of radiographic images using a trained model corresponding to the number of acquired frames. For example, the image processing unit 22 may perform gradation processing, emphasis processing, grid stripe reduction processing, etc., performed by the diagnostic image processing unit 27 on the moving image using a trained model.

[0119] In this case, the training data may include a set of data in which one or more images before various processing are used as input data and one image after various processing is used as ground truth data. The various processing may be performed by any known method. These processing may be performed according to a region of interest set in a radiological image. For example, gradation processing may be performed to broaden the gradation of the region of interest, and enhancement processing may be performed to enhance the region of interest. The trained model may have the same configuration as the trained model described in Example 1. The trained model may also be generated using transfer learning. The image processing unit 22 may perform some of these processes using the trained model, and perform other processes as diagnostic image processing, for example, by rule-based processing.

[0120] In this case, the inference processing unit 262 can prepare multiple trained models with different numbers of input frames as shown in Figures 8(a) to 8(c) as described in the first to third embodiments, and operate according to the flow shown in Figure 7. When the input data consists of multiple frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the training data, as in the first embodiment.

[0121] Furthermore, the inference processing unit 262 may use a trained model to perform image processing that combines the noise reduction processing described in Example 1, the super-resolution processing described in Example 3, and the diagnostic image processing described above. In this case, the training data may include a set of data in which one or more images before the various combined image processing processes are used as input data and one image after the various processing processes is used as ground truth data. Even in this configuration, when the input data consists of multiple frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the training data, as described above. Therefore, the image processing performed by the inference processing unit 262 can include at least one of noise reduction processing, super-resolution processing, gradation processing, enhancement processing, and gridline reduction processing. With this configuration, the control unit 20 can apply desired image processing to moving images efficiently and in real time using the trained model.

[0122] (Variation 1) The machine learning model used by the inference processing unit 262 can be a CNN configuration that combines any layer configuration, such as a Variational Auto-Encoder (VAE), a Fully Convolutional Network (FCN), SegNet, DenseNet, etc. The machine learning model may also be configured using, for example, a Vision Transformer (VIT).

[0123] (Variation 2) Furthermore, the training data for the various trained models is not limited to data obtained using the radiation detector itself that actually performs imaging, but may be data obtained using the same type of radiation detector or the same kind of radiation detector, depending on the desired configuration. Note that the trained models according to the above-described embodiments and modifications are considered to extract, for example, the magnitude of the luminance values ​​of the radiographic image, the order, gradient, position, distribution, continuity, etc. of the bright and dark areas as part of the feature quantities, and use these for estimation processing related to the generation of radiographic images that have been subjected to various image processing.

[0124] Furthermore, the trained models according to the above-described embodiments and modifications can be provided in the control unit 20. The trained models may be configured, for example, by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or by a circuit that performs a specific function such as an ASIC. These trained models may also be provided in a separate server device connected to the control unit 20. In this case, the control unit 20 can use the trained models by connecting to a server or the like that has the trained models via any network such as the Internet. Here, the server that has the trained models may be, for example, a cloud server, a fog server, an edge server, or the like.

[0125] (Variation 3) In the above-described embodiment and modified examples, the radiation detector 10 is an indirect conversion type detector that converts radiation into visible light using the phosphor 11 and then converts the visible light into an electrical signal using a photoelectric conversion element. However, the radiation detector 10 may also be a direct conversion type detector that directly converts incident radiation into an electrical signal.

[0126] (Other Examples) The present disclosure can also be realized by providing software (programs) that realize one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the programs. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0127] In this case, the processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA), and may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0128] The above disclosure includes the following configurations, methods, and programs. (Configuration 1) 1. An image processing device that applies image processing to a moving image including a plurality of frames of radiographic images, a selection unit that selects a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of acquired frames; an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed; An image processing device comprising: (Configuration 2) 2. The image processing device according to claim 1, wherein the plurality of trained models include a trained model in which the number of input frames is 1 and a trained model in which the number of input frames is greater than 1. (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein the inference processing unit inputs a first frame, which is the frame to be processed, and zero or more second frames acquired before the first frame, into the selected trained model according to the number of frames to be input to the selected trained model, and infers an image by applying the image processing to the first frame. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the selection unit selects, from the plurality of trained models, a trained model in which the number of input frames is equal to or less than the number of acquired frames and in which the number of input frames is greater than other trained models in which the number of input frames is equal to or less than the number of acquired frames, according to the number of acquired frames. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein each of the plurality of trained models is obtained by training using training data including a number of images corresponding to the number of input frames and an image obtained by performing the image processing on an image to be processed among the images. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the image processing includes at least one of noise reduction processing that reduces noise in an image, super-resolution processing that improves image resolution, gradation processing that adjusts gradation of an image, enhancement processing that emphasizes specific pixels in an image, and grid stripe reduction processing that reduces grid stripes in an image. (Configuration 7) The plurality of trained models include a trained model group corresponding to a shooting mode, The image processing device according to any one of configurations 1 to 6, wherein the selection unit selects the trained model to be used for the image processing of the frame to be processed from the trained model group corresponding to the shooting mode of the moving image to be processed, according to the number of frames acquired. (Configuration 8) The image processing device according to configuration 7, wherein the imaging mode is set based on at least one of the sensitivity of a detector used for imaging, a bias voltage, noise characteristics, an amplification factor during readout, a frame rate, an image size, an accumulation time during signal reception, and an imaging technique. (Configuration 9) 9. The image processing device according to configuration 7 or 8, wherein the selection unit does not count the number of frames acquired in a shooting mode different from the shooting mode of the moving image to be processed as the number of acquired frames. (Configuration 10) 10. The image processing device according to any one of configurations 7 to 9, wherein the selection unit counts the number of frames acquired by a first photographing operation in a predetermined photographing mode that is performed after a first photographing operation in the predetermined photographing mode as the number of frames acquired. (Configuration 11) an image processing device according to any one of configurations 1 to 10; a radiation generating device that irradiates radiation; a radiation detection device that detects the irradiated radiation; 1. A radiography system comprising: (Method 1) 1. A method of operating an image processing apparatus for applying image processing to a motion image comprising a plurality of radiographic image frames, comprising: Selecting a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of frames acquired; performing an inference process using the selected trained model in the image processing of the frame to be processed; A method of operating an image processing device, comprising: (Program 1) A program that, when executed by a computer, causes the computer to perform the method for operating an image processing device described in Method 1.

[0129] Although the present invention has been described above with reference to examples and modifications, the present invention is not limited to the above examples and modifications. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. Furthermore, the above-described examples and modifications can be combined as appropriate within the scope of the present invention. [Explanation of symbols]

[0130] 10: Radiation detector, 20: Control unit (image processing device), 22: Image processing unit

Claims

1. 1. An image processing device that applies image processing to a moving image including a plurality of frames of radiographic images, a selection unit that selects a trained model to be used for image processing of the frame to be processed from a plurality of trained models that have different numbers of input frames according to the number of frames acquired since the start of shooting the moving image; an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the plurality of trained models include a trained model in which the number of input frames is one and a trained model in which the number of input frames is greater than one.

3. 2. The image processing device according to claim 1, wherein the inference processing unit inputs a first frame, which is the frame to be processed, and zero or more second frames acquired before the first frame, into the selected trained model according to the number of frames to be input of the selected trained model, and infers an image by applying the image processing to the first frame.

4. 2. The image processing device according to claim 1, wherein the selection unit selects, from the plurality of trained models, a trained model in which the number of input frames is equal to or less than the number of acquired frames and in which the number of input frames is greater than other trained models in which the number of input frames is equal to or less than the number of acquired frames, according to the number of acquired frames.

5. 2. The image processing device according to claim 1, wherein each of the plurality of trained models is obtained by learning using training data including a number of images corresponding to the number of input frames and an image obtained by performing the image processing on an image to be processed among the images.

6. 2. The image processing device according to claim 1, wherein the image processing includes at least one of a noise reduction process that reduces noise in an image, a super-resolution process that improves image resolution, a gradation process that adjusts gradation of an image, an enhancement process that enhances specific pixels in an image, and a grid stripe reduction process that reduces grid stripes in an image.

7. The plurality of trained models include a trained model group corresponding to a shooting mode, The image processing device according to claim 1 , wherein the selection unit selects the trained model to be used for the image processing of the frame to be processed from the trained model group corresponding to the shooting mode of the moving image to be processed, according to the number of frames acquired.

8. 8. The image processing device according to claim 7, wherein the imaging mode is set based on at least one of the sensitivity of a detector used for imaging, a bias voltage, noise characteristics, an amplification factor during readout, a frame rate, an image size, an accumulation time during signal reception, and an imaging technique.

9. The image processing device according to claim 7 , wherein the selection unit does not count, in the number of acquired frames, the number of frames acquired in a shooting mode different from the shooting mode of the moving image to be processed.

10. 8. The image processing device according to claim 7, wherein the selection unit counts the number of frames acquired in a first photographing operation in a predetermined photographing mode that is performed after the first photographing operation in the predetermined photographing mode as the number of frames acquired.

11. An image processing device according to any one of claims 1 to 10; a radiation detection device that detects radiation irradiated from the radiation generation device; 1. A radiography system comprising:

12. 1. A method of operating an image processing apparatus for applying image processing to a motion image comprising a plurality of radiographic image frames, comprising: Selecting a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of frames acquired since the start of shooting the moving image; performing an inference process using the selected trained model in the image processing of the frame to be processed; A method of operating an image processing device, comprising:

13. A program that, when executed by a computer, causes the computer to execute the method for operating an image processing apparatus according to claim 12.

Images