Image processing device, radiographic imaging system, method of operating image processing device, and program

Abstract

To provide an image processing device capable of applying image processing to a moving image suitably in real-time even immediately after photography.SOLUTION: An image processing device for applying image processing to a moving image containing multiple frames of radiographic images, comprises: a selection unit that selects a learned model to be used for image processing of the frame being the processing target from among the multiple learned models with mutually-different input frame numbers according to the acquired frame numbers; and an inference processing unit that performs inference processing using the selected learned model in the image processing of the frame being the processing target.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device, a radiation imaging system, an operation method of an image processing device, and a program. [Background technology]

[0002] In recent years, radiography systems equipped with a detector for detecting radiation such as X-rays have been widely used in the fields of industry and medicine. In particular, in the field of X-ray video photography, digital radiography systems that convert incident X-rays into visible light using a phosphor and obtain a video image using a semiconductor sensor have become widespread. Here, a video image refers to a set of multiple still images collected continuously, and each still image in the video image will be referred to as a frame hereinafter.

[0003] In such a radiography system, various image processing is applied to the image acquired by the semiconductor sensor to enhance diagnostic value. One example is noise reduction processing. In a series of imaging processes, various noises such as quantum noise due to fluctuations of X-ray quanta and system noise generated from detectors and circuits are generated and superimposed on the image. This phenomenon may deteriorate the graininess of the obtained moving image and reduce diagnostic performance. In particular, in medical X-ray video imaging, imaging with a small X-ray dose is recommended from the viewpoint of exposure to the subject, so it is important to apply image processing to the captured image to reduce noise appropriately in order to improve diagnostic performance and improve image quality.

[0004] In response to this, Patent Document 1 proposes a rule-based technology that creates rules for accurately determining motion from video images by considering the effects of noise, and performs suitable noise reduction by weighting and adding multiple frames of video images in a time series according to the determination results. Furthermore, more advanced noise reduction processing that applies machine learning-based technologies such as deep learning has been put to practical use in recent years. For example, Non-Patent Document 1 proposes a technology that inputs frames before and after a frame to be subjected to noise reduction, and obtains a noise-reduced image using a trained neural network. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-48782 A [Non-patent literature]

[0006] [Non-Patent Document 1] “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”, M Tassano,et.al, IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1351~1360 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques may have the following problems. According to Patent Document 1, by combining rule-based motion detection with a recursive filter, it is possible to use an image of a frame prior to the current frame (hereinafter referred to as a past frame) and perform weighted addition combining time information and spatial information. However, with rule-based motion detection processing, it is difficult to create appropriate rules for all cases of various subject structures contained in a captured image, and afterimages may occur due to noise reduction.

[0008] Also, according to Non-Patent Document 1, a good noise reduction effect can be obtained by processing that applies machine learning-based technology. However, the configuration described in Non-Patent Document 1 requires multiple frames to be input to the neural network, which causes a problem that suitable processing cannot be performed until all frames are acquired. Furthermore, the configuration described in Non-Patent Document 1 requires input of past frames and future frames after the current frame in addition to the current frame. Therefore, after acquiring the current frame, it is difficult to perform processing (hereinafter referred to as real-time processing) that displays the results of processing the current frame before acquiring the next frame.

[0009] Here, in medical X-ray video imaging, from the viewpoint of radiation exposure to the subject, it is desirable to output all captured frames as images, and a configuration is required that does not cause unnecessary radiation exposure. Also, in order to perform prompt medical treatment, it is required to provide an image that has been suitably processed in real time, such as by applying image processing such as noise reduction, even when there is only one image frame immediately after imaging.

[0010] One embodiment of the present disclosure has been made in consideration of the above problems, and one of its objectives is to provide an image processing device that can apply image processing to moving images appropriately and in real time, even immediately after shooting. [Means for solving the problem]

[0011] An image processing device according to one embodiment of the present disclosure is an image processing device that applies image processing to a moving image including a plurality of radiation image frames, and is equipped with a selection unit that selects a trained model to be used for the image processing of a frame to be processed from a plurality of trained models having different numbers of input frames in accordance with the number of frames being acquired, and an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed. Effect of the Invention

[0012] According to an embodiment of the present disclosure, image processing can be applied to moving images suitably and in real time even immediately after shooting. [Brief description of the drawings]

[0013] [Figure 1] 1 shows an example of a schematic configuration of a radiation imaging system and a radiation detector according to a first embodiment. [Diagram 2] 2 illustrates an example of a schematic configuration of a control unit and a noise reduction processing unit according to the first embodiment. [Figure 3A] 1 illustrates an example of a schematic configuration of a trained model according to a first embodiment. [Figure 3B] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Figure 3C] FIG. 4 is a diagram for explaining an operation example of a learning process according to the first embodiment. [Figure 4] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Diagram 5] 4 is a table for explaining the operation of the radiation imaging system and the noise reduction processing unit according to the first embodiment. [Figure 6] 4 shows examples of images before and after image processing according to the first embodiment. [Figure 7] 4 is a flowchart of the operation of the radiation imaging system according to the first embodiment. [Figure 8] 1 illustrates an example of a schematic configuration of a CNN according to a first embodiment. [Figure 9A] FIG. 11 is a schematic diagram showing the operation of a radiation imaging system according to the second embodiment. [Figure 9B] FIG. 11 is a schematic diagram showing the operation of a radiation imaging system according to the second embodiment. [Figure 9C] FIG. 11 is a schematic diagram showing the operation of a radiation imaging system according to the second embodiment. [Figure 9D] FIG. 11 is a schematic diagram showing the operation of a radiation imaging system according to the second embodiment. [Figure 9E] FIG. 11 is a schematic diagram showing the operation of a radiation imaging system according to the second embodiment. [Figure 10] 1 illustrates an example of a schematic configuration of a CNN according to a second embodiment. [Figure 11] 13 illustrates an example of a schematic configuration of an image processing unit and a super-resolution processing unit according to a third embodiment. [Figure 12] 13 illustrates an example of a schematic configuration of a trained model according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present disclosure is applied or various conditions. In addition, the same reference numerals are used in the drawings to indicate elements that are the same or functionally similar.

[0015] In the following, a radiography system using X-rays as an example of radiation will be described. However, the radiation may be X-rays or other radiation. In the following examples, the term radiation may include, for example, electromagnetic radiation such as X-rays and gamma rays, and particle radiation such as alpha rays, beta rays, particle beams, proton beams, heavy ion beams, and meson beams.

[0016] In the following, the machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. Neural networks and deep learning may also be used. Appropriately, any of the above algorithms that can be used can be used in the following embodiments and modifications. Furthermore, learning data refers to a data set used for learning the machine learning model, and is composed of a pair of input data input to the machine learning model and correct answer data (teacher data) that is the correct answer for the output result of the machine learning model.

[0017] A trained model refers to a model that has been trained in advance using appropriate training data for a machine learning model that follows any machine learning algorithm, such as deep learning. However, although a trained model has been obtained by training in advance using appropriate training data, this does not mean that no further training is performed, and additional training can be performed. Additional training can also be performed after the device is installed at the site of use.

[0018] Example 1 (Configuration of Radiography System) Hereinafter, a radiation imaging system, an image processing device, and an operation method of the image processing device according to a first embodiment of the present disclosure will be described with reference to Fig. 1(a) and Fig. 1(b). Fig. 1(a) shows an example of a schematic configuration of a radiation imaging system 1 according to this embodiment. In the following description, an object O to be inspected is described as a human body, but the object O to be inspected photographed by the radiation imaging system according to the present disclosure is not limited to a human body, and may be other animals or plants, or objects to be subjected to non-destructive inspection, etc.

[0019] The radiation imaging system 1 according to this embodiment is provided with a radiation detector 10, a control unit 20, a radiation generator 30, an input unit 40, and a display unit 50. The radiation imaging system 1 may also include an external storage device 70, such as a server, connected to the control unit 20 via a network 60, such as the Internet or an intranet.

[0020] The radiation generator 30 includes a radiation generating source such as an X-ray tube and is capable of emitting radiation. The radiation detector 10 detects the radiation emitted by the radiation generator 30 and is capable of generating a radiation image corresponding to the detected radiation. Therefore, the radiation detector 10 can generate a radiation image of the object O to be inspected by detecting the radiation that is emitted by the radiation generator 30 and has passed through the object O to be inspected.

[0021] Here, Fig. 1(b) shows an example of a schematic configuration of a radiation detector 10 according to this embodiment. The radiation detector 10 is provided with a phosphor 11 and an imaging sensor 12. The phosphor 11 converts radiation incident on the radiation detector 10 into light of a wavelength detectable by the imaging sensor 12. The phosphor 11 may include, for example, CsI or GOS (Gd2O2S). The imaging sensor 12 includes a photoelectric conversion element made of, for example, a-Si or crystalline Si, and is capable of detecting light corresponding to the radiation converted by the phosphor 11 and outputting a signal corresponding to the detected light. The radiation detector 10 is capable of generating a radiation image by performing A / D conversion or the like on the signal output by the imaging sensor 12.

[0022] 1(b), the radiation detector 10 may include a calculation unit, an A / D conversion unit, etc. Also, a grid may be provided between the radiation detector 10 and the object O to reduce scattered radiation that is generated when radiation passes through the object O and reaches the radiation detector 10.

[0023] The control unit 20 is connected to the radiation detector 10, the radiation generator 30, the input unit 40, and the display unit 50. The control unit 20 can acquire a radiation image output from the radiation detector 10 and perform image processing on the radiation image, and control the driving of the radiation detector 10 and the radiation generator 30. In this way, the control unit 20 can control the radiation generator 30 to generate radiation under predetermined imaging conditions at appropriate timing, and can capture video at any frame rate. The control unit 20 can also function as an example of an image processing device.

[0024] The control unit 20 may be connected to an external storage device 70 via an arbitrary network 60 such as the Internet or an intranet, and may acquire radiological images and the like from the external storage device 70. Furthermore, the control unit 20 may be connected to other radiation detectors, radiation generators, and the like via the network 60. The control unit 20 may be connected to the external storage device 70 and the like by wire or wirelessly.

[0025] The input unit 40 includes input devices such as a mouse, a keyboard, a trackball, a touch panel, etc., and can be operated by an operator to input instructions to the control unit 20. The display unit 50 includes, for example, any monitor, and can display information and images output from the control unit 20, information input by the input unit 40, etc.

[0026] In this embodiment, the control unit 20, input unit 40, display unit 50, etc. are configured as separate devices, but they may also be configured as an integrated unit. For example, the input unit 40 and the display unit 50 may be configured as a touch panel display. In this embodiment, the control unit 20 configures an image processing device, but it is only necessary for the image processing device to be able to acquire a radiation image and perform image processing on the radiation image, and it is not necessary for the image processing device to control the driving of the radiation detector 10 or the radiation generator 30.

[0027] The control unit 20 may be connected to the radiation detector 10, the radiation generator 30, etc. by wire or wirelessly. Furthermore, the external storage device 70 may constitute an image system such as a PACS in a hospital, or may be a server outside the hospital.

[0028] (Control Unit Configuration) Next, a more specific configuration of the control unit 20 will be described with reference to Fig. 2(a) and Fig. 2(b). Fig. 2(a) shows an example of a schematic configuration of the control unit 20 according to this embodiment, and Fig. 2(b) shows an example of a schematic configuration of the noise reduction processing unit 26 according to this embodiment. The control unit 20 is provided with an acquisition unit 21, an image processing unit 22, a display control unit 23, a drive control unit 24, and a storage unit 25.

[0029] The acquiring unit 21 can acquire radiographic images output by the radiation detector 10, various information input by the input unit 40, etc. The acquiring unit 21 can also acquire radiographic images, patient information, etc. from an external storage device 70, etc.

[0030] The image processing unit 22 is provided with a noise reduction processing unit 26 and a diagnostic image processing unit 27, and can perform image processing according to the present disclosure on the radiological image acquired by the acquisition unit 21. In this embodiment, noise reduction processing will be described as an example of image processing performed by the image processing unit 22.

[0031] 2(b), the noise reduction processing unit 26 is provided with a learning processing unit 261 and an inference processing unit 262. In addition to the configuration of the inference processing unit 262 and the trained model selection unit 263, the learning processing unit 261 is provided with a training data generation unit 264 and a parameter update unit 265. With this configuration, the noise reduction processing unit 26 can train a machine learning model for performing noise reduction processing, and apply noise reduction processing suitable for radiological images using the machine learning model.

[0032] The diagnostic image processor 27 can perform diagnostic image processing for converting the image on which the noise reduction is performed by the noise reduction processor 26 into an image suitable for diagnosis. The diagnostic image processing includes, for example, gradation processing for adjusting the gradation of the image, emphasis processing for emphasizing specific pixels in the image, and grid stripe reduction processing for reducing grid stripes in the image. The diagnostic image processor 27 may perform gradation processing, emphasis processing, grid stripe reduction processing, etc., according to a region of interest (ROI) set in the radiation image. For example, the gradation processing may be performed so as to widen the gradation of the region of interest, and the emphasis processing may be performed so as to emphasize the region of interest. Here, the region of interest may be set according to an instruction from an operator, or may be set based on the imaging site, disease name information, finding information, etc.

[0033] Next, a description will be given of the configuration of the learning processing unit 261. The learning processing unit 261 performs a learning process that is applied when training a machine learning model, and includes an inference processing unit 262 and a trained model selection unit 263, as well as a training data generation unit 264 and a parameter update unit 265.

[0034] When performing the learning process, an image is input to the learning processing unit 261, and learning data is created by the learning data generating unit 264. Here, an example of a configuration is shown in which an image to which artificial noise has been added is used as input data and an image to which no artificial noise has been added is used as correct answer data, as a set of learning data for learning noise reduction processing. The learning data generating unit 264 performs a process of creating a set of learning data by adding artificial noise created by simulating the characteristics of a radiation image to the input image. Here, the noise added by the learning data generating unit 264 reflects the amount of noise calculated by the learning data generating unit 264 that may vary due to manufacturing variations. Details of the artificial noise to be added will be described later.

[0035] The parameter update unit 265 performs processing for updating the parameters of the machine learning model held by the inference processing unit 262 based on the calculation results of the inference processing unit 262 for the input data and the supervised data.

[0036] The inference processing unit 262 infers and generates an image in which image processing is applied to the radiographic image by using the radiographic image as an input of a trained model that has been trained using the training data as described above. In addition, the trained model selection unit 263 selects the trained model used by the inference processing unit 262. Note that details of the selection of the trained model by the trained model selection unit 263 will be described later.

[0037] Here, the learning processing unit 261 does not need to be included in the control unit 20. For example, the configuration of the learning processing unit 261 other than the inference processing unit 262 and the trained model selection unit 263 may be configured on hardware other than the control unit 20, such as a server, and a trained model may be created by performing learning in advance using appropriate training data. In this case, the control unit 20 may access the separate hardware through the inference processing unit 262 and perform only processing using the trained model. Also, a trained model created in advance may be provided in the noise reduction processing unit 26, and the trained model provided with the inference processing unit 262 may be used. Alternatively, the learning processing unit 261 may be included in the control unit 20, allowing additional learning to be performed using training data acquired after installation.

[0038] The display control unit 23 can control the display on the display unit 50, and can cause the display unit 50 to display radiation images before and after image processing by the image processing unit 22, patient information, and the like. The drive control unit 24 can control the driving of the radiation detector 10, the radiation generator 30, and the like. Therefore, the control unit 20 can control the capture of radiation images by controlling the driving of the radiation detector 10 and the radiation generator 30 by the drive control unit 24.

[0039] The storage unit 25 can store an operating system (OS), device drivers for peripheral devices, and programs for implementing various application software including programs for performing processes described below. The storage unit 25 can also store information acquired by the acquisition unit 21, radiographic images processed by the image processing unit 22, and the like. For example, the storage unit 25 can store radiographic images acquired by the acquisition unit 21, and radiographic images that have been subjected to noise reduction processing described below.

[0040] The control unit 20 can be configured using a general computer including a processor, a memory, etc., but may be configured as a dedicated computer for the radiation imaging system 1. Here, the control unit 20 functions as an example of an image processing device according to this embodiment, but the image processing device according to this embodiment may be a separate (external) computer communicatively connected to the control unit 20. The control unit 20 and the image processing device may be, for example, a personal computer, and a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may be used. The processor may be a CPU (Central Processing Unit). The processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or an FPGA (Field-Programmable Gate Array), etc.

[0041] Each function of the control unit 20 may be realized by a processor such as a CPU or MPU executing a software module stored in the storage unit 25. The processor may be, for example, a GPU or FPGA. Each function may be configured by a circuit that performs a specific function, such as an ASIC. For example, the image processing unit 22 may be realized by dedicated hardware such as an ASIC, and the display control unit 23 may be realized by a dedicated processor such as a GPU that is different from the CPU. The storage unit 25 may be configured by any storage medium, such as an optical disk such as a hard disk, or a memory.

[0042] (Machine learning model configuration) Next, an example of a machine learning model constituting the trained model according to this embodiment will be described with reference to Figures 3A to 3C. An example of the machine learning model used by the inference processing unit 262 according to this embodiment is a multi-layer neural network.

[0043] FIG. 3A shows a schematic configuration example of a neural network model according to this embodiment. The configuration 33 of the neural network model shown in FIG. 3A is designed to output inference data 32 in which noise has been reduced according to a previously learned tendency for input data 31. The output noise-reduced inference data 32 is based on the learning content in the machine learning process, and the neural network according to this embodiment learns features for discriminating between signals and noise contained in an input radiation image. In the example shown in FIG. 3A, the input data 31 is a current frame and one or more frames prior to the current frame, and the noise-reduced inference data 32 is a frame in which the noise of the current frame has been reduced. It is also possible to configure a learned model in which the input data 31 is the current frame and the number of input frames is one.

[0044] At least a part of the multi-layered neural network may be, for example, a convolutional neural network (CNN). Also, at least a part of the multi-layered neural network may be based on a technology related to an autoencoder.

[0045] Here, a case where CNN is used as a machine learning model for noise reduction processing of a radiological image will be described. Fig. 3B shows an example of a schematic configuration 33 of a CNN constituting a neural network model according to this embodiment. In the example of the trained model according to this embodiment, when input data 31 which is a radiological image is input, inference data 32 can be output as a radiological image with reduced noise.

[0046] The CNN shown in FIG. 3B is composed of a plurality of layers that process and output a group of input values. The types of layers included in the CNN configuration 33 include a convolution layer, a downsampling layer, an upsampling layer, and a merge layer. Here, the CNN configuration 33 may further include an addition layer 34, and may configure a shortcut that adds input data before output. This allows the CNN to adopt a configuration that learns the difference between input data and output data, and can suitably handle a system that targets noise.

[0047] The convolution layer is a layer that performs convolution processing on a group of input values ​​according to parameters such as the set filter kernel size, the number of filters, the stride value, the dilation value, etc. The number of dimensions of the filter kernel size may also be changed according to the number of dimensions of the input image.

[0048] The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning out or combining input value groups. Specifically, such processing includes, for example, Max Pooling processing.

[0049] An upsampling layer is a layer that performs a process to make the number of output values ​​larger than the number of input values ​​by duplicating the input values ​​or adding values ​​interpolated from the input values, etc. Specifically, such a process includes, for example, upsampling by deconvolution.

[0050] A synthesis layer is a layer that inputs a group of values, such as a group of output values ​​of a layer or a group of pixel values ​​that make up an image, from multiple sources and performs processing to synthesize them by concatenating or adding them.

[0051] Note that different parameter settings for the layers and nodes that make up the neural network may affect the degree to which the trends trained from the learning data can be reproduced during inference. In other words, in many cases, appropriate parameters differ depending on the implementation form, so they can be changed to preferred values ​​as necessary.

[0052] In addition to changing the parameters as described above, CNN may be able to obtain better characteristics by changing the configuration 33 of the CNN. Better characteristics include, for example, outputting radiological images with reduced noise with higher accuracy, shortening the processing time, shortening the time required for training the machine learning model, and the like.

[0053] The CNN configuration 33 used in this embodiment is a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers. In the U-net type machine learning model, the position information (spatial information) made ambiguous in the multiple layers configured as the encoder can be used in the same dimensional layers (layers corresponding to each other) in the multiple layers configured as the decoder (for example, by using skip connections).

[0054] Although not shown, as an example of a modification of the CNN configuration, for example, a layer of an activation function (for example, ReLu: Rectifier Linear Unit) may be incorporated before and after the convolution layer.

[0055] Through these steps of CNN, noise features can be extracted from the input radiological image.

[0056] Here, the learning processing unit 261 includes a parameter update unit 265. As shown in Fig. 3C, the parameter update unit 265 calculates a loss function from inference data 32 obtained by applying the neural network model of the inference processing unit 262 to input data 31 in the learning data, and supervised data 35 in the learning data. After that, the parameter update unit 265 performs a process of updating the parameters of the neural network model based on the calculated loss function. Here, the loss function indicates the error between the inference data 32 and the supervised data 35.

[0057] The parameter update unit 265 can update the filter coefficients of the convolution layer, for example, by using the backpropagation method so as to reduce the error between the inference data 32 represented by the loss function and the correct answer data 35. The backpropagation method is a method of adjusting parameters between each node of the neural network so as to reduce the above error. Note that the learning may use a method (dropout) of randomly inactivating units (each neuron or each node) that constitute the CNN.

[0058] Furthermore, the trained model used by the inference processing unit 262 may be one generated using transfer learning. In this case, for example, transfer learning may be performed on a machine learning model trained on a radiographic image of an object O to be inspected that is different in type, etc., to generate a trained model to be used in noise reduction processing. By performing such transfer learning, a trained model can be efficiently generated even for an object O to be inspected that is difficult to obtain a large amount of training data. The object O to be inspected that is different in type, etc., referred to here may be, for example, an animal, a plant, or an object to be subjected to non-destructive inspection.

[0059] Here, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when performing learning multiple times using a machine learning model using CNN as described above, it is effective to perform the processing with the GPU. Therefore, the learning processing unit 261 according to this embodiment uses a GPU in addition to a CPU. Specifically, when executing a learning program including a machine learning model, the CPU and the GPU cooperate to perform calculations to perform learning. Note that in the learning process, calculations may be performed only by the CPU or the GPU. Also, each process of the inference processing unit 262 may be realized by using a GPU like the learning processing unit 261.

[0060] Although the configuration of the machine learning model has been described above, the machine learning model is not limited to the model using CNN as described above. The learning of the machine learning model used in this embodiment may be any model similar to machine learning using a model that can extract (express) the feature amount of learning data such as images by itself through learning.

[0061] Here, the learning processing unit 261 according to this embodiment can use any set of learning data for learning the noise reduction processing. The learning processing unit 261 can use, for example, learning data in which an image to which artificial noise has been added is used as input data and an image to which no artificial noise has been added is used as correct answer data. In addition, learning may be performed using, for example, an image before averaging as input data and an image after averaging as correct answer data, or an image before statistical processing such as MAP (maximum a posteriori probability) estimation processing as input data and an image after statistical processing as correct answer data.

[0062] (Operation of image processing unit) Next, the detailed operation of the image processing unit 22 in moving image shooting will be described with reference to Fig. 4(a) to Fig. 8. In moving image shooting, similar structures are often captured in past frames near the current frame. Therefore, when performing noise reduction on a target pixel in the current frame, not only spatial information such as similar structures in the same frame but also temporal information (time information) such as similar structures in past frames can be used for noise reduction.

[0063] From this viewpoint, by inputting a plurality of frames, the inference processing unit 262 can perform processing while utilizing more time information. In this case, in order to perform real-time processing, the inference processing unit 262 needs to adopt a configuration in which a total of N frames, consisting of the current frame and a predetermined number of past frames, are input as frames to be input. The past frames to be used differ depending on the frame rate during shooting and the required noise reduction performance, but the following will explain the case where N=10 frames are input (the current frame + the past 9 frames are input) as a suitable example.

[0064] FIG. 4(a) is a schematic diagram of the configuration of a neural network when N=10. In the example shown in FIG. 4(a), the number of the current frame is t, and the current frame and past frames numbered t-1 to t-9 are input in sequence to the trained CNN 41. By using such a CNN 41, the inference processing unit 262 can obtain a noise-reduced image F(t) that has been subjected to noise reduction processing using spatial information of the current frame and time information of nine past frames. The trained CNN 41 is trained using a set of training data in which consecutive N=10 frames are used as input data and a correct image corresponding to the frame numbered t is used as correct data, similar to the input described above.

[0065] As for the next frame, the inference processing unit 262 inputs the frame numbered t+1, which corresponds to the current frame, and the past frames numbered t to t-8 to the trained CNN 41, as shown in Fig. 4(b), to obtain a noise-reduced image F(t+1). By performing this in sequence, the inference processing unit 262 can obtain a suitable noise-reduced image in real time.

[0066] The problem here is when N (10) input frames are not available (t=0 to 8) immediately after the start of shooting. Figures 5(a) and 5(b) are tables showing the relationship between CNN input frames and current frame numbers when the number of frames is not available immediately after shooting. As shown in Figure 5(a), the input frames originally expected by CNN 41 are not available until shooting at t=9 is completed, and a situation occurs in which the inference processing unit 262 cannot operate CNN 41 normally.

[0067] As mentioned above, in medical X-ray video imaging, it is desirable to output all captured frames as images from the viewpoint of radiation exposure to the subject, and a configuration is required that does not cause unnecessary radiation exposure. Furthermore, in order to perform prompt medical treatment, it is desirable to provide an image with suitably reduced noise by real-time processing even when there is only one frame image immediately after imaging, so it is desirable to handle imaging where the current frame number is t=0 to 8.

[0068] 6 shows an example of a portion of an image in a certain frame. Image 61 is an example of the image before noise reduction, and images 62 and 63 are examples of the image after noise reduction processing.

[0069] Now, as in case 1 of FIG. 5(b), an example will be described in which processing is performed using an acquired frame (for example, frame with frame number 0) as a frame that has not been acquired when the number of input frames originally expected by the CNN 41 does not exist. In the example shown in FIG. 5(b), the current frame number t is assumed to be 5. In this case, as in image 62 of FIG. 6, not only is the effect of noise reduction low, but artifacts occur in the image, and an image unsuitable for diagnosis is output. On the other hand, as in case 2 of FIG. 5(b), when all the input frames originally expected by the CNN 41 are available (assuming that frames with frame numbers -1 to -4 have been obtained), an appropriate noise reduction effect can be obtained as in image 63 of FIG. 6.

[0070] In view of this situation, the configuration of the radiation imaging system 1 according to this embodiment will be described with reference to Fig. 7 to Fig. 8(c). Fig. 7 shows an example of the flow of the noise reduction processing unit 26 when capturing a moving image. Fig. 8(a) and Fig. 8(b) show an example of the schematic configuration of the CNN used in the inference processing unit 262.

[0071] In this embodiment, three types of CNNs are prepared for use in the inference process, and the trained model selection unit 263 can select an appropriate trained model as the trained model used by the inference processing unit 262. In addition, the three types of CNNs, the first CNN 81 (CCN1), the second CNN 82 (CCN2), and the third CNN 83 (CCN3), use different numbers of input frames, such as N1=1, N2=5, and N3=10, respectively.

[0072] Each CNN is trained by the learning processing unit 261 so as to obtain optimal performance with the number of input frames determined for each. Specifically, the first CNN 81 can train using, for example, a set of a frame with number t to which noise has been added and a frame with number t to which noise has not been added as training data. The second CNN 82 can train using, for example, a set of a frame with number t to which noise has been added, past frames with numbers t-1 to t-4, and a frame with number t to which noise has not been added as training data. Similarly, the third CNN 83 can train using, for example, a set of a frame with number t to which noise has been added, past frames with numbers t-1 to t-9, and a frame with number t to which noise has not been added as training data.

[0073] Hereinafter, the operation of the noise reduction processing unit 26 will be described with reference to Fig. 7. First, in step S701, when a moving image shooting sequence is started, the noise reduction processing unit 26 sets the initial frame number to t=0.

[0074] In step S702, the noise reduction processing unit 26 acquires the t-th frame image via the acquisition unit 21. Initially, the noise reduction processing unit 26 is in a state of acquiring one frame at t=0.

[0075] In step S703, the noise reduction processing unit 26 performs preprocessing for performing appropriate inference processing on the image acquired in step S702. Although the method of preprocessing is not limited, for example, in the noise reduction processing, for example, by performing square root transformation, logarithmic transformation, etc., quantum noise according to the Poisson distribution can be made approximately constant regardless of the intensity of the input radiation. In addition, as the preprocessing, a process of converting so that additive noise can be handled, or a process of making the average value 0 can be used. Furthermore, without being limited to these processes, the noise reduction processing unit 26 may perform appropriate preprocessing according to the contents of the image processing, such as normalizing the data from 0 to 1, or standardizing the data so that the average value is 0 and the standard deviation is 1. The frame that has been preprocessed will be used in the inference processing of the subsequent frame, so it can be temporarily stored in memory until it is used.

[0076] In step S704, the trained model selection unit 263 determines the number of frames currently acquired, and in particular, determines whether t is less than 4. Here, if t is less than 4, the process proceeds to step S705. In step S705, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used by the inference processing unit 262. The inference processing unit 262 performs inference processing using the selected first CNN 81 with one current frame as input. Under such circumstances, since there are few past frames acquired, it is difficult to use time information, and therefore the noise reduction processing unit 26 is configured to perform noise reduction using only spatial information.

[0077] On the other hand, if t is 4 or more, the process proceeds to step S706. In step S706, the trained model selection unit 263 determines whether t is 4 or more and less than 9. If t is 4 or more and less than 9, the process proceeds to step S707. In step S707, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b) as the trained model used by the inference processing unit 262. The inference processing unit 262 performs inference processing using the selected second CNN 82 with a total of five inputs, one current frame and four past frames. In the situation in which the process proceeds to step S707, although not sufficient, a certain number of past frames have been obtained, and the time information is available. For this reason, in step S707, the computational resources of the CNN are configured to be used so that noise reduction can be performed using both spatial information and time information. In this case, the noise reduction processing unit 26 cannot use as much temporal information as the third CNN 83, but can at least prevent the situation that generates artifacts such as those shown in the image 62 of FIG. 6, and can also appropriately reduce noise.

[0078] On the other hand, if t is 9 or more, the process proceeds to step S708. In step S708, since t is 9 or more, a sufficient number of past frames are available to make use of the time information. Therefore, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c) as the trained model to be used by the inference processing unit 262. Using the selected third CNN 83, the inference processing unit 262 performs processing with an input of one current frame and nine past frames, totaling 10 frames, thereby preventing the occurrence of artifacts and reducing noise by making maximum use of the time information.

[0079] In step S709, the noise reduction processing unit 26 performs post-processing on the inference processing result. The post-processing is to perform inverse processing of the various types of conversion such as normalization and smoothing performed in the pre-processing in step S703.

[0080] In step S710, the noise reduction processing unit 26 determines whether or not to end image acquisition. The noise reduction processing unit 26 may determine whether or not to end image acquisition based on, for example, the set shooting conditions or an instruction from the operator. If image acquisition is to be continued, the process proceeds to step S711. In step S711, the noise reduction processing unit 26 adds 1 to the frame number t, and the process proceeds to step S702, where the noise reduction processing unit 26 repeats the processes of steps S702 to S710.

[0081] By the processes of steps S701 to S711 described above, the noise reduction processing section 26 can perform real-time processing so as to obtain an image with suitably reduced noise even when there is only one frame of image immediately after shooting.

[0082] As described above, the radiation imaging system 1 according to this embodiment includes the control unit 20, the radiation generator 30, and the radiation detector 10. The radiation generator 30 functions as an example of a radiation generating device that irradiates radiation, and the radiation detector 10 functions as an example of a radiation detection device that detects the irradiated radiation. The control unit 20 functions as an example of an image processing device that applies image processing to a moving image including a plurality of radiation image frames.

[0083] The control unit 20 includes a trained model selection unit 263 and an inference processing unit 262. The trained model selection unit 263 functions as an example of a selection unit that selects a trained model to be used for image processing of a frame to be processed from a plurality of trained models having different numbers of input frames according to the number of frames acquired. The inference processing unit 262 functions as an example of an inference processing unit that performs inference processing using the selected trained model in image processing of the frame to be processed. Note that the image processing may include noise reduction processing that reduces noise in the image. With this configuration, the radiation imaging system 1 according to this embodiment can apply image processing to a moving image suitably and in real time even immediately after imaging.

[0084] The multiple trained models may include a trained model in which the number of input frames is 1 and a trained model in which the number of input frames is greater than 1. Therefore, the control unit 20 according to this embodiment can apply image processing to a moving image in an appropriate and real-time manner not only in a situation in which multiple frames are acquired, but also in a situation in which only one frame is acquired.

[0085] Furthermore, the inference processing unit 262 can input one first frame, which is a frame to be processed, and zero or more second frames acquired before the first frame, to the selected trained model according to the number of frames input to the selected trained model, and infer an image in which image processing has been applied to the first frame. With this configuration, the control unit 20 can apply image processing using frames prior to the frame to be processed, and can apply image processing to moving images suitably and in real time.

[0086] The trained model selection unit 263 selects, from among the trained models, a trained model in which the number of input frames is equal to or less than the number of acquired frames and which has a larger number of input frames than other trained models in which the number of input frames is equal to or less than the number of acquired frames, according to the number of acquired frames. With this configuration, the control unit 20 can execute image processing using a trained model that can perform more appropriate image processing according to the number of acquired frames.

[0087] Each of the trained models is obtained by training using training data including a number of images corresponding to the number of input frames and an image obtained by performing image processing on an image to be processed among the images. According to this configuration, the trained model to which a plurality of frames are input can use not only spatial information such as a similar structure around the same frame, but also time information such as a similar structure in the plurality of input frames for image processing. Therefore, the control unit 20 according to this embodiment can apply image processing to a moving image more suitably and in real time even immediately after shooting. Regarding the above-mentioned training data, an image to which artificial noise has been added corresponds to a number of images corresponding to the number of input frames before noise reduction processing, and an image before adding artificial noise can correspond to an image to which noise reduction processing has been performed on an image to be processed.

[0088] If the image processing unit 22 is unable to process the entire image at once due to its memory capacity or other performance, the image may be divided into small regions of appropriate size (for example, 256×256 pixels) and processed.

[0089] In addition, the number of frames to be input to the trained model is not limited to N=10, and may be any number equal to or greater than 2. When the influence of the movement of the object to be inspected O becomes large, a structure similar to the target pixel of the current frame cannot be obtained in the temporal information. Therefore, the number of frames to be input can be set within a certain real time period, taking into account the frame rate.

[0090] In addition, in the above example, three types of trained models are used: the first CNN 81 with one frame input, the second CNN 82 with five frames input, and the third CNN 83 with ten frames input. However, the trained model configuration is not limited to this, and the type of trained model to be used and the number of input frames of each model may be set according to a desired configuration. For example, the type of trained model and the number of input frames may be freely changed according to the sensitivity, bias voltage, noise characteristics, amplification factor at the time of readout, frame rate, image size, accumulation time at the time of signal reception, and imaging technique of the detector used for imaging.

[0091] Example 2 The image processing unit according to the second embodiment of the present disclosure will be described with reference to Figs. 9A to 10(c). Figs. 9A to 9E are diagrams showing an example of an operation flow of the radiation imaging system according to the present embodiment. In Figs. 9A to 9E, the fluoroscopic imaging mode refers to an imaging mode in which a moving image is captured by radiation imaging, and the general imaging mode refers to an imaging mode in which a still image is captured by radiation imaging. Here, the imaging mode means, for example, a set of settings of the sensitivity and bias voltage of the radiation detector in the radiation imaging system, noise characteristics, amplification factor at the time of reading, frame rate, image size, accumulation time at the time of signal reception, and imaging technique. Note that the configurations of the radiation imaging system according to the present embodiment other than the image processing unit 22 are the same as the configurations of the radiation imaging system 1 according to the first embodiment, and therefore the same reference numerals are used and the description is omitted.

[0092] Fig. 9A shows a flow when fluoroscopic photography (first fluoroscopic photography) is performed to acquire n+1 frame images from t=0 to n (n≧9), one still image is acquired by general photography, and then fluoroscopic photography (second fluoroscopic photography) is performed again. Here, the first fluoroscopic photography and the second fluoroscopic photography are performed in the same mode (fluoroscopic photography mode 1) of the radiation imaging system. In Fig. 9A, in the second fluoroscopic photography, the frame number is set to t' and is reassigned from 0.

[0093] Here, after the start of imaging, in the fluoroscopic imaging mode 1, from t=0 to 3, the trained model selection unit 263 selects the first CNN 81 (CNN1) shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 uses the selected first CNN 81 to perform inference processing of an input frame of one. Furthermore, in the range from t=4 to 8, the trained model selection unit 263 selects the second CNN 82 (CNN2) shown in FIG. 8(b), and the inference processing unit 262 uses the second CNN 82 to perform inference processing of five frames. Furthermore, in the range from t=9 to n, the trained model selection unit 263 selects the third CNN 83 (CNN3) shown in FIG. 8(c), and the inference processing unit 262 uses the third CNN 83 to perform inference processing of an input frame of ten.

[0094] Thereafter, when shooting in the general shooting mode, the trained model selection unit 263 selects a fourth CNN (CNN4), which is a CNN different from the first CNN 81 to the third CNN 83 and trained exclusively for the general shooting mode, as the trained model to be used for the inference process. The inference processing unit 262 performs the inference process using the selected fourth CNN. Here, it is preferable to use, as the fourth CNN, a CNN trained to suitably perform noise reduction processing according to the characteristics of the general shooting mode with a single frame as input.

[0095] As another configuration, if the fluoroscopic imaging mode used before the general imaging mode is limited, it is possible to use a CNN with multiple frames as input by adding a frame of fluoroscopic imaging performed before the general imaging to the input of the fourth CNN. In this case, the learning data may be an image taken in the general imaging mode and a frame of fluoroscopic imaging performed before the general imaging, and an image obtained by subjecting the image taken in the general imaging mode to noise reduction processing may be used as the correct answer data. Note that learning may be performed using learning data using an image to which artificial noise has been added, as in the first embodiment.

[0096] Next, imaging is again performed in the fluoroscopic imaging mode 1, and the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) for t'=0 to 3. The inference processing unit 262 performs inference processing of an input frame of one sheet using the selected first CNN 81. Furthermore, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b) for t'=4 to 8, and the inference processing unit 262 performs inference processing of an input frame of five sheets using the second CNN 82. Furthermore, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c) for t'=9 to n, and the inference processing unit 262 performs inference processing of an input frame of ten sheets using the third CNN 83.

[0097] In the flow shown in FIG. 9B, the imaging procedure is the same as that in FIG. 9A, but the difference is that the third CNN 83 is used from t'=0 in the second fluoroscopic imaging. As in this example, when the first and second imaging are in the same mode and the time interval between the two fluoroscopic imaging is short, such as when a general imaging mode or other short imaging is sandwiched between them, a configuration may be adopted in which the current frame t'=0 and t=n to n-8 are used as past frames. In this case, the trained model selection unit 263 selects the third CNN 83 from t'=0 in the second fluoroscopic imaging, and the inference processing unit 262 can perform inference processing of 10 input frames using the third CNN 83.

[0098] FIG. 9C shows a flow when fluoroscopic imaging is performed to obtain n+1 frame images from t=0 to n (n≧9), and then the imaging is temporarily stopped and short-term imaging is performed (here, t'=0 to 3), and then fluoroscopic imaging is performed (t″=0~). The temporary stop of imaging and short-term imaging may be performed by repeatedly momentarily turning on and off a switch such as a foot pedal for irradiating radiation.

[0099] In this case, after the start of imaging, during t=0 to 3 in fluoroscopic imaging mode 1, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing of an input frame using the selected first CNN 81. Furthermore, during t=4 to 8, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b), and the inference processing unit 262 performs inference processing of an input frame using the second CNN 82. Furthermore, during t=9 to n, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c), and the inference processing unit 262 performs inference processing of an input frame using the third CNN 83. Thereafter, the trained model selection unit 263 selects the first CNN 81 as the trained model to be used for inference processing between t'=0 and 3, and the inference processing unit 262 performs inference processing for one frame using the first CNN 81. Further thereafter, the radiation imaging system again switches to imaging in fluoroscopic imaging mode 1, and after t"=0, the system adopts a configuration in which it behaves in the same way as after the initial t=0.

[0100] As shown in FIG. 9D, the shooting procedure is the same as that in FIG. 9C, but between t′=0 to 3 and after t″=0, it is also possible to operate the third CNN 83 by using frames going back from t=n as past frames as necessary.

[0101] FIG. 9E shows an example in which fluoroscopic photography is performed in fluoroscopic photography mode 1 to acquire n+1 frame images from t=0 to n (n≧9), and then the mode is switched to another fluoroscopic photography mode 2 to acquire frames from t′=0 onward. In this case, after the start of photography, during the period from t=0 to 3 in fluoroscopic photography mode 1, the trained model selection unit 263 selects the first CNN 81 shown in FIG. 8(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing of an input frame using the selected first CNN 81. Furthermore, during the period from t=4 to 8, the trained model selection unit 263 selects the second CNN 82 shown in FIG. 8(b), and the inference processing unit 262 performs inference processing of an input frame using the second CNN 82. Furthermore, between t=9 and n, the trained model selection unit 263 selects the third CNN 83 shown in FIG. 8(c), and the inference processing unit 262 performs inference processing of the 10 input frames using the third CNN 83.

[0102] Thereafter, the radiation imaging system switches to fluoroscopic imaging mode 2 and processes frames after t' = 0. Figures 10(a) to 10(c) show schematic diagrams of CNNs used in another mode (fluoroscopic imaging mode 2 in this case) by the inference processing unit 262, and each CNN is optimized for learning the characteristics of images in fluoroscopic imaging mode 2.

[0103] As mentioned above, the type of CNN used and the number of input frames for each may be freely changed depending on the image resolution of the system used, the frame rate during shooting, the shooting technique, etc. For example, in a shooting mode with a low frame rate, if the number of input frames to the CNN is too large, it may take too long to prepare the number of frames that will provide maximum performance, or the subject may move too much, making it difficult to use the time information effectively. Therefore, when the frame rate is low, it is effective to configure the number of input frames to the CNN to be reduced.

[0104] 10(a) to 10(c) show an example of a configuration in which the number of input frames is different from that of the CNN shown in FIG. 8(a) to 8(c). In the example shown in FIG. 9D, when t'=0 to 1, the trained model selection unit 263 selects the fifth CNN 101 (CNN5) shown in FIG. 10(a) as the trained model to be used for inference processing. The inference processing unit 262 performs inference processing of an input frame of one sheet using the selected fifth CNN 101. In addition, when t'=2 to 4, the trained model selection unit 263 selects the sixth CNN 102 (CNN6) shown in FIG. 10(b), and the inference processing unit 262 performs inference processing of an input frame of three sheets using the sixth CNN 102. Furthermore, from t′=5 onwards, the trained model selection unit 263 selects the seventh CNN 103 (CNN7) shown in FIG. 10(c), and the inference processing unit 262 performs inference processing on the six input frames using the seventh CNN 103.

[0105] Here, the first to seventh CNNs exemplified above may be CNNs having the same network structure except for the number of input frames, with only the trained parameters being different, or each may have a different network structure.

[0106] As described above, in the control unit 20 according to this embodiment, the multiple trained models can include a trained model group corresponding to the imaging mode. In addition, the trained model selection unit 263 can select a trained model to be used for image processing of the frame to be processed from the trained model group corresponding to the imaging mode of the moving image to be processed according to the number of frames acquired. With this configuration, the control unit 20 can apply image processing to the moving image more suitably and in real time according to the imaging mode. Note that the imaging mode can be set based on at least one of the sensitivity of the detector used for imaging, bias voltage, noise characteristics, amplification factor at the time of reading, frame rate, image size, accumulation time at the time of signal reception, and imaging technique.

[0107] Furthermore, the trained model selection unit 263 may not count the number of frames acquired in a shooting mode different from the shooting mode of the moving image to be processed in the number of acquired frames. In this case, the control unit 20 may prevent a situation in which image processing is not performed appropriately due to the use of frames acquired in a different shooting mode as input.

[0108] Furthermore, in the second shooting in a predetermined shooting mode performed after the first shooting in the predetermined shooting mode, the trained model selection unit 263 can count the number of frames acquired in the first shooting as the number of frames acquired. With this configuration, the control unit 20 can use the frames acquired in the first shooting for image processing in the second shooting when the first shooting and the second shooting are performed in the same mode and the time interval between the two shootings is short. Therefore, even immediately after the start of the second shooting, the time information can be used for image processing, and image processing can be applied to the moving image more suitably and in real time.

[0109] According to the configuration of this embodiment, the radiation imaging system of this embodiment can perform real-time processing to obtain images with appropriately reduced noise for all frames, even if the imaging mode is changed during imaging.

[0110] Example 3 Thus far, the noise reduction processing by the noise reduction processing unit 26 has been described as an example of image processing performed by the image processing unit 22. However, the present disclosure is not limited thereto, and the above configuration can be adopted for any image processing using a machine learning model for a moving image. Here, an example of super-resolution processing for improving image resolution as image processing for a moving image will be described for the image processing unit according to the third embodiment of the present disclosure. Note that the configurations of the radiation imaging system according to this embodiment other than the image processing unit 22 are the same as the configuration of the radiation imaging system 1 according to the first embodiment, and therefore the same reference numerals will be used and description will be omitted.

[0111] 11A shows an example of a schematic configuration of the image processing unit 22 according to the present embodiment. The image processing unit 22 according to the present embodiment includes a super-resolution processing unit 116 instead of the noise reduction processing unit 26.

[0112] 11(b) shows an example of a schematic configuration of the super-resolution processing unit 116. The configuration of the super-resolution processing unit 116 is similar to that of the noise reduction processing unit 26 according to the first embodiment, and the super-resolution processing unit 116 includes a learning processing unit 261 and an inference processing unit 262. The learning processing unit 261 includes a learning data generating unit 264 and a parameter updating unit 265 in addition to the inference processing unit 262 and the learned model selecting unit 263.

[0113] FIG. 12 shows a schematic configuration example of a neural network model used in the super-resolution processing in this embodiment. The configuration 123 of the neural network model shown in FIG. 12 is designed to output inference data 122 with improved resolution according to a previously learned tendency for input data 121. The input data 121 includes one or more frames with a resolution lower than a desired resolution, and may be, for example, a current frame and one or more frames in the past of the current frame. The neural network model has a configuration 123 that learns a set of learning data in which an image with a low resolution is used as input data and an image with a desired resolution is used as correct answer data. The neural network model is configured to output inference data 122, which is an image after super-resolution processing, when input data 121, which is a radiation image, is input according to the configuration 123. It is also possible to configure a learned model in which the number of frames input is one, with the input data 121 being the current frame.

[0114] The learning data can be generated by, for example, a set in which an image with a low resolution is used as input data, and an image with a desired resolution generated by applying a known super-resolution process to the input data is used as correct data. The learning data may also be generated by a set in which an image obtained by using a radiation detector capable of acquiring an image with a desired resolution is used as correct data, and an image with a lowered resolution of the image is used as an input image. The learning data may also be generated by a set in which an image obtained by setting a low resolution as an imaging condition is used as input data, and an image obtained by setting a high resolution (desired resolution) as an imaging condition is used as correct data. In addition, when the input data is a plurality of frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the learning data, as in the first embodiment.

[0115] An example of the machine learning model used by the inference processing unit 262 according to the present embodiment may be a multi-layered neural network, and for example, CNN may be used for at least a part of the multi-layered neural network. Furthermore, a technology related to an autoencoder may be used for at least a part of the multi-layered neural network. The trained model used by the inference processing unit 262 may be generated using transfer learning. In this case, for example, transfer learning may be performed on a machine learning model trained on a radiation image of an object O of a different type, etc., to generate a trained model to be used in the super-resolution processing. By performing such transfer learning, a trained model can be efficiently generated even for an object O of which it is difficult to obtain a large amount of training data. The object O of a different type, etc., referred to here may be, for example, an animal, a plant, or an object of non-destructive testing.

[0116] Even in such a system, for example, as shown in Figures 8(a) to 8(c), multiple trained models with different numbers of input frames can be prepared and operated according to the flow shown in Figure 7.

[0117] In the control unit 20 according to this embodiment, the image processing can include super-resolution processing for improving the resolution of the image. In this configuration, the super-resolution processing unit 116 performs processing using a trained model, so that real-time processing can be performed to obtain an image with an improved resolution even immediately after shooting.

[0118] In the first to third embodiments, the image processing unit 22 performs noise reduction processing or super-resolution processing on a moving image of a radiographic image using a trained model corresponding to the number of acquired frames. In contrast, the image processing unit 22 may perform other image processing as image processing on the moving image of a radiographic image using a trained model corresponding to the number of acquired frames. For example, the image processing unit 22 may perform gradation processing, emphasis processing, grid stripe reduction processing, etc., performed by the diagnostic image processing unit 27 on the moving image using the trained model.

[0119] In this case, the learning data may include a set of data in which one or more images before various processing are used as input data and one image after various processing is used as correct answer data. The various processing may be performed by any known method. These processing may be performed according to a region of interest set in a radiation image. For example, the gradation processing may be performed so as to widen the gradation of the region of interest, and the emphasis processing may be performed so as to emphasize the region of interest. The configuration of the trained model may be the same as that of the trained model described in the first embodiment. Furthermore, the trained model may also be generated using transfer learning. The image processing unit 22 may perform some of these processes using the trained model, and perform other processes as diagnostic image processing, for example, by rule-based processing.

[0120] In this case, the inference processing unit 262 can prepare multiple trained models with different numbers of input frames as shown in Fig. 8(a) to Fig. 8(c) as described in the first to third embodiments, and can operate according to the flow shown in Fig. 7. When the input data is multiple frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the training data, as in the first embodiment.

[0121] Furthermore, the inference processing unit 262 may perform image processing that combines the noise reduction processing described in the first embodiment, the super-resolution processing described in the third embodiment, and the diagnostic image processing described above, using a trained model. In this case, the training data may include a set of data in which one or more images before the various combined image processing are applied are used as input data, and one image after the various processing is applied is used as correct answer data. Even in this configuration, when the input data is a plurality of frames, the frame to be processed and a frame acquired earlier than the frame can be used as input data for the training data, as in the above. Therefore, the image processing performed by the inference processing unit 262 can include at least one of noise reduction processing, super-resolution processing, gradation processing, enhancement processing, and grid stripe reduction processing. According to such a configuration, the control unit 20 can apply desired image processing to a moving image suitably and in real time using a trained model.

[0122] (Variation 1) The machine learning model used by the inference processing unit 262 may be a CNN configuration that combines any layer configuration, such as a Variational Auto-Encoder (VAE), a Fully Convolutional Network (FCN), SegNet, DenseNet, etc. The machine learning model may also be a configuration that uses, for example, a Vision Transformer (VIT).

[0123] (Variation 2) Furthermore, the training data for the various trained models is not limited to data obtained using the radiation detector itself that actually captures images, and may be data obtained using the same type of radiation detector or the same kind of radiation detector, depending on the desired configuration. Note that the trained models according to the above-mentioned embodiments and modified examples are considered to extract, for example, the magnitude of the luminance value of the radiation image, the order, inclination, position, distribution, continuity, etc. of the bright and dark areas as part of the feature amounts, and use these in the estimation process related to the generation of a radiation image that has been subjected to various image processing.

[0124] The trained models according to the above-described embodiments and modifications can be provided in the control unit 20. The trained models may be configured, for example, by a software module executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit performing a specific function such as an ASIC. These trained models may also be provided in another server device connected to the control unit 20. In this case, the control unit 20 can use the trained model by connecting to a server or the like equipped with the trained model via any network such as the Internet. Here, the server equipped with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like.

[0125] (Variation 3) In the above-described embodiment and modified examples, the radiation detector 10 is an indirect conversion type detector that converts radiation into visible light using the phosphor 11 and then converts the visible light into an electrical signal using a photoelectric conversion element. In contrast, the radiation detector 10 may be a direct conversion type detector that directly converts incident radiation into an electrical signal.

[0126] (Other Examples) The present disclosure can also be realized by providing software (programs) that realize one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the programs. The computer has one or more processors or circuits, and may include separate computers or a network of separate processors or circuits to read and execute computer-executable instructions.

[0127] In this case, the processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). The processor or circuitry may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0128] The above disclosure includes the following configurations, methods, and programs. (Configuration 1) 1. An image processing device that applies image processing to a moving image including a plurality of radiographic image frames, A selection unit that selects a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of acquired frames; an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed; An image processing device comprising: (Configuration 2) 2. The image processing device according to claim 1, wherein the plurality of trained models include a trained model in which the number of input frames is 1 and a trained model in which the number of input frames is greater than 1. (Configuration 3) The image processing device of configuration 1 or 2, wherein the inference processing unit inputs a first frame, which is the frame to be processed, and zero or more second frames acquired before the first frame, to the selected trained model according to the number of frames input to the selected trained model, and infers an image by applying the image processing to the first frame. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the selection unit selects, from the plurality of trained models, a trained model in which the number of input frames is equal to or less than the number of acquired frames and in which the number of input frames is greater than the number of input frames of other trained models in which the number of input frames is equal to or less than the number of acquired frames, in accordance with the number of acquired frames. (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein each of the plurality of trained models is obtained by training using training data including a number of images corresponding to the number of frames to be input and an image obtained by subjecting an image to the image processing to a processing target image among the images. (Configuration 6) The image processing device according to any one of configurations 1 to 5, wherein the image processing includes at least one of a noise reduction processing for reducing noise in an image, a super-resolution processing for improving a resolution of an image, a gradation processing for adjusting a gradation of an image, an enhancement processing for enhancing a specific pixel in an image, and a grid stripe reduction processing for reducing grid stripes in an image. (Configuration 7) The plurality of trained models includes a trained model group corresponding to a shooting mode, The image processing device according to any one of configurations 1 to 6, wherein the selection unit selects the trained model to be used for the image processing of the frame to be processed from the trained model group corresponding to the shooting mode of the moving image to be processed according to the number of frames acquired. (Configuration 8) The image processing device according to configuration 7, wherein the imaging mode is set based on at least one of the sensitivity of a detector used for imaging, a bias voltage, noise characteristics, an amplification factor during readout, a frame rate, an image size, an accumulation time during signal reception, and an imaging technique. (Configuration 9) 9. The image processing device according to configuration 7 or 8, wherein the selection unit does not count, in the number of acquired frames, the number of frames acquired in a shooting mode different from the shooting mode of the moving image to be processed. (Configuration 10) The image processing device according to any one of configurations 7 to 9, wherein the selection unit counts the number of frames acquired in a first photographing operation in a predetermined photographing mode, the second photographing operation being performed after the first photographing operation in the predetermined photographing mode, as the number of frames acquired. (Configuration 11) An image processing device according to any one of configurations 1 to 10; A radiation generating device that irradiates radiation; a radiation detection device that detects the irradiated radiation; 1. A radiography system comprising: (Method 1) 1. A method of operating an image processing apparatus for applying image processing to a motion picture comprising a plurality of radiographic image frames, comprising the steps of: Selecting a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of frames acquired; performing an inference process using the selected trained model in the image processing of the frame to be processed; A method of operating an image processing device, comprising: (Program 1) A program that, when executed by a computer, causes the computer to execute the method for operating an image processing device according to method 1.

[0129] Although the present invention has been described above with reference to the embodiments and modifications, the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned embodiments and modifications can be appropriately combined within the scope of the present invention. [Explanation of symbols]

[0130] 10: radiation detector, 20: control unit (image processing device), 22: image processing unit

Claims

1. 1. An image processing device that applies image processing to a moving image including a plurality of radiation image frames, A selection unit that selects a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of acquired frames; an inference processing unit that performs inference processing using the selected trained model in the image processing of the frame to be processed; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the plurality of trained models include a trained model in which the number of input frames is one and a trained model in which the number of input frames is greater than one.

3. 2. The image processing device according to claim 1, wherein the inference processing unit inputs a first frame, which is the frame to be processed, and zero or more second frames acquired in the past from the first frame, to the selected trained model according to the number of frames to be input of the selected trained model, and infers an image by applying the image processing to the first frame.

4. 2. The image processing device according to claim 1, wherein the selection unit selects, from the plurality of trained models, a trained model in which the number of input frames is equal to or less than the number of acquired frames and in which the number of input frames is greater than the number of input frames of other trained models in which the number of input frames is equal to or less than the number of acquired frames, in accordance with the number of acquired frames.

5. 2. The image processing device according to claim 1, wherein each of the plurality of trained models is obtained by learning using training data including a number of images corresponding to the number of input frames and an image obtained by subjecting a processing target image among the images to the image processing.

6. 2. The image processing device according to claim 1, wherein the image processing includes at least one of a noise reduction process for reducing noise in an image, a super-resolution process for improving a resolution of an image, a gradation process for adjusting a gradation of an image, an enhancement process for enhancing a specific pixel in an image, and a grid stripe reduction process for reducing grid stripes in an image.

7. The plurality of trained models includes a trained model group corresponding to a shooting mode, The image processing device according to claim 1 , wherein the selection unit selects the trained model to be used for the image processing of the frame to be processed from the trained model group corresponding to a shooting mode of the moving image to be processed, in accordance with the number of frames acquired.

8. The image processing device according to claim 7 , wherein the imaging mode is set based on at least one of the sensitivity of a detector used for imaging, a bias voltage, noise characteristics, an amplification factor during readout, a frame rate, an image size, an accumulation time during signal reception, and an imaging technique.

9. The image processing device according to claim 7 , wherein the selection unit does not count, in the number of acquired frames, a number of frames acquired in a shooting mode different from a shooting mode of the moving image to be processed.

10. 8. The image processing device according to claim 7, wherein the selection unit counts the number of frames acquired in a first photographing operation in a predetermined photographing mode, the second photographing operation being performed after the first photographing operation in the predetermined photographing mode, as the number of frames acquired.

11. An image processing device according to any one of claims 1 to 10; A radiation generating device that irradiates radiation; a radiation detection device that detects the irradiated radiation; 13. A radiography system comprising:

12. 1. A method of operating an image processing apparatus for applying image processing to a motion picture comprising a plurality of radiographic image frames, comprising the steps of: Selecting a trained model to be used for the image processing of the frame to be processed from a plurality of trained models having different numbers of input frames according to the number of frames acquired; performing an inference process using the selected trained model in the image processing of the frame to be processed; A method of operating an image processing device, comprising:

13. A program that, when executed by a computer, causes the computer to execute the method for operating an image processing apparatus according to claim 12.

Images