Image processing device, radiography system, image processing method, and program

Abstract

To provide an image processing device that performs machine learning-based noise reduction processing and can perform both real-time display and output of moving images. [Solution] The image processing apparatus disclosed herein is An acquisition unit that acquires moving images obtained using a radiation detector, A first image processing unit performs a first image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. The system includes a second image processing unit that performs a second image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting the image to a system different from the display system.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, a radiation imaging system, an image processing method, and a program.

Background Art

[0002] In recent years, radiation imaging systems equipped with a detection unit for detecting radiation such as X-rays have been widely used in fields such as industry and medicine. In particular, in the field of X-ray fluoroscopy, digital radiation imaging systems that convert incident X-rays into visible light using a phosphor and obtain a moving image using a semiconductor sensor have become widespread. Here, a moving image is a set of a plurality of still images collected continuously, and hereinafter, each still image in the moving image is referred to as a frame. Also, a moving image is also expressed as a fluoroscopic image.

[0003] In such a radiation imaging system, various image processes are applied to the image acquired by the semiconductor sensor to enhance the diagnostic ability (an index indicating the value of an image for diagnosis). As an example, noise reduction processing can be mentioned.

[0004] Here, in Patent Document 1, a rule-based technique for performing suitable noise reduction has been proposed. Specifically, a rule for accurately determining motion from a moving image by considering the influence of noise is created, and suitable noise reduction is performed by weighted addition of a plurality of frames of the moving image in time series according to the determination result. A rule-based technique has been proposed.

[0005] Furthermore, recently, more highly performant noise reduction processing applying machine learning-based techniques such as deep learning has been put into practical use. For example, in Non-Patent Document 1, a technique has been proposed in which frames before and after the frame to be subjected to noise reduction are input and a noise reduction processed image is obtained using a learned neural network.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2013-48782 [Non-patent literature]

[0007] [Non-Patent Document 1] “FastDVDnet:Towards Real-Time Deep Video Denoising Without Flow Estimation”, M Tassano,et.al,IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2020,pp.1354~1363 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] When capturing moving images (fluoroscopic images), it is necessary to perform both real-time display of the moving images and output of the moving images (for example, output to PACS (Picture Archiving and Communication Systems)).

[0009] In some cases, machine learning-based noise reduction can achieve better noise reduction than rule-based noise reduction. However, for example, machine learning-based noise reduction may take longer than rule-based noise reduction. Therefore, it may not be possible to perform both real-time display and output of video. As a result, this may impair the convenience of the photographer capturing video (perspective images).

[0010] Therefore, the purpose of this disclosure is to provide an image processing device that can perform machine learning-based noise reduction processing and also perform both real-time display and output of moving images. [Means for solving the problem]

[0011] The image processing apparatus disclosed herein is An acquisition unit that acquires moving images obtained using a radiation detector, A first image processing unit performs a first image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. The system includes a second image processing unit that performs a second image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting the image to a system different from the display system. [Effects of the Invention]

[0012] According to this disclosure, an image processing device can be provided that performs machine learning-based noise reduction processing and can perform both real-time display of moving images and output of moving images. [Brief explanation of the drawing]

[0013] [Figure 1] This is an example of a schematic configuration of a radiography system according to Embodiment 1. [Figure 2] An example of the schematic configuration of the control unit according to Embodiment 1 is shown. [Figure 3] This document shows an example of the general configuration and operation of a trained model according to Embodiment 1. [Figure 4] This is an example of the flow of the control unit according to Embodiment 1. [Figure 5] This is an example 2 of the flow of the control unit according to Embodiment 1. [Modes for carrying out the invention]

[0014] Hereinafter, exemplary embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be modified depending on the configuration of the apparatus to which the present disclosure applies or various conditions. In addition, the same reference numerals are used between drawings to indicate elements that are identical or functionally similar.

[0015] Hereinafter, a radiographic system using X-rays as an example of radiation will be described. However, the radiation may be X-rays or other types of radiation. In the following embodiments, the term "radiation" can include electromagnetic radiation such as X-rays and γ-rays, as well as particle radiation such as α-rays, β-rays, particle beams, proton beams, heavy ion beams, and neutron beams.

[0016] Note that, hereinafter, the machine learning model refers to a learning model by a machine learning algorithm. Specific algorithms of machine learning include the nearest neighbor method, the naive Bayes method, decision trees, and support vector machines. Also, neural networks and deep learning may be used. Appropriately, the applicable ones among the above algorithms can be used and applied to the following embodiments. Also, the learning data refers to a data set used for learning the machine learning model, and is composed of a pair of input data input to the machine learning model and correct answer data (teacher data) that is the correct answer of the output result of the machine learning model.

[0017] Note that the learned model refers to a model that has been learned in advance using appropriate learning data for a machine learning model according to an arbitrary machine learning algorithm such as deep learning. However, although the learned model has obtained it by learning using appropriate learning data in advance, it is not the case that no further learning is performed, and it is assumed that additional learning can be performed. The additional learning can also be performed after the device is installed at the place of use.

[0018] (Embodiment 1) Hereinafter, referring to FIG. 1, an example of the schematic configuration of the radiographic system according to Embodiment 1 of the present disclosure will be shown.

[0019] The radiographic system according to Embodiment 1 is provided with a radiation detector 10, a control unit 20, a radiation generator 30, an input unit 40, and a display unit 50.

[0020] The radiation generator 30 is equipped with a radiation source, such as an X-ray tube, and can emit radiation. The radiation detector 10 can detect the radiation emitted from the radiation generator 30 and generate a radiation image corresponding to the detected radiation.

[0021] The control unit 20 is connected to the radiation detector 10, the radiation generator 30, the input unit 40, and the display unit 50. The control unit 20 can acquire radiation images output from the radiation detector 10, perform image processing on the radiation images, and control the operation of the radiation detector 10 and the radiation generator 30. As a result, the control unit 20 can control the radiation generator 30 to generate radiation under predetermined shooting conditions at the appropriate timing, enabling not only still image capture but also video recording at any frame rate. Video recording includes fluoroscopy, continuous shooting, tomosynthesis, DSA (Digital subtraction angiography), and cone-beam CT (Computed Tomography).

[0022] The input unit 40 is equipped with input devices such as a mouse, keyboard, trackball, or touch panel, and can receive instructions from the control unit 20 when operated by the operator.

[0023] The display unit 50 may include, for example, any monitor and can display information and images output from the control unit 20, as well as information input by the input unit 40. The display unit 50 may also be a display system composed of multiple monitors. The display system may consist of, for example, a general-purpose monitor and a medical monitor.

[0024] The control unit 20 is equipped with an acquisition unit 21, an image processing unit 22, a display control unit 23, a drive control unit 24, a storage unit 25, and an output control unit 102.

[0025] The acquisition unit 21 can acquire radiation images output by the radiation detector 10 and various information input by the input unit 40. The acquisition unit 21 can also acquire radiation images and patient information from external storage devices 70A, such as a PACS (Picture Archiving and Communication System) or a RIS (Radiology Information Systems), via the network 60.

[0026] The image processing unit 22 is equipped with a noise reduction processing unit 26 and a diagnostic image processing unit 27, and can perform the image processing according to this disclosure on radiographic images acquired by the acquisition unit 21. Furthermore, the image processing unit 22 can also perform reconstruction processing to create tomographic images performed in tomosynthesis imaging and cone-beam CT imaging.

[0027] In this first embodiment, we describe an example in which the image processing unit 22 has a real-time image processing unit 22A and a sequential image processing unit 22B. The real-time image processing unit 22A performs image processing each time a moving image is acquired. The sequential image processing unit 22B does not perform image processing each time a moving image is acquired, but performs image processing periodically. The process of performing image processing periodically is also referred to as batch processing.

[0028] The image processing unit 22A includes a noise reduction processing unit 26A and a diagnostic image processing unit 27A. The image processing unit 22B includes a noise reduction processing unit 26B and a diagnostic image processing unit 27B. In this embodiment 1, the noise reduction processing unit 26A and the noise reduction processing unit 26B perform noise reduction processing using the same machine learning model.

[0029] The noise reduction processing units 26A and 26B may perform noise reduction processing using different machine learning models. For example, the machine learning model of the noise reduction processing unit 26A and the machine learning model of the noise reduction processing unit 26B may have different parameters set in the machine learning model. Furthermore, the parameters related to image processing (post-processing) on ​​the image output from the machine learning model may also be different.

[0030] Real-time image processing is an example of the first image processing method. The real-time image processing unit 22A is an example of the first image processing unit. Furthermore, sequential image processing is an example of the second image processing method. The sequential image processing unit 22B is an example of the second image processing unit.

[0031] The reason for having both a real-time image processing unit 22A and a sequential image processing unit 22B is to achieve real-time display. When this radiography system receives an instruction to start fluoroscopy, it is necessary to quickly display the fluoroscopic image received by the acquisition unit 21 on the display unit 50. If there is only one image processing unit 22, it becomes difficult to immediately display the fluoroscopic image on the display unit 50 if noise reduction processing using machine learning is being performed for other purposes (reproduction, reconstruction, output). Noise reduction processing using machine learning requires processing time, thus occupying the image processing unit. As a result, it becomes difficult to quickly display the fluoroscopic image on the display unit 50. In this embodiment 1, for the sake of clarity, an example with two image processing units 22 is described, but the configuration is not limited to this. For example, there may be four image processing units 22 (two for real-time and two for sequential).

[0032] Furthermore, both the image processing unit 22A and the image processing unit 22B perform image processing on the original image. The original image is the image acquired by the radiation detector 10 after processing (offset correction, gain correction, and loss correction) has been performed to correct the characteristics of the radiation detector.

[0033] Aside from the noise reduction processes mentioned above, another process that occupies the image processing unit is the reconstruction process used to create tomographic images in tomosynthesis and cone-beam CT imaging. Reconstruction algorithms include the shift-add method, filtered back projection (FBP), and iterative approximation methods. Iterative approximation methods, in particular, are known to be computationally intensive, and are expected to occupy the image processing unit for a long time.

[0034] The display control unit 23 can control the display of the display unit 50. For example, it can display radiographic images before and after image processing by the image processing unit 22, patient information, etc., on the display unit 50.

[0035] The drive control unit 24 can control the driving of the radiation detector 10 and the radiation generator 30, etc. Therefore, the control unit 20 can use the drive control unit 24 to control the driving of the radiation detector 10 and the radiation generator 30. In other words, the control unit 20 can control the acquisition of radiation images.

[0036] The memory unit 25 can store information and radiation images acquired by the acquisition unit 21. The memory unit 25 can also store programs for implementing various application software, including operating systems (OS), device drivers for peripheral devices, and programs for performing processes described later.

[0037] The output control unit 102 can output radiation images to an external storage device 70B, such as a PACS image storage and communication system, via the network 60.

[0038] The real-time image processing unit 22A processes moving images, such as those acquired by the acquisition unit 21 (e.g., through-view or continuous shooting), using pre-set image processing parameters. It then quickly transfers the processed image to the display control unit 23. By performing these processes in a pipeline, moving images can be streamed in real time. This image processing also includes processes such as converting the image to the display unit's bit (e.g., 8-bit, 32-bit, etc.) so that it can be displayed on the display unit, and adjusting the brightness of the display unit's colors (e.g., gamma correction).

[0039] The sequential image processing unit 22B can play back and reconstruct images stored in the memory unit 25. It can also sequentially perform image processing for purposes such as outputting to an external storage device 70B. The image processing unit 22B can also perform image processing using image processing parameters adjusted in the input unit 40. While the sequential image processing unit 22B generally performs image processing sequentially in the instructed order, it can also perform image processing via interrupts depending on priority.

[0040] The image processing control unit 101 can switch between the image processing unit 22A for real-time processing and the image processing unit 22B for sequential processing, depending on the conditions.

[0041] The 103, which includes the real-time image processing unit 22A and the display control unit 23 described in this embodiment 1, can be implemented by a graphics processor 1. The graphics processor is, for example, a GPU (Graphical Processing Unit). In this embodiment, since display control also needs to be performed within the GPU, a GPU with output for the display unit 50 (for example, what is called a graphics board) is included.

[0042] The graphics processor 1 (103) performs image transfer between image processing units to ensure high-speed processing. This transfer includes, for example, the transfer of images between the noise reduction processing unit 26A and the diagnostic image processing unit 27A, or between the image processing unit 22A and the display control unit 23. The transfer can be performed within the memory of the graphics processor 1 (103) (for example, video memory). When transferring images between different architectures, for example, the image can be copied before transfer. While copying and transferring images takes time, it allows for secure transfer. Alternatively, images can be transferred after exclusive access (after encryption). Transferring images after exclusive access is faster than copying and transferring them. Examples of architectures include CUDA®, DirectX®, Direct3D®, OpenCL®, and OpenGL®, which are examples of GPU architectures.

[0043] The 104, which includes the sequential image processing unit 22B described in Embodiment 1, can be implemented by a graphics processor 2. The graphics processor is, for example, a GPU (Graphical Processing Unit). Unlike the graphics processor 1, the graphics processor 2 does not need to have an output for the display unit 50. The graphics processor 1 may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), etc.

[0044] In this embodiment 1, it is desirable that the performance of the graphics processor 1 (103) is higher than that of the graphics processor 2 (104). In other words, compared to the graphics processor 1 (103), which requires high performance because real-time processing is necessary, the graphics processor 2 (104) may have the same or lower performance. Here, performance may be considered as the processing time when the same processing is performed. Alternatively, it may be considered as an indicator of processing performance such as FLOPS (Floating-point Operations Per Second). Alternatively, it may be considered as the number of cores installed in the graphics processor, or the version of the graphics processor, etc.

[0045] Graphics processors tend to be more expensive the higher their performance. By making graphics processor 2 (104) less powerful than graphics processor 1 (103), the price of the control unit 20 can be reduced. Graphics processor 1 (103) is an example of a first GPU, and graphics processor 2 (104) is an example of a second GPU.

[0046] Furthermore, in the control unit 20, processing other than that performed by the graphics processor 1 (103) and the graphics processor 2 (104) can be handled by the CPU because the processing load is relatively small. However, this does not mean that processing can only be done by the CPU; it can also be done by a GPU, MPU, FPGA, etc.

[0047] (Configuration of the noise reduction processing unit) The configuration of the noise reduction processing unit 26A will be explained using Figure 2. Note that the configuration of the noise reduction processing unit 26B is the same as that of the noise reduction processing unit 26A.

[0048] As shown in Figure 2, the noise reduction processing unit 26A is equipped with a learning processing unit 261. In addition to the inference processing unit 262 and the trained model selection unit 263, the learning processing unit 261 is equipped with a training data generation unit 264 and a parameter update unit 265. Furthermore, the noise reduction processing unit 26A has a pre-processing unit 266 that converts the image input to the noise reduction processing unit 26A into a format suitable for processing by the learning processing unit 261, and a post-processing unit 267 that applies appropriate processing to the output result of the learning processing unit 261. With this configuration, the noise reduction processing unit 26A can train a machine learning model for noise reduction processing. Furthermore, the noise reduction processing unit 26A can apply noise reduction processing suitable for radiographic images using the trained machine learning model. Note that the noise reduction processing unit 26A may also perform noise reduction processing using trained parameters learned by other learning devices. In other words, the noise reduction processing unit 26A does not have to be configured to perform both machine learning model training and noise reduction processing (inference processing using trained parameters).

[0049] Furthermore, the diagnostic image processing unit 27 can perform diagnostic image processing on the image that has undergone noise reduction by the noise reduction processing unit 26A to convert it into an image suitable for diagnosis. Diagnostic image processing includes, for example, tone processing to adjust the gradation of the image, enhancement processing to highlight specific pixels in the image, and grid fringe reduction processing to reduce grid fringes in the image. The diagnostic image processing unit 27 may, for example, perform tone processing, enhancement processing, grid fringe reduction processing, etc., according to a region of interest (ROI) set in the radiographic image. For example, tone processing may be performed to broaden the gradation of the region of interest, and enhancement processing may be performed to enhance the region of interest. Here, the region of interest may be set according to the operator's instructions, or it may be set based on the imaging site, disease name information, findings information, etc.

[0050] Next, the configuration of the learning processing unit 261 will be described. The learning processing unit 261 performs the learning process that is applied when training a machine learning model. The learning processing unit 261 includes an inference processing unit 262, a trained model selection unit 263, a training data generation unit 264, and a parameter update unit 265.

[0051] When performing the learning process, the learning processing unit 261 receives images that have been appropriately processed by the preprocessing unit 266, and the learning data generation unit 264 creates the learning data. Here, an example configuration is shown in which an image with artificial noise added (input data) and an image without added noise (ground truth data) are used as the set of learning data for learning the noise reduction process. The learning data generation unit 264 creates a set of learning data by adding artificial noise, which is created by simulating the features of radiation images, to the input image. Here, the noise added by the learning data generation unit 264 may reflect the amount of noise that may vary due to manufacturing variations of the radiation detector 10, as calculated by the learning data generation unit 264.

[0052] The parameter update unit 265 updates the parameters of the machine learning model held by the inference processing unit 262 based on the calculation results of the inference processing unit 262 on the input data and the ground truth data.

[0053] When a radiation image is input to a trained model that has been trained using the training data described above, the inference processing unit 262 generates an image in which image processing has been applied to the radiation image through inference processing. The trained model selection unit 263 selects a trained model to be used by the inference processing unit 262. Here, the trained models obtained by the series of training processes of the training processing unit 261 may be multiple for each model of radiation detector 10, for example, multiple for each type of phosphor 11, or multiple for each type of imaging sensor 12, or multiple for each binning, sensitivity, image size, frame rate, and imaging procedure for a single model of radiation detector 10. The trained model selection unit 263 selects at least one trained model from among the multiple trained models to be used by the inference processing unit 262.

[0054] Here, a portion of the learning processing unit 261 does not need to be included in the control unit 20. For example, the components other than the inference processing unit 262 and the trained model selection unit 263 may be configured on hardware other than the control unit 20 (such as a server). This hardware creates a trained model by performing training in advance using appropriate training data. In this case, the control unit 20 may access this other hardware via the inference processing unit 262 to obtain the trained model and perform only processing using that trained model. Alternatively, the trained model may be provided in advance in the noise reduction processing unit 26A, and the control unit 20 may perform only processing using that trained model.

[0055] Alternatively, the learning processing unit 261 may be included in the control unit 20, allowing for additional learning using the learning data acquired after installation (sale) at the customer's site.

[0056] (Machine learning model configuration) Next, with reference to Figures 3(a) to 3(c), an example of a machine learning model that constitutes the trained model according to this embodiment will be described. An example of a machine learning model used by the inference processing unit 262 according to this embodiment is a multi-layer neural network.

[0057] Figure 3(a) shows a schematic example of the neural network model according to this embodiment. The neural network model configuration 33 shown in Figure 3(a) is designed to output noise-reduced inference data 32 in response to input data 31, according to a pre-learned trend. The output noise-reduced inference data 32 is based on the learning content in the machine learning process. The neural network according to this embodiment learns features for distinguishing between signals and noise contained in the input radiation image. In the example shown in Figure 3(a), the input data 31 includes the current frame and one or more frames prior to the current frame. Alternatively, the input data 31 includes the current frame and one or more frames in the future. Alternatively, the input data 31 includes any set of frames including the current frame, one or more frames prior to the current frame, and one frame in the future. The noise-reduced inference data 32 is the current frame with reduced noise. It is also possible to configure a trained model with only the current frame (1 image) as input data 31, so that the number of input frames is 1.

[0058] Furthermore, at least a portion of the multi-layer neural network may be a convolutional neural network (CNN), for example. Additionally, at least a portion of the multi-layer neural network may utilize techniques related to autoencoders or vision transformers (ViT).

[0059] This section describes an example of using a Convolutional Neural Network (CNN) as a machine learning model for noise reduction processing of radiographic images. Figure 3(b) shows an example of a schematic configuration 33 of the CNN that constitutes the neural network model according to this embodiment. In the example of the trained model according to this embodiment, when input data 31, which is a radiographic image, is input, inference data 32 can be output as a radiographic image with reduced noise.

[0060] The CNN shown in Figure 3(b) is composed of multiple layers responsible for processing the input data set and producing an output. The types of layers included in the CNN configuration 33 are convolutional layers, downsampling layers, upsampling layers, and merge layers. Here, the CNN configuration 33 further includes an additive layer 34, and it is preferable to configure a shortcut that adds the input data before output. This allows the CNN to adopt a configuration that learns the difference between the input data and the output data, and can suitably handle systems that target noise.

[0061] A convolutional layer is a layer that performs convolution on an input set of values ​​according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The dimensionality of the filter kernel size may also be changed depending on the dimensionality of the input image.

[0062] A downsampling layer is a layer that performs a process to reduce the number of output values ​​to less than the number of input values ​​by decimating or combining input values. Specifically, one example of such a process is Max Pooling.

[0063] An upsampling layer is a layer that performs a process to increase the number of output values ​​to the number of input values ​​by duplicating the input values ​​or adding interpolated values ​​from the input values. Specifically, one example of such a process is upsampling by deconvolution.

[0064] A synthesis layer is a layer that takes input from multiple sources, such as the output values ​​of a certain layer or the pixel values ​​that make up an image, and processes them by concatenating or adding them together.

[0065] It should be noted that different parameter settings for the layers and nodes that make up the neural network may result in differences in the degree to which the trained trends from the training data can be reproduced during inference. In other words, the appropriate parameters often differ depending on the implementation method, so they can be changed as needed.

[0066] In addition to changing the parameters as described above, the CNN may also achieve better characteristics by changing its configuration. These better characteristics include, for example, outputting radiation images with better noise reduction, shorter processing times, and shorter training times for machine learning models.

[0067] The CNN configuration 33 used in this embodiment is a U-net type machine learning model having the functionality of an encoder consisting of multiple layers including multiple downsampling layers, and the functionality of a decoder consisting of multiple layers including multiple upsampling layers. In a U-net type machine learning model, for example, skip connections can be used. That is, positional information (spatial information) that has been obscured in the multiple layers configured as an encoder can be used in layers of the same dimension (layers corresponding to the dimensions of the encoder) in the multiple layers configured as a decoder.

[0068] Although not shown in the diagram, one example of modifying the CNN configuration is to incorporate layers with activation functions (e.g., ReLu: Rectifier Linear Unit) before and after the convolutional layers.

[0069] Through these steps in the CNN, noise features can be extracted from the input radiation images.

[0070] Here, the learning processing unit 261 includes a parameter update unit 265. As shown in Figure 3(c), the parameter update unit 265 calculates a loss function from the inference data 32 obtained by applying the neural network model of the inference processing unit 262 to the input data 31 in the learning data, and from the ground truth data 35 in the learning data. Subsequently, the parameter update unit 265 updates the parameters of the neural network model based on the calculated loss function. Here, the loss function represents the error between the inference data 32 and the ground truth data 35.

[0071] The parameter update unit 265 can update the filter coefficients of the convolutional layer, for example, using backpropagation, so as to reduce the error between the inference data 32 represented by the loss function and the ground truth data 35. Backpropagation is a method for adjusting the parameters between each node of the neural network so as to reduce the above error. In addition, a method of randomly deactivating the units (each neuron or each node) that make up the CNN (dropout) may be used for training.

[0072] Furthermore, the pre-trained model used by the inference processing unit 262 may be generated using transfer learning. In this case, for example, a pre-trained model used for noise reduction processing may be generated by performing transfer learning on a machine learning model trained on radiographic images of objects O of different types. By performing such transfer learning, it is possible to efficiently generate a pre-trained model even for objects O of which it is difficult to obtain a large amount of training data. Here, objects O of different types may be, for example, animals, plants, or objects used in non-destructive testing.

[0073] Here, the GPU can perform calculations efficiently by processing more data in parallel. Therefore, when performing training multiple times using a machine learning model that utilizes a CNN as described above, it is effective to perform the processing on the GPU. Accordingly, the learning processing unit 261 in this embodiment uses a GPU in addition to the CPU. Specifically, when executing a learning program that includes a machine learning model, the CPU and GPU work together to perform calculations to perform learning. Note that the learning process may be performed by the CPU or the GPU alone. Furthermore, each process of the inference processing unit 262 may also be implemented using the GPU in the same way as the learning processing unit 261.

[0074] The above describes the configuration of the machine learning model, but the machine learning model is not limited to the CNN-based model shown above. The machine learning model used in this embodiment can be any machine learning-like model that is capable of extracting (representing) the features of training data such as images through learning.

[0075] Here, the learning processing unit 261 according to this embodiment can use any set of training data for learning noise reduction processing. For example, the learning processing unit 261 can use training data in which images with artificial noise added are input data and images without added noise are ground truth data. In addition, for example, learning may be performed using images before averaging as input data and images after averaging as ground truth data, or images before statistical processing such as MAP (maximum posterior probability) estimation as input data and images after statistical processing as ground truth data. Furthermore, although examples of supervised learning have been shown so far, the learning method is not limited to this, and any unsupervised learning or semi-supervised learning method may be used.

[0076] (Flowchart of the control unit in Embodiment 1) Hereinafter, with reference to Figure 4, an example of the flow of the control unit 20 in Embodiment 1 of the radiography system according to Embodiment 1 of this disclosure will be described.

[0077] As described above, the image processing control unit 101 switches between using the real-time image processing unit 22A and the sequential image processing unit 22B depending on the conditions. This flowchart describes an example of switching based on the purpose of performing image processing.

[0078] The image processing control unit 101 determines the purpose of the image processing (S801). The purpose of the image processing is defined (set), for example, when the radiography system is constructed. Alternatively, the purpose of the image processing is input by the photographer, for example, via a GUI provided on the display unit 50.

[0079] If the purpose is image processing for X-ray vision or photography, image processing is performed by the real-time image processing unit 22A (S802).

[0080] Once image processing is performed, the display unit 50 enables real-time streaming display (S803) under the control of the display control unit 23.

[0081] In S801, if the purpose of the image processing is for output to a PACS or similar system, the image processing is performed by the sequential image processing unit 22B (S807).

[0082] Once image processing is performed, the output control unit 102 controls the output to PACS or other devices (S808).

[0083] If the purpose of image processing in S801 is to reconstruct an image stored in the memory unit 25 or the like, or to reconstruct a tomographic image, the process proceeds to S809.

[0084] In S809, it is determined whether the predicted processing time is longer or shorter than a threshold. The predicted processing time is, for example, a value predicted from the GPU performance or the processing content. If the processing time is short (shorter than the threshold), image processing (S804) is performed by the real-time image processing unit 22A.

[0085] Once image processing is performed, the display unit 50 plays and displays the moving image under the control of the display control unit 23 (S806).

[0086] If, in S809, the predicted processing time is determined to be long (longer than a certain threshold), image processing (S805) is performed by the sequential image processing unit 22B. In other words, no image processing is performed by the real-time image processing unit 22A.

[0087] Once image processing is performed, the display unit 50 plays and displays the moving image under the control of the display control unit 23 (S806).

[0088] By making a decision in S809 based on the predicted processing time, the reconstruction algorithm can switch between the processing in S804 and the processing in S805. For example, if the purpose of image processing is tomosynthesis reconstruction or cone-beam CT reconstruction, the reconstruction algorithm can switch between S804 (real-time image processing) and S805 (sequential image processing). Specifically, for example, the shift-add method and the filtered back projection method have short predicted processing times, so they are processed as real-time image processing. The iterative approximation method has a long predicted processing time, so it is processed as sequential image processing. By performing processing with long predicted processing times in the sequential image processing unit 22B, it is possible to prevent the real-time image processing unit 22A from being occupied for a long time. As a result, the fluoroscopic image can be displayed on the display unit 50 quickly.

[0089] (Flowchart of the control unit in Embodiment 1, part 2) Hereinafter, with reference to Figure 5, an example of the flow of the control unit 20 in Embodiment 1 of the radiography system according to Embodiment 1 of this disclosure will be described.

[0090] As described above, the image processing control unit 101 switches between the real-time image processing unit 22A and the sequential image processing unit 22B depending on the conditions. In this flow, an example is described in which the switching is performed based on the path through which the image is acquired (image source).

[0091] The image processing control unit 101 determines the path of the image (S901). If the image was acquired from the acquisition unit 21, it is necessary to display it on the display unit 50 as soon as possible, so the real-time image processing unit 22A performs image processing (S802).

[0092] Once image processing is performed, the display unit 50 enables real-time streaming display (S803) under the control of the display control unit 23.

[0093] If the image is one acquired from the storage unit 25, it is not necessary to display it immediately, so the sequential image processing unit 22B performs image processing (S805).

[0094] Subsequently, the purpose of the image processing is determined (S801), and playback (S806) or output (S808) is performed according to the result of that determination. The operations of S801, S806, and S808 are as explained with reference to Figure 4.

[0095] (Other embodiments) Furthermore, the disclosed technology can also be realized by performing the following process: that is, the disclosed technology can also be realized by supplying software (programs) that implement one or more functions of the various embodiments described above to a system or device via a network or storage medium, and the computer (or CPU, MPU, etc.) of that system or device reads and executes the program. The computer may have one or more processors or circuits and may include a network of separate computers or separate processors or circuits for reading and executing computer executable instructions. In this case, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). The processor or circuit may also include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0096] (Composition 1) An acquisition unit that acquires moving images obtained using a radiation detector, A first image processing unit performs a first image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. An image processing apparatus comprising: a second image processing unit that performs a second image processing on the acquired video image, including noise reduction processing using a machine learning model and outputting to a system different from the display system.

[0097] (Configuration 2) The aforementioned display system is a monitor, A system different from the aforementioned display system is the image processing device described in Configuration 1, which is a PACS.

[0098] (Composition 3) The image processing apparatus according to configuration 1 or 2, wherein the noise reduction process in the first image processing is a real-time process, and the noise reduction process in the second image processing is a batch process.

[0099] (Composition 4) An image processing apparatus according to any one of configurations 1 to 3, wherein the noise reduction process in the first image processing and the noise reduction process in the second image processing are noise reduction processes using the same machine learning model.

[0100] (Composition 5) The first image processing unit is comprised of a first GPU, The image processing apparatus according to any one of configurations 1 to 4, wherein the second image processing unit is a second GPU with lower performance than the first GPU.

[0101] (Composition 6) When the acquired video image is to be played back or reconstructed, and the predicted processing time for the playback or reconstruction is shorter than a threshold, the first image processing unit performs the playback or reconstruction. The image processing apparatus according to any one of configurations 1 to 5, wherein when the acquired video image is to be played back or reconstructed, and the predicted processing time for the playback or reconstruction is longer than a threshold, the second image processing unit performs the playback or reconstruction.

[0102] (Composition 7) When the acquired video image is to be played back or reconstructed, and the processing for the playback or reconstruction is a shift-add method or a filter-corrected back projection method, the first image processing unit performs the playback or reconstruction. The image processing apparatus according to any one of configurations 1 to 6, wherein when the acquired video image is to be played back or reconstructed, and the process for the playback or reconstruction is performed by a successive approximation method, the second image processing unit performs the playback or reconstruction.

[0103] (Composition 8) The image processing apparatus according to configuration 6, wherein the acquired video image is a video image obtained by performing tomosynthesis imaging or CT imaging.

[0104] (Composition 9) The system further comprises a storage unit for storing the aforementioned moving images, An image processing apparatus according to any one of configurations 1 to 8, which performs the first image processing or the second image processing on a moving image acquired from the storage unit using the second image processing unit.

[0105] (Composition 10) The image processing apparatus according to any one of configurations 1 to 9, wherein the acquired video image is a video image that has been corrected by performing at least one of the following corrections: offset correction, gain correction, and loss correction, which correct the characteristics of the radiation detector.

[0106] (Composition 11) The image processing apparatus according to any one of configurations 1 to 10, wherein the first image processing unit performs image processing using preset image processing parameters, and the second image processing unit performs image processing using image processing parameters set by the operator.

[0107] (Composition 12) The aforementioned radiation detector, A radiography system comprising an image processing device according to any one of configurations 1 to 11, which is connected to the aforementioned radiation detector in a manner that enables communication.

[0108] (Method 1) The acquisition process involves obtaining moving images using a radiation detector, A first image processing step is performed on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. An image processing method for an image processing apparatus, comprising: a second image processing step of performing a second image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a system different from the display system.

[0109] (Program 1) A program that causes a computer to execute the image processing method described in Method 1. [Explanation of Symbols]

[0110] 10. Radiation detectors 20 Control Unit (Image Processing Unit) 22A Image Processing Unit (for real-time use) 22B Image Processing Unit (for sequential operation) 103 Graphics processor 1 104 Graphics Processor 2

Claims

1. An acquisition unit that acquires moving images obtained using a radiation detector, A first image processing unit performs a first image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. An image processing apparatus comprising: a second image processing unit that performs a second image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a system different from the display system.

2. The aforementioned display system is a monitor, The image processing apparatus according to claim 1, wherein the system different from the display system is PACS.

3. The image processing apparatus according to claim 1, wherein the noise reduction process in the first image processing is a real-time process, and the noise reduction process in the second image processing is a batch process.

4. The image processing apparatus according to claim 1, wherein the noise reduction process in the first image processing and the noise reduction process in the second image processing are noise reduction processes using the same machine learning model.

5. The first image processing unit is composed of a first GPU, The image processing apparatus according to claim 1, wherein the second image processing unit is comprised of a second GPU having lower performance than the first GPU.

6. When the acquired video image is to be played back or reconstructed, and the predicted processing time for the playback or reconstruction is shorter than a threshold, the first image processing unit performs the playback or reconstruction. The image processing apparatus according to claim 1, wherein when the acquired video image is to be played back or reconstructed, and the second image processing unit performs the playback or reconstruction when the predicted processing time for the playback or reconstruction is longer than a threshold.

7. When the acquired video image is to be played back or reconstructed, and the processing for the playback or reconstruction is the shift-add method or the filter-corrected back projection method, the first image processing unit performs the playback or reconstruction. The image processing apparatus according to claim 1, wherein when the acquired video image is to be played back or reconstructed, and the process for the playback or reconstruction is performed by a successive approximation method, the second image processing unit performs the playback or reconstruction.

8. The image processing apparatus according to claim 6, wherein the acquired video image is a video image obtained by performing tomosynthesis imaging or CT imaging.

9. The system further comprises a storage unit for storing the aforementioned moving images, The image processing apparatus according to claim 1, wherein the second image processing unit performs the first image processing or the second image processing on a moving image acquired from the storage unit.

10. The image processing apparatus according to claim 1, wherein the acquired video image is a video image that has been corrected by at least one of the following corrections: offset correction, gain correction, and loss correction, which correct the characteristics of the radiation detector.

11. The image processing apparatus according to claim 1, wherein the first image processing unit performs image processing using preset image processing parameters, and the second image processing unit performs image processing using image processing parameters set by the operator.

12. The aforementioned radiation detector, A radiation imaging system comprising an image processing device according to any one of claims 1 to 11, which is connected to the radiation detector in a manner that enables communication.

13. The acquisition process involves obtaining moving images using a radiation detector, A first image processing step is performed on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a display system. An image processing method for an image processing apparatus, comprising: a second image processing step of performing a second image processing on the acquired video image, which includes noise reduction processing using a machine learning model and outputting to a system different from the display system.

14. A program that causes a computer to execute the image processing method described in claim 13.

Images