Image processing device, radiography system, image processing method for image processing device, and program

Abstract

To provide an image processing device that performs noise reduction processing using a pre-trained model and can change the amount of afterimage reduction. [Solution] The image processing apparatus disclosed herein is An image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, The system includes an inference processing unit that uses a plurality of the second frame images and at least one of the first frame image and the third frame image to acquire a frame image with reduced noise compared to the second frame image.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing apparatus, a radiation imaging system, an image processing method for an image processing apparatus, and a program. [Background Art]

[0002] In recent years, radiation imaging systems including a detection unit for detecting radiation such as X-rays have been widely used in fields such as industry and medicine. In particular, in the field of X-ray moving image capturing, digital radiation imaging systems that convert incident X-rays into visible light using a phosphor and obtain moving images using a semiconductor sensor are widely spread. Here, the moving image refers to a set of a plurality of continuously acquired still images, and hereinafter, each individual still image in a moving image is referred to as a frame.

[0003] In such radiation imaging systems, various image processing is applied to images acquired by a semiconductor sensor to improve diagnostic performance (an index indicating the value of an image for diagnosis). One example is noise reduction processing. It is known that various types of noise, such as quantum noise caused by fluctuations in X-ray quanta and system noise generated from detectors, circuits, etc., are generated and superimposed on images during a series of imaging processes. This phenomenon may degrade the granularity of the obtained moving image and lower diagnostic performance.

[0004] In particular, in medical X-ray moving image capturing, imaging with a low X-ray dose is recommended from the perspective of reducing the radiation exposure of the subject. Therefore, in order to improve diagnostic performance, it is important to perform image processing that appropriately reduces noise on the captured image to improve image quality.

[0005] Furthermore, since the same subject is continuously captured in moving image capturing, it is important that there is little flicker (a phenomenon in which the brightness of the moving image fluctuates minutely) in the signal of the subject between respective frames.

[0006] Furthermore, in video recording, it is necessary to obtain moving images in which moving subjects can be clearly seen, so minimizing afterimages is important.

[0007] In this context, Patent Document 1 proposes a rule-based technique for performing suitable noise reduction. Specifically, it proposes creating rules to accurately determine motion from video by considering the effects of noise, and then weighting and adding multiple frames of video in time series according to the determination results. Furthermore, in recent years, more advanced noise reduction processing using machine learning-based techniques such as deep learning has been put into practical use. Non-Patent Document 1 proposes a technique in which frames before and after the frame to be noise-reduced are input, and a noise-reduced image is obtained using a trained neural network. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2013-48782 [Non-patent literature]

[0009] [Non-Patent Document 1] “FastDVDnet:Towards Real-Time Deep Video Denoising Without Flow Estimation”, M Tassano,et.al,IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2020,pp.1354~1363 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] According to Patent Document 1, by combining rule-based motion detection with a recursive filter, it is possible to use images from frames prior to the current frame (hereinafter referred to as past frames) and perform weighted summation that combines the temporal and spatial information of the signal. Furthermore, according to Patent Document 1, the weights of the temporal information and spatial information can be freely changed. Therefore, the noise reduction effect and the afterimage reduction effect can be changed. On the other hand, in rule-based motion detection processing, it is difficult to create appropriate rules for various movements of the subject. Therefore, afterimages may occur, especially with subjects that move a lot, along with noise reduction.

[0011] Furthermore, according to Non-Patent Literature 1, processing using machine learning-based technology can achieve a better noise reduction effect compared to rule-based processing, suppressing noise and afterimages. However, the processing performed by the neural network is a black box. Therefore, it is difficult to change the amount of afterimage reduction by, for example, changing parameters in a trained neural network (trained model). As a result, afterimages may become larger for fast-moving subjects.

[0012] Therefore, one of the objectives of this disclosure is to provide an image processing device that can perform noise reduction processing using a trained model and change the amount of afterimage reduction. [Means for solving the problem]

[0013] The image processing apparatus disclosed herein is An image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, Multiple second frame images and the first frame image and Using at least one of the frame images of the third frame image, a frame image with reduced noise from the second frame image is obtained. Where It is equipped with a science department. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to provide an image processing apparatus that performs noise reduction processing using a trained model and is capable of changing the amount of afterimage reduction. [Brief Description of the Drawings]

[0015] [Figure 1] An example of a schematic configuration of a radiation imaging system and a radiation detector according to Embodiment 1 is shown. [Figure 2] An example of a schematic configuration of a control unit according to Embodiment 1 is shown. [Figure 3] An example of a schematic configuration and an operation example of the trained model according to Embodiment 1 is shown. [Figure 4] An example of a schematic configuration of a CNN according to Embodiment 1 is shown. [Figure 5] It is a diagram for explaining an operation example of a noise reduction processing unit according to Embodiment 1. [Figure 6] It is a flowchart of the operation of the radiation imaging system according to Embodiment 1. [Figure 7] It is a flowchart of the operation of the radiation imaging system according to Embodiment 1. [Mode for Carrying Out the Invention]

[0016] Hereinafter, exemplary embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. However, dimensions, materials, shapes, relative positions of components, and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of the apparatus to which the present disclosure is applied or various conditions. In the drawings, the same reference numerals are used throughout the drawings to indicate elements that are the same or functionally similar.

[0017] Hereinafter, a radiation imaging system using X-rays as an example of radiation will be described. However, the radiation may be X-rays or other types of radiation. In the following embodiments, the term "radiation" can include, for example, electromagnetic radiation such as X-rays and γ-rays, and particle radiation such as α-rays, β-rays, particle beams, proton beams, heavy ion beams, and meson beams.

[0018] In the following description, a machine learning model refers to a learning model generated by a machine learning algorithm. Specific examples of machine learning algorithms include the k-nearest neighbor algorithm, naive Bayes method, decision tree, and support vector machine. Furthermore, a neural network or deep learning may also be used. Any available algorithm among the above algorithms can be appropriately selected and applied to the following embodiments and modifications. In addition, training data refers to a data set used for training a machine learning model, and is composed of pairs of input data input to the machine learning model and correct answer data (training data) that is the correct result for the output of the machine learning model.

[0019] A trained model refers to a model obtained by performing preliminary training using appropriate training data on a machine learning model based on any machine learning algorithm such as deep learning. However, although a trained model is obtained through preliminary training using appropriate training data, it is not fixed such that no further training can be performed, and additional training may also be performed. Additional training can be performed even after the apparatus is installed at the place of use.

[0020] (Embodiment 1) (Configuration of Radiation Imaging System) Hereinafter, a radiation imaging system, an image processing apparatus, and an operating method of the image processing apparatus according to Embodiment 1 of the present disclosure will be described with reference to FIGS. 1(a) and 1(b).

[0021] Figure 1(a) shows a schematic example of the configuration of the radiography system 1 according to this embodiment. In the following description, the object to be inspected O will be described as a human body, but the object to be inspected O photographed by the radiography system according to this disclosure is not limited to a human body, but may be other animals, plants, or objects to be tested using non-destructive testing.

[0022] The radiography system 1 according to this embodiment includes a radiation detector 10, a control unit 20, a radiation generator 30, an input unit 40, and a display unit 50. The radiography system 1 may also include an external storage device 70, such as a server, connected to the control unit 20 via a network 60, such as the Internet or an intranet.

[0023] The radiation generator 30 is equipped with a radiation source, such as an X-ray tube, and can emit radiation. The radiation detector 10 can detect the radiation emitted from the radiation generator 30 and generate a radiation image corresponding to the detected radiation. Therefore, the radiation detector 10 can generate a radiation image of the object under inspection O by detecting the radiation emitted from the radiation generator 30 and passing through the object under inspection O.

[0024] Here, Figure 1(b) shows an example of a schematic configuration of the radiation detector 10 according to this embodiment. The radiation detector 10 is provided with a phosphor 11 and an imaging sensor 12. The phosphor 11 converts radiation incident on the radiation detector 10 into light of a wavelength detectable by the imaging sensor 12. The phosphor 11 may include, for example, CsI or GOS (Gd2O2S). The imaging sensor 12 includes a photoelectric conversion element composed of, for example, a-Si or crystalline Si, and can detect light corresponding to the radiation converted by the phosphor 11 and output a signal corresponding to the detected light. The radiation detector 10 can generate a radiation image by performing A / D conversion or the like on the signal output by the imaging sensor 12.

[0025] Although not shown in Figure 1(b), the radiation detector 10 may include a calculation unit, an A / D conversion unit, etc. Furthermore, a grid may be installed between the radiation detector 10 and the object under inspection O to reduce scattered radiation that is generated when radiation passes through the object under inspection O and reaches the radiation detector 10.

[0026] The control unit 20 is connected to the radiation detector 10, the radiation generator 30, the input unit 40, and the display unit 50. The control unit 20 can acquire radiation images output from the radiation detector 10, perform image processing on the radiation images, and control the operation of the radiation detector 10 and the radiation generator 30. As a result, the control unit 20 can control the radiation generator 30 to generate radiation under predetermined shooting conditions at the appropriate timing, enabling video recording at any frame rate. Furthermore, the control unit 20 can function as an example of an image processing device.

[0027] The control unit 20 may be connected to an external storage device 70 via any network 60 such as the Internet or an intranet, and may acquire radiation images, etc., from the external storage device 70. Furthermore, the control unit 20 may be connected to other radiation detectors, radiation generators, etc., via the network 60. The control unit 20 may be connected to the external storage device 70, etc., by wire or by wireless connection.

[0028] The input unit 40 is equipped with input devices such as a mouse, keyboard, trackball, or touch panel, and can receive instructions from the control unit 20 by being operated by the operator. The display unit 50 includes, for example, any monitor and can display information and images output from the control unit 20, as well as information input by the input unit 40.

[0029] In this embodiment, the control unit 20, input unit 40, display unit 50, etc., are configured as separate devices, but they may be configured as an integrated unit. For example, the input unit 40 and display unit 50 may be configured as a touch panel display. Also, in this embodiment, the control unit 20 constitutes the image processing device, but the image processing device only needs to be able to acquire radiation images and perform image processing on the radiation images, and does not need to control the driving of the radiation detector 10 or the radiation generator 30.

[0030] Furthermore, the control unit 20, radiation detector 10, radiation generator 30, etc. may be connected by wire or wirelessly. In addition, the external storage device 70 may constitute an image system such as PACS within the hospital, or it may be a server outside the hospital.

[0031] (Configuration of the control unit) Next, the more specific configuration of the control unit 20 will be described with reference to Figures 2(a), 2(b), and 2(c).

[0032] Figure 2(a) shows an example of a schematic configuration of the control unit 20 according to this embodiment, and Figure 2(b) shows an example of a schematic configuration of the noise reduction processing unit 26 according to this embodiment. The control unit 20 is provided with an acquisition unit 21, an image processing unit 22, a display control unit 23, a drive control unit 24, and a storage unit 25. Figure 2(c) shows an example of a schematic configuration of the inference processing unit 261 according to this embodiment.

[0033] The acquisition unit 21 can acquire radiation images output by the radiation detector 10 and various information input by the input unit 40. The acquisition unit 21 can also acquire radiation images, patient information, etc. from an external storage device 70 or the like.

[0034] The image processing unit 22 is equipped with a noise reduction processing unit 26 and a diagnostic image processing unit 27, and can perform the image processing according to this disclosure on the radiographic image acquired by the acquisition unit 21. In this embodiment, noise reduction processing will be described as an example of image processing performed by the image processing unit 22.

[0035] As shown in Figure 2(b), the noise reduction processing unit 26 includes a learning processing unit 261 and an inference processing unit 262. In addition to the configuration of the inference processing unit 262 and the trained model selection unit 263, the learning processing unit 261 also includes a training data generation unit 264 and a parameter update unit 265. Furthermore, the noise reduction processing unit 26 has a pre-processing unit 266 that converts the image input to the noise reduction processing unit 26 into a form suitable for processing by the learning processing unit 261, and a post-processing unit 267 that applies appropriate processing to the output result of the learning processing unit 261. The noise reduction processing unit 26 also has an input adjustment unit 268 that selects the image to be input to the noise reduction processing unit 26 and arranges it in an appropriate form. With this configuration, the noise reduction processing unit 26 can train a machine learning model for noise reduction processing. Furthermore, the noise reduction processing unit 26 can apply noise reduction processing suitable for radiographic images using the trained machine learning model. Note that the noise reduction processing unit 26 may also perform noise reduction processing using trained parameters learned by other learning devices. In other words, the noise reduction processing unit 26 does not necessarily have to be configured to perform both machine learning model training and noise reduction processing (inference processing using trained parameters).

[0036] Furthermore, the diagnostic image processing unit 27 can perform diagnostic image processing on the image that has undergone noise reduction by the noise reduction processing unit 26 to convert it into an image suitable for diagnosis. Diagnostic image processing includes, for example, tone processing to adjust the gradation of the image, enhancement processing to highlight specific pixels in the image, and grid fringe reduction processing to reduce grid fringes in the image. The diagnostic image processing unit 27 may, for example, perform tone processing, enhancement processing, grid fringe reduction processing, etc., according to a region of interest (ROI) set in the radiographic image. For example, tone processing may be performed to broaden the gradation of the region of interest, and enhancement processing may be performed to enhance the region of interest. Here, the region of interest may be set according to the operator's instructions, or it may be set based on the imaging site, disease name information, findings information, etc.

[0037] Next, the configuration of the learning processing unit 261 will be described. The learning processing unit 261 performs the learning process that is applied when training a machine learning model. The learning processing unit 261 includes an inference processing unit 262, a trained model selection unit 263, a training data generation unit 264, and a parameter update unit 265.

[0038] When performing the learning process, the input data is first adjusted by the input adjustment unit 268 to the learning processing unit 261. Then, images that have undergone appropriate preprocessing by the preprocessing unit 266 are input. After that, the learning data generation unit 264 creates the learning data. Here, an example configuration is shown in which an image with artificial noise added (input data) and an image without added noise (ground truth data) are used as the set of learning data for learning noise reduction processing. The learning data generation unit 264 creates a set of learning data by adding artificial noise, which is created by simulating the features of radiation images, to the input image. Here, the noise added by the learning data generation unit 264 may reflect the amount of noise that may vary due to manufacturing variations of the radiation detector 10, as calculated by the learning data generation unit 264. Details of the learning process will be described later.

[0039] The parameter update unit 265 updates the parameters of the machine learning model held by the inference processing unit 262 based on the calculation results of the inference processing unit 262 on the input data and the ground truth data.

[0040] When a radiation image is input to a trained model trained using the training data described above, the inference processing unit 262 generates an image in which image processing has been applied to the radiation image through inference processing. The trained model selection unit 263 selects a trained model to be used by the inference processing unit 262. Here, the trained models obtained by the series of training processes of the training processing unit 261 may be multiple for each type of radiation detector 10, for example, multiple for each type of phosphor 11, or multiple for each type of imaging sensor 12, or multiple for each type of radiation detector 10, such as binning, sensitivity, image size, frame rate, and imaging procedure. The trained model selection unit 263 selects at least one trained model from among the multiple trained models to be used by the inference processing unit 262. Details of the inference process will be described later.

[0041] Here, a portion of the learning processing unit 261 does not need to be included in the control unit 20. For example, the components other than the inference processing unit 262 and the trained model selection unit 263 may be configured on hardware other than the control unit 20 (such as a server). This hardware creates a trained model by performing training in advance using appropriate training data. In this case, the control unit 20 may access this other hardware via the inference processing unit 262 to obtain the trained model and perform only processing using that trained model. Alternatively, a pre-created trained model may be provided in the noise reduction processing unit 26, and the control unit 20 may perform only processing using that trained model.

[0042] Alternatively, the learning processing unit 261 may be included in the control unit 20, allowing for additional learning using the learning data acquired after installation (sale) at the customer's site.

[0043] The display control unit 23 can control the display on the display unit 50. For example, it can display radiation images and patient information before and after image processing by the image processing unit 22 on the display unit 50. The drive control unit 24 can control the driving of the radiation detector 10 and the radiation generator 30, etc. Therefore, the control unit 20 can control the acquisition of radiation images by controlling the driving of the radiation detector 10 and the radiation generator 30 with the drive control unit 24.

[0044] The memory unit 25 can store programs for implementing various application software, including the operating system (OS), device drivers for peripheral devices, and programs for performing the processing described later. The memory unit 25 can also store information acquired by the acquisition unit 21 and radiation images processed by the image processing unit 22. For example, the memory unit 25 can store radiation images acquired by the acquisition unit 21, or radiation images that have undergone noise reduction processing, as described later.

[0045] The control unit 20 can be configured using a general-purpose computer including a processor and memory, but it may also be configured as a dedicated computer for the radiography system 1. Here, the control unit 20 functions as an example of an image processing device according to this embodiment, but the image processing device according to this embodiment may be a separate (external) computer that is communicatively connected to the control unit 20. Furthermore, the control unit 20 and the image processing device may be, for example, personal computers, and desktop PCs, notebook PCs, or tablet PCs (portable information terminals) may be used. The processor may be a CPU (Central Processing Unit). The processor may also be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or an FPGA (Field-Programmable Gate Array).

[0046] Each function of the control unit 20 may be realized by a processor such as a CPU or MPU executing software modules stored in the storage unit 25. The processor may be, for example, a GPU or FPGA. Furthermore, each function may be configured by a circuit that performs a specific function, such as an ASIC. For example, the image processing unit 22 may be realized by dedicated hardware such as an ASIC, and the display control unit 23 may be realized using a dedicated processor such as a GPU that is different from the CPU. The storage unit 25 may be configured by any storage medium such as an optical disk such as a hard disk or memory.

[0047] (Machine learning model configuration) Next, with reference to Figures 3(a) to 3(c), an example of a machine learning model that constitutes the trained model according to this embodiment will be described. An example of a machine learning model used by the inference processing unit 262 according to this embodiment is a multi-layer neural network.

[0048] Figure 3(a) shows a schematic example of the neural network model according to this embodiment. The neural network model configuration 33 shown in Figure 3(a) is designed to output noise-reduced inference data 32 in response to input data 31, according to a pre-learned trend. The output noise-reduced inference data 32 is based on the learning content in the machine learning process. The neural network according to this embodiment learns features for distinguishing between signals and noise contained in the input radiation image. In the example shown in Figure 3(a), the input data 31 includes the current frame and one or more frames prior to the current frame. Alternatively, the input data 31 includes the current frame and one or more frames in the future. Alternatively, the input data 31 includes any set of frames including the current frame, one or more frames prior to the current frame, and one frame in the future. The noise-reduced inference data 32 is the current frame with reduced noise. It is also possible to configure a trained model with only the current frame (1 image) as input data 31, so that the number of input frames is 1.

[0049] Furthermore, at least a portion of the multi-layer neural network may be a convolutional neural network (CNN), for example. Additionally, at least a portion of the multi-layer neural network may utilize techniques related to autoencoders or vision transformers (ViT).

[0050] This section describes an example of using a Convolutional Neural Network (CNN) as a machine learning model for noise reduction processing of radiographic images. Figure 3B shows an example of a schematic configuration 33 of the CNN that constitutes the neural network model according to this embodiment. In the example of the trained model according to this embodiment, when input data 31, which is a radiographic image, is input, inference data 32 can be output as a radiographic image with reduced noise.

[0051] The CNN shown in Figure 3(b) is composed of multiple layers responsible for processing the input data set and producing an output. The types of layers included in the CNN configuration 33 are convolutional layers, downsampling layers, upsampling layers, and merge layers. Here, the CNN configuration 33 further includes an additive layer 34, and it is preferable to configure a shortcut that adds the input data before output. This allows the CNN to adopt a configuration that learns the difference between the input data and the output data, and can suitably handle systems that target noise.

[0052] A convolutional layer is a layer that performs convolution on an input set of values ​​according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The dimensionality of the filter kernel size may also be changed depending on the dimensionality of the input image.

[0053] A downsampling layer is a layer that performs a process to reduce the number of output values ​​to less than the number of input values ​​by decimating or combining input values. Specifically, one example of such a process is Max Pooling.

[0054] An upsampling layer is a layer that performs a process to increase the number of output values ​​to more than the number of input values ​​by duplicating the input values ​​or adding interpolated values ​​from the input values. Specifically, one example of such a process is upsampling by deconvolution.

[0055] A synthesis layer is a layer that takes a set of values, such as the output values ​​of a certain layer or the pixel values ​​that make up an image, as input from multiple sources and performs processing to combine them by concatenating or adding them together.

[0056] It should be noted that different parameter settings for the layers and nodes that make up the neural network may result in differences in the degree to which the trained trends from the training data can be reproduced during inference. In other words, the appropriate parameters often differ depending on the implementation method, so they can be changed as needed.

[0057] In addition to changing the parameters as described above, the CNN may also achieve better characteristics by changing its configuration. These better characteristics include, for example, outputting radiation images with better noise reduction, shorter processing times, and shorter training times for machine learning models.

[0058] The CNN configuration 33 used in this embodiment is a U-net type machine learning model having the functionality of an encoder consisting of multiple layers including multiple downsampling layers, and the functionality of a decoder consisting of multiple layers including multiple upsampling layers. In a U-net type machine learning model, for example, skip connections can be used. That is, positional information (spatial information) that has been obscured in the multiple layers configured as an encoder can be used in layers of the same dimension (layers corresponding to the dimensions of the encoder) in the multiple layers configured as a decoder.

[0059] Although not shown in the diagram, one example of modifying the CNN configuration is to incorporate layers with activation functions (e.g., ReLu: Rectifier Linear Unit) before and after the convolutional layers.

[0060] Through these steps in the CNN, noise features can be extracted from the input radiation images.

[0061] Here, the learning processing unit 261 includes a parameter update unit 265. As shown in Figure 3(c), the parameter update unit 265 calculates a loss function from the inference data 32 obtained by applying the neural network model of the inference processing unit 262 to the input data 31 in the learning data, and from the ground truth data 35 in the learning data. Subsequently, the parameter update unit 265 updates the parameters of the neural network model based on the calculated loss function. Here, the loss function represents the error between the inference data 32 and the ground truth data 35.

[0062] The parameter update unit 265 can update the filter coefficients of the convolutional layer, for example, using backpropagation, so as to reduce the error between the inference data 32 represented by the loss function and the ground truth data 35. Backpropagation is a method for adjusting the parameters between each node of the neural network so as to reduce the above error. In addition, a method of randomly deactivating the units (each neuron or each node) that make up the CNN (dropout) may be used for training.

[0063] Furthermore, the pre-trained model used by the inference processing unit 262 may be generated using transfer learning. In this case, for example, a pre-trained model used for noise reduction processing may be generated by performing transfer learning on a machine learning model trained on radiographic images of objects O of different types. By performing such transfer learning, it is possible to efficiently generate a pre-trained model even for objects O of which it is difficult to obtain a large amount of training data. Here, objects O of different types may be, for example, animals, plants, or objects used in non-destructive testing.

[0064] Here, the GPU can perform calculations efficiently by processing more data in parallel. Therefore, when performing training multiple times using a machine learning model that utilizes a CNN as described above, it is effective to perform the processing on the GPU. Accordingly, the learning processing unit 261 in this embodiment uses a GPU in addition to the CPU. Specifically, when executing a learning program that includes a machine learning model, the CPU and GPU work together to perform calculations to perform learning. Note that the learning process may be performed by the CPU or the GPU alone. Furthermore, each process of the inference processing unit 262 may also be implemented using the GPU in the same way as the learning processing unit 261.

[0065] The above describes the configuration of the machine learning model, but the machine learning model is not limited to the CNN-based model shown above. The machine learning model used in this embodiment can be any machine learning-like model that is capable of extracting (representing) the features of training data such as images through learning.

[0066] Here, the learning processing unit 261 according to this embodiment can use any set of training data for learning noise reduction processing. For example, the learning processing unit 261 can use training data in which images with artificial noise added are input data and images without added noise are ground truth data. In addition, for example, learning may be performed using images before averaging as input data and images after averaging as ground truth data, or images before statistical processing such as MAP (maximum posterior probability) estimation as input data and images after statistical processing as ground truth data. Furthermore, although examples of supervised learning have been shown so far, the learning method is not limited to this, and any unsupervised learning or semi-supervised learning method may be used.

[0067] (Operation of the noise reduction processing unit) Next, Figure 4 will be used to explain the detailed operation of the noise reduction processing unit 26 during video recording. In video recording, frames near the current frame often capture structures similar to those of the current frame. Therefore, when performing noise reduction on the target pixel of the current frame, not only spatial information but also temporal information is used. Spatial information refers to information indicating similar structures around the target pixel in the current frame. Temporal information refers to information indicating similar structures in frames near the current frame (past or future frames).

[0068] The inference processing unit 262 can process while utilizing more temporal information by inputting multiple frames to the CNN41. In this case, in order to perform real-time processing, the inference processing unit 262 adopts a configuration in which the current frame and a predetermined number of past frames are input. Alternatively, it adopts a configuration in which the current frame and a predetermined number of future frames are input. Let N be the number of frames input to the CNN41 (N is an integer of 2 or more). The number of frames input to the CNN41 will vary depending on the frame rate during shooting and the required noise reduction performance. Below, as a preferred example, we will explain the case where N=10 and the current frame and past frames are input.

[0069] Figure 4(a) is a schematic diagram of the neural network configuration for N=10. In the example shown in Figure 4(a), the current frame number is denoted as t. In Figure 4(a), the current frame t and past frames numbered t-1 to t-9 are sequentially input to a trained CNN41. CNN41 is an example of a machine learning model used by the inference processing unit 262. Using such a CNN41, the inference processing unit 262 can obtain a noise-reduced image F(t) by utilizing the spatial information of the current frame t and the temporal information of the nine past frames.

[0070] Figure 4(b) is a schematic diagram of an alternative neural network configuration for N=10. In the example shown in Figure 4(b), the current frame t and past frames numbered t-1 to t-6 are input to a trained CNN41. Since N=10, the number of past frames is reduced by 3 compared to Figure 4(a), and 3 current frames t are added as input. This configuration allows the use of the same CNN41 as in Figure 4(a), but reduces the amount of past frames used compared to the case in Figure 4(a). Note that the current frame t is an example of the second frame image. Also, the past frames t-1 to t-6 are an example of the third frame image.

[0071] When this is generalized, it becomes as shown in FIG. 4(c). In a configuration where the current frame t and past frames t-1 to t-(N-1) are input, when reducing the number of past frames to be used by A frames, current frame t×(A+1) and past frames up to t-(N-A-1) can be used as inputs. A indicates the number of frames, satisfying 0≦A<N. Further, when the number of current frames t is M, M=A+1, and the number of past frames to be used is N-M.

[0072] Note that the case where N=10, A=0, and M=1 corresponds to FIG. 4(a), and the case where N=10, A=3, and M=4 corresponds to FIG. 4(b).

[0073] With this configuration, the balance between spatial information and temporal information used in noise reduction processing can be adjusted.

[0074] FIG. 5 is a graph showing a change in an afterimage amount 51 when the number of frames A is changed, and a change in an SN ratio 52 when the number of frames A is changed (when N=10). As shown in FIG. 5, as the number of frames A is increased (temporal information is decreased), the afterimage amount 51 is improved (reduced). On the other hand, it can be seen that as the number of frames A is increased (temporal information is decreased), the SN ratio 52 is deteriorated (decreased). When A=0, the maximum amount of temporal information can be used, so the afterimage amount 51 is the largest and the SN ratio 52 is the highest. When A=9, no temporal information is used and only spatial information is used, so the afterimage amount 51 is the lowest and the SN ratio 52 is the lowest.

[0075] The afterimage amount 51 decreases as A increases. Further, the SN ratio 52 decreases as A increases. There is a trade-off relationship between improving the afterimage amount 51 and improving the SN ratio 52. An operator can find the optimal number of frames A according to the movement of a subject by changing the number of frames A.

[0076] With this configuration, both a noise reduction effect and an afterimage reduction effect can be achieved using machine learning-based technology. Furthermore, it becomes possible for an operator to intuitively adjust the trade-off between the noise reduction effect (improvement effect of the SN ratio) and the afterimage reduction effect.

[0077] (Another example of a noise reduction processing unit configuration) So far, we have explained the case where N=10 and only the current frame and past frames are used as input. However, the frames used as input are not limited to this example. For example, as illustrated in Figure 4(d), it is also possible to use future frames in addition to the current frame and past frames. Note that past frames are sometimes referred to as past frame images. Similarly, future frames are sometimes referred to as future frame images.

[0078] Figure 4(d) shows an example where the current frame t is 1 frame, past frames are B frames, and future frames are C frames. The number of frames N input to CNN41 is N = B + C + 1. Here, as shown in Figure 4(c), let's assume that we reduce the number of frames by A in total, including past and future frames. For example, in the case of a configuration where we reduce the number of frames uniformly from both the current and future frames (i.e., reduce the number of frames by A / 2 from the current frame and A / 2 from the future frame), we will input past frames up to t - B + (A / 2). We will also input future frames up to t + C - A / 2. The current frame t is an example of the second frame image. The past frame represented by t - B + (A / 2) is an example of the third frame image. The future frame represented by t + CA / 2 is an example of the first frame image.

[0079] The number of past frames (B) and future frames (C) used can be freely changed. Furthermore, when reducing the total number of past and future frames by A, methods other than uniformly reducing the current and future frames (50% reduction each) are also possible. That is, the ratio can be freely changed; for example, reducing 40% from the current frame and 60% from the future frame.

[0080] (Operation of the inference processing unit) Next, Figure 6 will be used to describe the detailed operation of the noise reduction processing unit 26 and the inference processing unit 262 during video recording. Figure 6 shows an example of the flow of the noise reduction processing unit 26 and the inference processing unit 262.

[0081] First, in step S601, the input adjustment unit 268 acquires a parameter that specifies the trade-off between the amount of afterimage and the signal-to-noise ratio. The parameter may be made adjustable by the user from the display unit 50. The shape of the parameter is arbitrary, but it can be converted to the aforementioned number of frames A as appropriate.

[0082] In step S602, the input adjustment unit 268 acquires the image of the current frame and images of past or future frames specified by A via the acquisition unit 21, and arranges them in an appropriate order. For example, it is desirable for the frames to be arranged in order from past to future, as shown in Figure 4(d).

[0083] In step S603, the preprocessing unit 266 performs preprocessing on the image acquired in step S602 to perform appropriate inference processing, thereby obtaining a preprocessed image. The method of preprocessing is not particularly limited. For example, in noise reduction processing, preprocessing may include square root transformation, logarithmic transformation, Anscombe transformation, etc. These transformations make quantum noise following a Poisson distribution approximately constant regardless of the intensity of the irradiated radiation, so that the noise contained in the input image can be treated as additive noise. Furthermore, as preprocessing, centering can be performed to set the mean of the data to 0 in order to stabilize processing by the neural network. Alternatively, standardization can be performed to set the standard deviation of the data to 1. Alternatively, normalization can be performed to normalize the data to a range of 0 to 1. Alternatively, both centering to set the mean of the data to 0 and standardization to set the standard deviation of the data to 1 can be performed. If the size of the image to be handled is large and batch processing by the neural network is difficult, it may be divided into multiple ROIs of arbitrary size. To improve the quality of the calculation results at the image boundaries, it is also possible to perform padding of an appropriate size.

[0084] The results of the above preprocessing can be temporarily stored in memory as needed for use in subsequent inference processing in the frames. It is desirable that the preprocessing performed in preprocessing 266 be the same during both inference and training.

[0085] In step S604, the inference processing unit 262 performs inference processing on the preprocessed image obtained in step S603 using the trained model. This makes it possible to obtain an image to which noise reduction processing has been applied.

[0086] In step S605, the post-processing unit 267 performs post-processing on the results of the inference processing obtained in step S603. The post-processing involves performing processes that reverse the operations performed in the pre-processing in step S603, such as inverse transformations like various normalizations and levelings, removal of padding, and merging of multiple divided ROIs.

[0087] In step S606, the noise reduction processing unit 26 determines whether or not to terminate image acquisition. The noise reduction processing unit 26 may determine the termination of image acquisition based, for example, on set shooting conditions or instructions from the operator. If image acquisition is to continue, the process moves to step S607. In step S607, the noise reduction processing unit 26 adds 1 to the frame number t and then moves the process to step S601, and the noise reduction processing unit 26 repeats the process from steps S601 to S606.

[0088] (Operation of the learning processing unit) Furthermore, CNN41 uses N frames as input data for training by the learning processing unit 261, just as it does during inference. In addition, CNN41 learns patterns in which the number of frames A is randomly varied as input data. By performing training as described above, the characteristics shown in Figure 5 can be obtained (the afterimage amount 51 decreases as A increases, and the signal-to-noise ratio 52 decreases as A increases).

[0089] Figure 7 will be used to explain the detailed operation of the noise reduction processing unit 26 and the learning processing unit 261 in video recording. Figure 7 shows an example of the flow of the noise reduction processing unit 26 and the learning processing unit 261. Hereafter, the learning processing unit 261 will be assumed to perform supervised learning, and the example will be explained using a set of training data for learning the noise reduction process, in which images with artificial noise added are used as input data and images without added noise are used as ground truth data. In this case, the CNN 41 to be trained will be assumed to be a system that takes input consisting of a total of N current frames and past frames, as shown in Figure 4(c), and outputs a current frame with noise reduced.

[0090] In step S701, the learning data generation unit 264 randomly selects image data from the storage unit 25 that stores a plurality of pieces of image data. As data for learning noise reduction processing for moving images, it is preferable to use a plurality of moving images including a plurality of frames, for example. Further, it is desirable that the moving image has as high an SN ratio as possible. The moving image may be, for example, one obtained by performing separate noise reduction processing on the moving image in advance to improve the SN ratio. The learning data generation unit 264 randomly selects a moving image, and further acquires an image with a random frame number=Tr from the selected moving image. Note that the image with frame number=Tr is an example of a fourth frame image.

[0091] In step S702, the input adjustment unit 268 randomly selects the number of frames A, where 0≦A<N. By performing learning including patterns in which the number of frames A is randomly changed, it becomes possible to adjust the trade-off relationship between the SN ratio improvement effect and afterimages during inference processing by changing the number of frames A.

[0092] In step S703, the input adjustment unit 268 reads past frames up to Tr-(N-A-1) in accordance with the number of frames A selected in step S702, the number of input frames N of the CNN 41, and the random frame number=Tr selected in step S701. The read past frames are arranged in the order as shown in Fig. 4(c).

[0093] In step S704, the learning data generation unit 264 adds artificial noise, created by simulating the characteristics of radiation images, to the images from frame number Tr-(NA-1) to frame number Tr. The artificial noise should ideally simulate actual radiation images, taking into account factors such as the sensitivity of the radiation detector 10, the noise characteristics of the readout circuit, and the MTF (Modulation Transfer Function) of the phosphor 11. Through the steps described above, learning data can be constructed using the images from Tr-(NA-1) to frame number Tr with added artificial noise as input data, and the image of frame number Tr without added artificial noise as ground truth data. Note that the image of frame number Tr with added artificial noise is an example of the fifth frame image.

[0094] In step S705, the preprocessing unit 266 performs preprocessing on the training data acquired in step S702 to perform appropriate inference processing, thereby obtaining preprocessed training data. Details of the preprocessing are as described above.

[0095] In step S706, the inference processing unit 262 inputs the pre-processed input data into the CNN41, applies the parameters of the CNN41 during training, and outputs the inference result.

[0096] In step S707, the parameter update unit 265 updates the parameters of the CNN41 to minimize an appropriate loss function based on the inference results obtained in step S706 and the ground truth data.

[0097] In step S708, the learning processing unit 261 determines whether or not to terminate the learning process. The determination of whether to terminate the learning process can be made based on any criteria, such as the number of iterations, the value of the loss function, or whether or not overfitting has occurred. If the learning process is complete, the updating of the parameters of CNN41 is stopped, and the trained CNN can be used for the various inference processes described above. If the learning process is to continue, the process moves to step S701, and the learning processing unit 261 repeats the processes from steps S701 to S708.

[0098] Furthermore, based on the trade-off relationship between the improvement in signal-to-noise ratio and afterimages, as shown in Figure 5, the number of frames A may be limited to any number greater than or equal to 0. The smaller the number of A, the fewer patterns can be trained, thereby increasing the training efficiency of CNN41.

[0099] With the configuration described above, it is possible to achieve both noise reduction and afterimage reduction using machine learning-based technology. Furthermore, the operator can intuitively adjust the trade-off between the noise reduction effect (improvement of the signal-to-noise ratio) and the afterimage reduction effect.

[0100] (Variation 1) The machine learning model used by the inference processing unit 262 can be a combination of any layer configuration, such as a Variational Auto-Encoder (VAE), a Fully Convolutional Network (FCN), SegNet, or DenseNet, as part of the CNN configuration. Alternatively, the machine learning model may also use a configuration such as a Vision Transformer (ViT).

[0101] (Modification 2) Furthermore, the training data for various pre-trained models is not limited to data obtained using the radiation detector itself that actually performs the imaging, but may also be data obtained using the same type of radiation detector or data obtained using the same kind of radiation detector, depending on the desired configuration. It is thought that the pre-trained models according to the above embodiments and modifications extract, for example, the magnitude of the brightness values ​​of the radiation image as part of the features and use them in the estimation process related to the generation of radiation images that have undergone various image processing. Other features include the order, slope, position, distribution, and continuity of the bright and dark areas.

[0102] Furthermore, the trained models according to the embodiments and modifications described above can be provided in the control unit 20. The trained models may consist of, for example, software modules executed by a processor such as a CPU, MPU, GPU, or FPGA, or they may consist of circuits that perform specific functions such as ASICs. These trained models may also be provided in a separate server device connected to the control unit 20. In this case, the control unit 20 can use the trained models by connecting to the server equipped with the trained models via any network such as the Internet. Here, the server equipped with the trained models may be, for example, a cloud server, a fog server, an edge server, etc.

[0103] (Variation 3) Furthermore, in the embodiments and modifications described above, the radiation detector 10 is an indirect conversion type detector that first converts radiation into visible light using a phosphor 11 and then converts the visible light into an electrical signal using a photoelectric conversion element. In contrast, the radiation detector 10 may also be a direct conversion type detector that directly converts incident radiation into an electrical signal.

[0104] (Other embodiments) Furthermore, the disclosed technology can also be realized by performing the following process: that is, the disclosed technology can also be realized by supplying software (programs) that implement one or more functions of the various embodiments described above to a system or device via a network or storage medium, and the computer (or CPU, MPU, etc.) of that system or device reads and executes the program. The computer may have one or more processors or circuits and may include a network of separate computers or separate processors or circuits for reading and executing computer executable instructions. In this case, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). The processor or circuit may also include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0105] (Composition 1) An image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, An image processing apparatus comprising an inference processing unit that acquires a frame image with reduced noise from the second frame image using a plurality of the second frame images and at least one of the first frame image and the third frame image.

[0106] (Configuration 2) A plurality of past frame images acquired before the second frame image, including the third frame image, and a plurality of future frame images acquired after the second frame image, including the first frame image, The image processing apparatus according to Configuration 1, wherein the inference processing unit uses a plurality of second frame images and at least one of the plurality of past frame images and the plurality of future frame images to acquire a frame image with reduced noise from the second frame image.

[0107] (Composition 3) The inference processing unit has a trained model that, upon receiving N frames (N≧2) as input, acquires a single frame image with reduced noise. The image processing apparatus according to configuration 2, wherein when the number of second frame images input to the trained model is M (1 ≤ M ≤ N), the sum of the number of past frame images and the number of future frame images is NM.

[0108] (Composition 4) The image processing apparatus according to configuration 3, wherein the trained model is trained using a fourth frame image as ground truth data and a plurality of frame images as input data, wherein artificial noise is added to each frame of a moving image including the fourth frame image.

[0109] (Composition 5) The aforementioned trained model is trained using multiple fifth frame images, which are obtained by adding artificial noise to the fourth frame image included in the input data. The image processing apparatus according to configuration 4, wherein the number of fifth frame images used for training is randomly changed each time training is performed.

[0110] (Composition 6) The number of the second frame images is changeable by operation by an operator, as described in any of configurations 1 to 5 of the image processing apparatus.

[0111] (Composition 7) When the operator controls the number of the second frame images to increase, The inference processing unit is configured to acquire a frame image in which the afterimage is reduced compared to the frame image before augmentation, according to the image processing apparatus according to configuration 6.

[0112] (Composition 8) When the operator controls the number of the second frame images to decrease, The inference processing unit is an image processing apparatus according to any one of configurations 6 to 7, which acquires a frame image in which noise has been reduced compared to the frame image before reduction.

[0113] (Composition 9) A radiation detector that detects radiation, A radiography system comprising an image processing device according to any one of configurations 1 to 8, which is connected to the aforementioned radiation detector in a manner that enables communication.

[0114] (Method 1) An image processing method for an image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, An image processing method for an image processing apparatus, comprising the step of obtaining a frame image with reduced noise than the second frame image using a plurality of the second frame images and at least one of the first frame image and the third frame image.

[0115] (Program 1) A program that causes a computer to execute the image processing method described in Method 1. [Explanation of Symbols]

[0116] 10. Radiation detectors 20 Control Unit (Image Processing Unit) 22 Image Processing Unit

Claims

1. An image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, An image processing apparatus comprising a processing unit that acquires a frame image with reduced noise from the second frame image using a plurality of the second frame images and at least one of the first frame image and the third frame image.

2. A plurality of past frame images acquired before the second frame image, including the third frame image, and a plurality of future frame images acquired after the second frame image, including the first frame image, The image processing apparatus according to claim 1, wherein the processing unit obtains a frame image with reduced noise from the second frame image using a plurality of second frame images and at least one of the plurality of past frame images and the plurality of future frame images.

3. The processing unit has a trained model that, upon receiving N frames (N≧3) as input, acquires a single frame image with reduced noise. The image processing apparatus according to claim 2, wherein when the number of second frame images input to the trained model is M (2 ≤ M ≤ N - 1), the sum of the number of past frame images and the number of future frame images is N - M.

4. The image processing apparatus according to claim 3, wherein the trained model is trained using a fourth frame image as ground truth data and a plurality of frame images as input data, wherein artificial noise is added to each frame of a moving image including the fourth frame image.

5. The aforementioned trained model is trained using multiple fifth frame images, which are obtained by adding artificial noise to the fourth frame image included in the input data. The image processing apparatus according to claim 4, wherein the number of fifth frame images used for training is randomly changed each time training is performed.

6. The image processing apparatus according to claim 1, wherein the number of the second frame images can be changed by operation by an operator.

7. When the operator controls the number of the second frame images to increase, The image processing apparatus according to claim 6, wherein the processing unit acquires a frame image in which the afterimage is reduced compared to the frame image before augmentation.

8. When the operator controls the number of the second frame images to decrease, The image processing apparatus according to claim 6, wherein the processing unit acquires a frame image in which noise has been reduced compared to the frame image before reduction.

9. The processing unit has a trained model, The image processing apparatus according to claim 1, wherein the processing unit inputs a plurality of second frame images and at least one of the first frame image and the third frame image to the trained model to obtain a frame image with reduced noise from the second frame image.

10. A radiation detector that detects radiation, A radiation imaging system comprising an image processing device according to any one of claims 1 to 9, which is connected to the radiation detector in a manner that enables communication.

11. An image processing method for an image processing apparatus that performs image processing on a moving image including a first frame image, a second frame image acquired before the first frame image, and a third frame image acquired before the second frame image, An image processing method for an image processing apparatus, comprising the step of obtaining a frame image with reduced noise from the second frame image using a plurality of the second frame images and at least one of the first frame image and the third frame image.

12. A program that causes a computer to execute the image processing method described in claim 11.

Images