Medical data processing device, medical image diagnostic device, and method for generating trained models

Abstract

To improve the accuracy of medical data recovery. [Solution] The medical data processing device according to the embodiment has a storage unit and a processing unit. The storage unit takes as input first medical data relating to data captured by a first medical imaging device and second medical data relating to the same imaging target as the first medical data but with different imaging parameters relating to data captured by a second medical imaging device that is the same as or different from the first medical imaging device, and outputs third medical data in which the missing portion of the first medical data is restored. According to a trained model, the storage unit generates third medical data in which the missing portion of the first medical data is restored from the first medical data to be processed and the second medical data relating to different imaging parameters relating to the first medical data to be processed.

Description

Technical Field

[0001] Embodiments of the present invention relate to a medical data processing device, a medical image diagnostic device, and a learned model generation method.

Background Art

[0002] In machine learning using medical data such as medical image data and its raw data, in order to restore original data from medical data with some missing parts, there is a method of applying a deep neural network (DNN: Deep Neural Network) learned from a large amount of learning data. For example, in magnetic resonance imaging (MRI), there is a method of applying a DNN to undersampled k-space data to generate k-space data with the missing part restored, and obtaining a restored image based on the restored k-space data.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the invention is to improve the restoration accuracy of medical data.

Means for Solving the Problems

[0005] The medical data processing device according to the embodiment includes a processing unit. It takes as input first medical data relating to data captured by the first diagnostic imaging device and second medical data relating to the same imaging target as the first medical data but with different imaging parameters relating to data captured by a second diagnostic imaging device that is the same as or different from the first medical imaging device, and outputs third medical data in which the missing portion of the first medical data has been restored. According to a trained model, the processing unit generates third medical data in which the missing portion of the first medical data has been restored from the first medical data to be processed and the second medical data relating to different imaging parameters relating to the first medical data to be processed. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a diagram showing the configuration and processing overview of the medical data processing system to which the medical data processing device according to this embodiment belongs. [Figure 2] Figure 2 shows the structure of the multilayer network according to this embodiment. [Figure 3] Figure 3 shows the configuration of the medical imaging diagnostic device according to this embodiment. [Figure 4] Figure 4 shows an example of a combination of inputs and outputs of a trained model according to this embodiment. [Figure 5] Figure 5 shows another example of input and output combinations of the trained model according to this embodiment. [Figure 6] Figure 6 shows the detailed structure of the trained model according to this embodiment. [Figure 7] Figure 7 shows a typical flow of DNN reconstruction processing by the medical data processing device shown in Figure 3. [Figure 8] Figure 8 schematically shows the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 9]Figure 9 is another diagram schematically showing the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 10] Figure 10 is another diagram schematically showing the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 11] Figure 11 is another diagram schematically showing the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 12] Figure 12 is another diagram schematically showing the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 13] Figure 13 is another diagram that schematically shows the relationship between the input and output of the trained deep neural network in the forward propagation process according to this embodiment. [Figure 14] Figure 14 shows the configuration of another medical data processing device according to this embodiment. [Figure 15] Figure 15 shows the configuration of the model learning device shown in Figure 1. [Figure 16] Figure 16 shows a typical flow of the model learning process executed by the processing circuit of the model learning device shown in Figure 15, according to the model learning program. [Figure 17] Figure 17 shows the configuration of a magnetic resonance imaging apparatus related to Application Example 1. [Figure 18] Figure 18 is a schematic diagram illustrating the processing of the medical data processing device shown in Figure 17. [Figure 19] Figure 19 shows the configuration of an X-ray computed tomography (X-ray) scanner related to Application Example 2. [Figure 20] Figure 20 is a schematic diagram illustrating the processing of the medical data processing device shown in Figure 19. [Figure 21] Figure 21 shows the configuration of the PET / CT apparatus related to Application Example 3. [Figure 22] Figure 22 shows a typical flow of processing for the medical data processing device shown in Figure 21. [Figure 23] Figure 23 is a schematic diagram illustrating the processing of the medical data processing device shown in Figure 21. [Figure 24] Figure 24 shows the configuration of an ultrasound diagnostic device related to Application Example 4. [Figure 25] Figure 25 is a schematic diagram illustrating the processing of the medical data processing device shown in Figure 24. [Figure 26] Figure 26 is a schematic diagram showing the processing of the medical data processing device according to Example 1. [Figure 27] Figure 27 shows an overview of the DNN according to Example 1. [Figure 28] Figure 28 shows the results using simulated radial data (21 spokes per frame), and is a reconstructed image using NUFFT+PI. [Figure 29] Figure 29 shows the results using simulated radial data (21 spokes per frame), and represents the reconstructed image using the standard reconstruction method (M=N=1). [Figure 30] Figure 30 shows the results using simulated radial data (21 spokes per frame), and is a diagram showing the reconstructed image obtained by the reconstruction method (M=5 and N=3) according to the embodiment. [Figure 31] Figure 31 shows the results using simulated radial data (21 spokes per frame), and represents the true reconstructed image. [Figure 32] Figure 32 shows the results of the actual stack of stars data (21 spokes per frame), and is a reconstructed image using NUFFT+PI. [Figure 33] Figure 33 shows the results of the actual stack of stars data (21 spokes per frame), and is a reconstructed image using the standard reconstruction method (M=N=1). [Figure 34]Figure 34 shows the results of the actual stack of stars data (21 spokes per frame), and is a diagram showing the reconstructed image obtained by the reconstruction method (M=5 and N=3) according to the embodiment. [Figure 35] Figure 35 schematically shows the dense input performed in Example 2. [Figure 36] Figure 36 shows an example of DNN reconstruction using dense input according to Example 2. [Figure 37] Figure 37 shows an example of DNN reconstruction using dense input according to Example 3. [Modes for carrying out the invention]

[0007] The medical data processing device, magnetic resonance imaging device, and trained model generation method according to this embodiment will be described below with reference to the drawings.

[0008] Figure 1 is a diagram showing the configuration and processing overview of the medical data processing system 100 to which the medical data processing device 1 according to this embodiment belongs. As shown in Figure 1, the medical data processing system 100 according to this embodiment includes the medical data processing device 1, the medical imaging device 3, the model learning device 5, and the learning data storage device 7.

[0009] The learning data storage device 7 stores learning data that includes multiple learning samples. For example, the learning data storage device 7 is a computer with a built-in large-capacity storage device. Alternatively, the learning data storage device 7 may be a large-capacity storage device that is connected to the computer via a cable or communication network. As the storage device, an HDD (Hard Disk Drive), SSD (Solid State Drive), integrated circuit storage device, etc., can be used as appropriate.

[0010] The model learning device 5, based on the learning data stored in the learning data storage device 7, causes the machine learning model to perform machine learning according to the model learning program and generates a trained machine learning model (hereinafter referred to as the trained model). The model learning device 5 is a computer such as a workstation having processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The model learning device 5 and the learning data storage device 7 may be connected via a cable or a communication network, or the learning data storage device 7 may be mounted on the model learning device 5. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via a cable or a communication network. The model learning device 5 and the learning data storage device 7 do not have to be connected via a communication network. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via a portable storage medium on which the learning data is stored.

[0011] The machine learning model according to this embodiment is a parameterized composite function composed of multiple functions that take medical data as input and output medical data in which missing parts of the medical data have been restored. A parameterized composite function is defined by a combination of multiple tunable functions and parameters. The machine learning model according to this embodiment may be any parameterized composite function that satisfies the above requirements, but it is assumed to be a multi-layer network model (hereinafter referred to as a multi-layer network).

[0012] The medical imaging device 3 generates the medical data to be processed. Conceptually, the medical data according to this embodiment includes raw data collected by performing medical imaging on a subject using the medical imaging device 3 or another medical imaging device, and medical image data generated by performing restoration processing on said raw data. The medical imaging device 3 may be any modality device as long as it is capable of generating medical data. For example, the medical imaging device 3 according to this embodiment may be a single modality device such as a magnetic resonance imaging device (MRI device), an X-ray computed tomography device (CT device), an X-ray diagnostic device, a PET (Positron Emission Tomography) device, a SPECT (Single Photon Emission CT) device, and an ultrasound diagnostic device, or it may be a composite modality device such as a PET / CT device, a SPECT / CT device, a PET / MRI device, or a SPECT / MRI device.

[0013] The medical data processing device 1 generates output medical data corresponding to the input medical data to be processed, collected by the medical imaging device 3, using a trained model learned by the model learning device 5 according to a model learning program. The medical data processing device 1 and the model learning device 5 may be connected communicatively via a cable or communication network, or the medical data processing device 1 and the model learning device 5 may be implemented on a single computer. In this case, the trained model is supplied from the model learning device 5 to the medical data processing device 1 via a cable or communication network. The medical data processing device 1 and the model learning device 5 do not necessarily have to be connected communicatively. In this case, the trained model is supplied from the model learning device 5 to the medical data processing device 1 via a portable storage medium in which the trained model is stored. The supply of the trained model may be at any point between the manufacture of the medical data processing device 1 and its installation in a medical facility, or during maintenance, or at any other point in time. The supplied trained model is stored in the medical data processing device 1. Furthermore, the medical data processing device 1 may be a computer installed in a medical imaging diagnostic device equipped with a medical imaging device 3, a computer connected to the medical imaging diagnostic device via a cable or network for communication, or a computer independent of the medical imaging diagnostic device.

[0014] A typical configuration of the multilayer network according to this embodiment will be described below. Here, a multilayer network is a network that has a structure in which only adjacent layers are connected, and information propagates in one direction from the input layer to the output layer. The multilayer network according to this embodiment consists of L layers, as shown in Figure 2, consisting of an input layer (l=1), an intermediate layer (l=2,3,···,L-1), and an output layer (l=L). Note that the following is an example, and the configuration of the multilayer network is not limited to the following description.

[0015] Let the number of units in the l-th layer be I, and the input u to the l-th layer be (l) Let (1-1) be the output z from the i-th layer. (l)If we express them as shown in equation (l-2), the relationship between the input to the l-th layer and the output from the l-th layer can be expressed by equation (1-3).

[0016]

number

[0017] Here, the upper right subscript (l) indicates the sheaf number. Also, in equation (1-3), f(u) is the activation function, and various functions can be selected depending on the purpose, such as the logistic sigmoid function (logistic function), hyperbolic tangent function, normalized linear function (ReLU: Rectified Linear Unit), linear map, identity map, maxout function, etc.

[0018] Let J be the number of units in the (l+1)th layer, and W be the weighted matrix between the (l)th layer and the (l+1)th layer. (l+1) Equation (2-1) shows the bias b in the (1+1)th layer. (l+1) If we express them as shown in equation (2-2), then the input u to the (l+1)th layer is (l+1) , output z from the (1+1)th layer (l+1) These can be expressed by equations (2-3) and (2-4), respectively.

[0019]

number

[0020] In the multilayer network according to this embodiment, medical data expressed by equation (3-1) is input to the input layer (l=1). Furthermore, in this input layer, the input data x is directly converted into the output data z. (1) Therefore, the relationship in equation (3-2) holds true.

[0021]

number

[0022] Here, if the medical data input to the input layer is called "input medical data", for the input medical data x, various forms can be selected according to the purpose. Some typical examples are listed below. (1) The input medical data x is regarded as one piece of image data, and each component x p (p = 1, 2, ···, N) is defined as the value (pixel value or voxel value) at each position constituting the one piece of image data. (2) The input medical data x is regarded as M pieces of image data (for example, a plurality of image data with different imaging conditions from each other), and among each component x p from 1 ≤ p ≤ q is the first piece of image data, from q + 1 ≤ p ≤ r is the second piece of image data, from r + 1 ≤ p ≤ s is the third piece of image data, ···, and in the input layer, the range of the input unit is assigned for each piece of image data. (3) The input medical data x is regarded as M pieces of image data, and each component x p is defined as a vector in which the values (pixel values or voxel values) at each position of one piece of image data are arranged vertically. (4) The input medical data x is regarded as raw data (RAW data) such as k-space data or projection data, and the forms of (1) to (3) etc. are adopted. (5) The input medical data x is regarded as image data or raw data subjected to convolution processing, and the forms of (1) to (3) etc. are adopted.

[0023] In the intermediate layer (l = 2, 3, ···, L - 1) following the input layer, by sequentially executing the calculations according to Formula (2-3) and Formula (2-4), the output z (2) , ··· z (L-1) of each layer can be calculated.

[0024] The output z (L) of the output layer (the L-th layer) is expressed as in the following Formula (4-1). The multi-layer network according to the present embodiment is a forward propagation type network in which the image data x input to the input layer propagates while being coupled only between adjacent layers from the input layer side to the output layer side. Such a forward propagation type network can be expressed as a composite function as in Formula (4-2).

[0025]

number

[0026] The composite function defined by equation (4-2) is, from equations (2-3) and (2-4), a weighted matrix W (l+1) Linear relationships between each layer using the method, and activation function f(u) in each layer. (l+1) ) a nonlinear (or linear) relationship, bias b (l+1) It is defined as a combination of these. In particular, the weighted matrix W (l+1) Bias b (l+1) This is called the network parameter p. The composite function defined by equation (4-2) changes its form as a function depending on how the parameter p is chosen. Therefore, the multilayer network according to this embodiment can be defined as a function in which the output layer can output a desirable result y by appropriately choosing the parameter p that constitutes equation (4-2).

[0027] To appropriately select parameter p, training is performed using training data and an error function. Here, training data refers to the input x n The desired output (correct output) for this is d n Then, the training sample (x) can be expressed as shown in equation (5-1). n d n This is the set D(n=1,···,S).

[0028]

number

[0029] Furthermore, the error function is x n Output from a multilayer network with input data d nThis function represents the degree of proximity to a given value. Typical examples of error functions include the squared error function, the maximum likelihood estimation function, and the cross-entropy function. The choice of error function depends on the problem handled by the multilayer network (e.g., regression problems, binary problems, multi-class classification problems, etc.).

[0030] Let the error function be denoted as E(p), and one training sample (x n d n The error function E is calculated using only ). n This is denoted as (p). The current parameter is p. (t) When following gradient descent, the error function E(p) is expressed using the gradient vector (6-1), and when following stochastic gradient descent, the error function E n Equation (6-3) using the gradient vector of (p) gives a new parameter p (t+1) It will be updated.

[0031]

number

[0032] Here, ε is the learning rate that determines the magnitude of the parameter p update.

[0033] By following equation (6-1) or equation (6-3), and repeatedly shifting the current p slightly in the negative gradient direction, we can determine the parameter p that minimizes the error function E(p).

[0034] Note that in order to calculate equation (6-1) or equation (6-3), the gradient vector of E(p) shown in equation (6-2) or E shown in equation (6-4) is required. n We need to calculate the gradient vector of (p). For example, if the error function is a squared error function, we need to differentiate the error function shown in equation (7-1) with respect to the weight coefficients of each layer and the bias of each unit.

[0035]

number

[0036] On the other hand, since the final output y is a composite function represented by equation (4-2), E(p) or E n Calculating the gradient vector of (p) is complex, and the computational complexity is enormous.

[0037] Such problems in gradient calculation can be solved by backpropagation. For example, the weight w connecting the i-th unit of the (l-1)th layer and the j-th unit of the l-th layer. ji (l) The derivative of the error function with respect to can be expressed as shown in equation (8-1) below.

[0038]

number

[0039] Input u to the j-th unit of layer l j (l) That's abnormal n The amount of change given is the output z from the j-unit. j (l) Each input u to each unit k of the (l+1) layer via k (l+1) This occurs solely through the change of . From this, the first term on the right-hand side of equation (8-1) can be expressed as equation (9-1) using the chain rule of differentiation.

[0040]

number

[0041] Here, the left side of equation (9-1) is δ j (l) If we assume this, then using the relationship between equations (10-1) and (10-2), equation (9-1) can be rewritten as equation (10-3).

[0042]

number

[0043] From equation (10-3), the δ on the left side j (l) is, δ k (l+1) It can be seen that it can be calculated from (k=1,2,····). That is, δ related to the kth unit of the (l+1)th layer located on the output layer one level above. k (l+1) Given δ for the lth layer, j (l) Furthermore, δ related to the k-th unit of the (l+1)th layer can be calculated. k (l+1) Regarding this as well, the δ related to the k-th unit of the (l+2)th layer located on the output layer one level above it. k (l+2) Given the necessary parameters, the calculation can be performed. By repeating this process sequentially, we can reach the top layer, the output layer.

[0044] First, the δ related to the k-th unit of the output layer, which is the L-th layer. k (L) If obtained, then by repeatedly performing sequential calculations toward the lower layers (i.e., the input layer) using equation (10-3) (backpropagation), the δ at any layer can be obtained. k (l+1) It is possible to calculate this.

[0045] On the other hand, the second term on the right-hand side of equation (8-1) can be calculated as shown in equation (11-2) by using equation (11-1), which is obtained by expressing equation (2-3) in terms of components for the i-th layer.

[0046]

number

[0047] Therefore, the weight w connecting the i-th unit of the (l-1)th layer and the j-th unit of the l-th layer ji (l) The derivative of the error function with respect to is given by δ(8-1) and (10-3) j (l) Using equation (11-2), it can be expressed as follows: equation (12-1).

[0048]

number

[0049] From equation (12-1), the weight w connecting the i-th unit of the (l-1)th layer and the j-th unit of the l-th layer is obtained. ji (l) The derivative of the error function with respect to is δ with respect to the j-th unit. j (l) And the output from the i-th unit is z i (l-1) It can be seen that it is given by the product of . Note that δ j (l) The calculation for can be obtained by backpropagation using equation (10-3) as described above, and the first value of the backpropagation, i.e., δ related to the Lth layer, which is the output layer, can also be obtained. j (L) This can be calculated as shown in the following equation (13-1).

[0050]

number

[0051] By following the above procedure, a certain training sample (x) is applied to the multilayer network according to this embodiment. n d n Learning can be achieved using ). Note that the sum of errors E = Σ for multiple training samples n E n For the gradient vector related to x, the procedure described above is used for training samples (x n d n This can be obtained by repeatedly performing the calculation in parallel for each step and calculating the sum shown in equation (14-1) below.

[0052]

number

[0053] The details of the medical data processing system 100 using a multilayer network according to this embodiment will be described below. In the following description, the medical data processing device 1 is assumed to be connected to the medical imaging device 3 and to be incorporated together with the medical imaging device 3 into a medical diagnostic imaging device.

[0054] Figure 3 shows the configuration of a medical imaging diagnostic apparatus 9 according to this embodiment. As shown in Figure 3, the medical imaging diagnostic apparatus 9 includes a medical data processing device 1 and a medical imaging device 3. For example, the medical imaging device 3 corresponds to a stand, and the medical data processing device 1 corresponds to a console connected to the stand. The medical data processing device 1 may be provided on the stand of the medical imaging diagnostic apparatus 9, or it may be realized by a component other than the console or stand of the medical imaging diagnostic apparatus 9. Such a component may be, for example, a computer or dedicated computing device other than the console, installed in the machine room when the medical imaging diagnostic apparatus 9 is a magnetic resonance imaging apparatus.

[0055] The medical imaging device 3 performs medical imaging on a subject using an imaging principle corresponding to the modality of the medical imaging device 3, and collects raw data about the subject. The collected raw data is transmitted to the medical data processing device 1. For example, the raw data is k-space data when the medical imaging device 3 is a magnetic resonance imaging device, projection data or sinogram data when it is an X-ray computed tomography device, echo data when it is an ultrasound diagnostic device, coincidence data or sinogram data when it is a PET device, or projection data or sinogram data when it is a SPECT device. Also, when the medical imaging device 3 is an X-ray diagnostic device, the raw data is X-ray image data.

[0056] If the medical imaging device 3 is a stand for a magnetic resonance imaging device, the stand repeatedly applies a gradient magnetic field via a gradient magnetic field coil and an RF pulse via a transmitting coil under the application of a static magnetic field via a static magnetic field magnet. An MR signal is emitted from the subject due to the application of the RF pulse. The emitted MR signal is received via a receiving coil. The received MR signal is subjected to signal processing such as A / D conversion by a receiving circuit. The MR signal after A / D conversion is called k-space data. The k-space data is transmitted as raw data to the medical data processing device 1.

[0057] If the medical imaging device 3 is a stand for an X-ray computed tomography (X-ray) scanner, the stand rotates the X-ray tube and X-ray detector around the subject, irradiating the subject with X-rays from the X-ray tube, and detecting the X-rays that pass through the subject with the X-ray detector. The X-ray detector generates an electrical signal with a pulse height corresponding to the detected X-ray dose. This electrical signal is subjected to signal processing such as A / D conversion by a data acquisition circuit. The electrical signal after A / D conversion is called projection data or sinogram data. The projection data or sinogram data is transmitted as raw data to the medical data processing device 1.

[0058] When the medical imaging device 3 is an ultrasound probe for an ultrasound diagnostic device, the ultrasound probe transmits an ultrasound beam into the subject's body from multiple ultrasound transducers and receives the ultrasound reflected from the subject's body via the ultrasound transducers. The ultrasound transducers generate an electrical signal with a peak value corresponding to the sound pressure of the received ultrasound. This electrical signal is converted using A / D conversion by an A / D converter provided in the ultrasound probe, etc. The electrical signal after A / D conversion is called echo data. The echo data is transmitted as raw data to the medical data processing device 1.

[0059] If the medical imaging device 3 is a PET scanner stand, the stand simultaneously measures a pair of 511 keV gamma rays generated by the annihilation of positrons from radionuclides accumulated in the subject with electrons surrounding those radionuclides using a simultaneous measurement circuit. This generates digital data containing digital values ​​related to the energy values ​​and detection positions of the pair of gamma rays (LOR (Line Of Response)) emitting. This digital data is called coincidence data or sinogram data. The coincidence data or sinogram data is transmitted as raw data to the medical data processing device 1.

[0060] If the medical imaging device 3 is the C-arm of an X-ray diagnostic device, the X-rays are generated from the X-ray tube installed on the C-arm. The X-rays generated from the X-ray tube and transmitted through the subject are received by an X-ray detector such as an FPD (Flat Panel Display) installed on or independently of the C-arm. The X-ray detector generates an electrical signal with a pulse height corresponding to the detected X-ray dose, and this electrical signal is subjected to signal processing such as A / D conversion. The electrical signal after A / D conversion is called X-ray image data. The X-ray image data is transmitted as raw data to the medical data processing device 1.

[0061] As shown in Figure 3, the medical data processing device 1 has a processing circuit 11, memory 13, input interface 15, communication interface 17, and display 19 as hardware resources.

[0062] The processing circuit 11 has a processor such as a CPU or GPU. By starting a program installed in memory 13 or the like, the processor executes the imaging control function 111, the normal restoration function 112, the input selection function 113, the forward propagation function 114, the image processing function 115, and the display control function 116, etc. Note that each of the functions 111 to 116 is not limited to being implemented by a single processing circuit. A processing circuit may be configured by combining multiple independent processors, and each of the functions 111 to 116 may be implemented by each processor executing a program.

[0063] In the imaging control function 111, the processing circuit 11 controls the medical imaging device 3 according to the imaging conditions and performs medical imaging on the subject. The imaging conditions according to this embodiment include the imaging principle of the medical imaging device 3 and various imaging parameters. The imaging principle corresponds to the type of medical imaging device 3, specifically, a magnetic resonance imaging device, an X-ray computed tomography device, a PET device, a SPECT device, and an ultrasound diagnostic device. The imaging parameters include, for example, FOV (Field of View), imaging area, slice position, frame (temporal phase of the medical image), temporal resolution, matrix size, presence or absence of contrast agent, etc. In the case of magnetic resonance imaging, the imaging parameters further include, for example, the type of imaging sequence, parameters such as TR (Time to Repeat), TE (Echo Time), FA (Flip Angle), and the type of k-space filling trajectory. In the case of X-ray computed tomography, imaging parameters further include X-ray conditions (tube current, tube voltage, and X-ray exposure duration, etc.), scan type (non-helical scan, helical scan, synchronous scan, etc.), tilt angle, reconstruction function, number of views per rotation of the rotating frame, rotation speed, and detector spatial resolution. In the case of ultrasound diagnostics, imaging parameters further include focal position, gain, transmission intensity, reception intensity, PRF, beam scanning method (sector scan, convex scan, linear scan, etc.), and scanning mode (B-mode scan, Doppler scan, color Doppler scan, M-mode scan, A-mode scan, etc.).

[0064] In the normal restoration function 112, the processing circuit 11 restores a medical image by applying normal restoration processing to the raw data transmitted from the medical imaging device 3. The normal restoration processing according to this embodiment includes restoration from raw data to raw data, restoration from raw data to image data, and restoration from image data to image data. The restoration process from raw data defined by one coordinate system to two-dimensional image data or three-dimensional image data defined by another coordinate system is also called reconstruction processing or image reconstruction processing. The normal restoration processing according to this embodiment refers to restoration processing other than DNN restoration described later, such as denoising type restoration and data error feedback type restoration. For example, image reconstruction related to the normal restoration processing according to this embodiment can be classified into analytical image reconstruction and iterative approximation image reconstruction. For example, analytical image reconstruction related to MR image reconstruction includes Fourier transform or inverse Fourier transform. Analytical image reconstruction related to CT image reconstruction includes the FBP (filtered back projection) method, the CBP (convolutional back projection) method, or applications thereof. Iterative image reconstruction methods include the EM (expectation maximization) method, the ART (algebraic reconstruction technique) method, or applications thereof.

[0065] In the input selection function 113, the processing circuit 11 selects input medical data for the trained model 90. In this embodiment, the input medical data selected consists of input medical data to be processed (hereinafter referred to as the input data to be processed) and auxiliary input medical data (hereinafter referred to as auxiliary input data). Typically, the input data to be processed is medical data to be restored that has data loss. In this embodiment, data loss is a concept that includes any difference between the actual medical data and the desired medical data concerning the subject. For example, data loss includes data degradation due to noise caused by various reasons, data loss due to a decrease in the number of sampling points of medical data due to decimation of projection data or k-space data, and information loss due to conversion from continuous values ​​to discrete values ​​that occurs during the A / D conversion process. The auxiliary input data is medical data provided to the trained model 90 to assist in the restoration of the data loss portion of the input data to be processed by the trained model 90 by providing the trained model 90 with the missing portion of the input data to be processed or a portion that substantially approximates the missing portion. Therefore, the auxiliary input data is defined as medical data relating to the same subject as the subject of the input data to be processed, and collected under different imaging conditions. The degree of difference in imaging conditions between the input image to be processed and the auxiliary input image is set to such an extent that the auxiliary input image can be considered substantially similar to the input image to be processed. By limiting the auxiliary input data to medical data of the same subject as the subject of the input data to be processed, the accuracy and reliability of the output data to be processed can be ensured. The auxiliary input images consist of one set or multiple sets of medical images with different imaging conditions.

[0066] In the forward propagation function 114, the processing circuit 11 receives input data to be processed relating to a subject, and input data to be used as auxiliary data relating to the same subject but collected under different imaging conditions than the input data to be processed. The processing circuit 11 then applies the trained model 90 to the input data to be processed and the auxiliary input data to generate output medical data corresponding to the input data to be processed. The output medical data is medical data from which the data loss portion included in the input data to be processed has been restored. In other words, the trained model 90 is a multilayer network whose parameter p has been trained to take input data to be processed containing data loss and auxiliary input data that compensates for the data loss as input, and output medical data that does not contain the data loss. Examples of combinations of input and output for the trained model 90 are shown in Figures 4 and 5.

[0067] Figure 4 shows an example of a combination of inputs and outputs for the trained model 90. For example, as shown in Figure 4, the trained model 90 accepts two inputs: the input image to be processed, which is the input data to be processed, and the auxiliary input image, which is the auxiliary input data. The input image to be processed is medical image data relating to the subject to be processed. The auxiliary input image is medical image data relating to the same subject as the subject to be processed, but acquired under different imaging conditions than the input image to be processed. In this case, the trained model 90 outputs the output image to be processed. The output image to be processed is medical image data relating to the subject to be processed, with the data loss portion included in the input image to be processed restored.

[0068] Figure 5 shows another example of input and output combinations for the trained model 90. For example, as shown in Figure 5, the trained model 90 accepts two inputs: raw data to be processed, which is the input data to be processed, and raw data to be supplementary, which is the supplementary input data. Raw data to be processed is raw data relating to the subject to be processed. Raw data to be supplementary is raw data relating to the same subject as the subject to be processed, but acquired under different imaging conditions than the raw data to be processed. In this case, the trained model 90 outputs raw data to be processed. Raw data to be processed is raw data relating to the subject to be processed, with data loss portions included in the raw data to be processed restored.

[0069] Furthermore, the raw data according to this embodiment is not limited to the original raw data collected by the medical imaging device 3. For example, the raw data according to this embodiment may be computational raw data generated by applying forward projection processing to a medical image generated by the normal restoration function 112 or the forward propagation function 114. Also, the raw data according to this embodiment may be raw data to which any data processing such as data compression processing, resolution decomposition processing, data interpolation processing, or resolution synthesis processing has been performed on the original raw data. In addition, in the case of three-dimensional raw data, the raw data according to this embodiment may be hybrid data to which restoration processing has been performed on only one or two axes. Similarly, the medical image according to this embodiment is not limited to the original medical image generated by the normal restoration function 112 or the forward propagation function 114. For example, the medical image according to this embodiment may be a medical image to which any image processing such as image compression processing, resolution decomposition processing, image interpolation processing, or resolution synthesis processing has been performed on the original medical image.

[0070] In the image processing function 115, the processing circuit 11 applies various image processing to the medical image generated by the normal restoration function 112, the output image to be processed generated by the forward propagation function 114, etc. For example, the processing circuit 11 performs 3D image processing such as volume rendering, surface volume rendering, pixel value projection processing, MPR (Multi-Planer Reconstruction) processing, and CPR (Curved MPR) processing. The processing circuit 11 may also perform alignment processing as an image processing.

[0071] In the display control function 116, the processing circuit 11 displays various information on the display 19. For example, the processing circuit 11 displays the medical image generated by the normal restoration function 112, the output image to be processed generated by the forward propagation function 114, and the medical image processed by the image processing function 115. The processing circuit 44 may also display the input data to be processed and the auxiliary input data selected by the input selection function 113.

[0072] Memory 13 is a storage device such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or integrated circuit storage device that stores various types of information. For example, memory 13 stores trained models generated by the model learning device 5. In addition to the above storage devices, memory 13 may also be a drive device that reads and writes various types of information to and from portable storage media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), flash memory, or semiconductor memory elements. Furthermore, memory 13 may be located in another computer connected to the medical data processing device 1 via a network.

[0073] The input interface 15 receives various input operations from the user, converts the received input operations into electrical signals, and outputs them to the processing circuit 11. Specifically, the input interface 15 is connected to input devices such as a mouse, keyboard, trackball, switch, button, joystick, touchpad, and touch panel display. The input interface 15 outputs electrical signals to the processing circuit 11 corresponding to the input operations to the input device. Furthermore, the input device connected to the input interface 15 may also be an input device provided on another computer connected via a network or the like.

[0074] The communication interface 17 is an interface for data communication between the medical imaging device 3, the model learning device 5, the learning data storage device 7, and other computers.

[0075] The display 19 displays various information according to the display control function 116 of the processing circuit 11. For example, the display 19 displays medical images generated by the normal restoration function 112, output images to be processed generated by the forward propagation function 114, and medical images processed by the image processing function 115. The display 19 also outputs a GUI (Graphical User Interface) to accept various operations from the user. For example, the display 19 can be a liquid crystal display (LCD), a cathode ray tube (CRT), an organic electroluminescent display (OELD), a plasma display, or any other display as appropriate.

[0076] The following describes the processing performed by the medical data processing device 1. In the following description, the medical data is assumed to be a medical image.

[0077] Figure 6 is a diagram showing the detailed structure of the trained model 90 according to this embodiment. As shown in Figure 6, the trained model 90 according to this embodiment has an input layer 91, an intermediate layer 93, and an output layer 95. The input layer 91 receives the input image to be processed and the auxiliary input image. The input image to be processed and the auxiliary input image are input in the above input formats (2), (3), or (5). For example, as shown in Figure 6, format (2) is adopted. In this case, the components (pixel values) of the input image to be processed and the auxiliary input image are input to the input layer 91 as a single input vector 92. Here, assuming that the number of pixels in the input image to be processed is q, the number of pixels in the auxiliary input image is r, and q+r=N, the input layer 91 is provided with N input units. The input layer 91 is divided into a range of input units for the input image to be processed (hereinafter referred to as the processing range) 921 and a range of input units for the auxiliary input image (hereinafter referred to as the auxiliary range) 922. The processing range 921 is the p-th pixel value x of the input image to be processed. p The input unit includes q input units where (1≦p≦q) is input, and the auxiliary range 922 is the p-th pixel value x of the auxiliary input image. p It contains r input units, each receiving input such that (1 ≤ p ≤ r).

[0078] When the raw input data to be processed and the auxiliary input data are input to the trained model 90, the components of the raw input data to be processed and the auxiliary input data are data values.

[0079] The output layer 95 outputs the output image to be processed. The output image to be processed is output from the output layer 95 in the form of a single output vector 96. The output vector 96 contains multiple components y. Each component y is the pixel value of each pixel in the output image to be processed. The range 961 of the output unit of the output layer 95 is limited to the range for a single output image to be processed. The number M of components y is not necessarily set to be the same as the number of pixels q of the input image to be processed. The number M may be less than or more than the number of pixels q.

[0080] The input image to be processed is always input to the input unit in the processing range 921 and not to the input unit in the auxiliary range 922. Conversely, the auxiliary input image is always input to the input unit in the auxiliary range 922 and not to the input unit in the processing range 921. In other words, the positions of the input units to which the processing input image and the auxiliary input image are input in the input layer 91 do not differ with each forward propagation by the trained model 90, and are always fixed. This is because the trained model 90 recognizes images input to the processing range 921 as the processing input image and images input to the auxiliary range 922 as the auxiliary input image. The positions of the input units to which the processing input image and the auxiliary input image are input in the input layer 91 are determined according to the positions to which the processing input image and the auxiliary input image were input during the training of the multilayer network. In other words, if, during training, the input image to be processed is input to the first half of the input layer 91 and the auxiliary input image is input to the second half of the input layer 91, the first half is set as the input position for the input image to be processed, and the second half is set as the input position for the auxiliary input image.

[0081] In the above description, it was assumed that the processing range 921 of the input unit for the input image to be processed is set to the first half of the input layer 91, and the auxiliary range 922 of the input unit for the auxiliary input image is set to the second half of the input layer 91. However, this embodiment is not limited to this. The processing range 921 of the input unit for the input image to be processed may be set to the second half of the input layer 91, and the auxiliary range 922 of the input unit for the auxiliary input image may be set to the first half of the input layer 91.

[0082] The auxiliary input images may be two or more sets of medical images with different imaging conditions. In this case, the components of the target input image and the two or more auxiliary input images are input to the input layer 91 as a single input vector 92. It is desirable that the range of the input units for the two or more auxiliary input images does not differ with each forward propagation and is always fixed. For example, if the range of the input unit for the first auxiliary input image relating to a frame 1 second after the target input image is q~r, and the range of the input unit for the second auxiliary input image relating to a frame 2 seconds after is r+1~s, then it is desirable that the first auxiliary input image relating to the frame 1 second after be input to the input unit in the range of q~r and not input to the input unit in the range of r+1~s. If necessary, the first auxiliary input image relating to the frame 1 second after may be input to the input unit in the range of r+1~s.

[0083] Next, an example of operation by the medical data processing device 1 according to this embodiment will be described. In the following description, the multilayer network according to this embodiment is assumed to be a deep neural network (DNN), which is a multilayer network model that mimics the neural circuits of a biological brain. The medical data processing device 1 performs DNN reconstruction based on the raw data collected by the medical imaging device 3 and generates medical image data related to the subject. DNN reconstruction according to this embodiment refers to a method of reconstructing raw data into a medical image using a trained model 90, which is a trained DNN 90.

[0084] The DNN in this embodiment can have any structure. For example, ResNet (Residual Network), DenseNet (Dense Convolutional Network), U-Net, etc., can be used as the DNN in this embodiment.

[0085] Figure 7 shows a typical flow of DNN reconstruction processing by the medical data processing device 1. At the start of Figure 7, it is assumed that the medical imaging device 3 has collected the raw data to be processed regarding the subject and transmitted it to the medical data processing device 1. The raw data to be processed includes data loss. When the user issues a command to start DNN reconstruction processing via an input device, the processing circuit 11 executes the DNN reconstruction program and starts the processing shown in Figure 7.

[0086] As shown in Figure 7, the processing circuit 11 performs a normal restoration function 112 (step SA1). In step SA1, the processing circuit 11 generates an input image to be processed by applying a normal reconstruction process to the raw data to be processed. Since the raw data to be processed contains data loss, the image quality of the input image to be processed is not good. The input image to be processed generated by applying a normal restoration process to the raw data to be processed can also be called a provisionally restored image.

[0087] When step SA1 is performed, the processing circuit 11 executes the input selection function 113 (step SA2). In step SA2, the processing circuit 11 selects the input image to be processed and the auxiliary input image. For example, the processing circuit 11 selects the input image to be processed generated in step SA1 as the input image. The processing circuit 11 then selects an auxiliary input image for the input image to be processed as the input image. The auxiliary input image is a medical image relating to the same subject as the input image to be processed, and is selected based on imaging conditions different from those of the input image to be processed. The processing circuit 11 may automatically select the auxiliary input image according to predetermined rules, or it may manually select the auxiliary input image according to instructions from the user via an input device or the like.

[0088] The auxiliary input image may be selected according to the type of trained DNN90 used. The trained DNN90 according to this embodiment is generated for each imaging condition of the auxiliary input image that differs from the imaging conditions of the input image to be processed. For example, when using a trained DNN90 that uses a medical image of a different slice from the input image to be processed as the auxiliary input image, the processing circuit 11 selects a medical image of a different slice from the medical image to be processed as the auxiliary input image. For example, a medical image of a slice that is physically (spatially and / or temporally) close to the slice of the input image to be processed is selected as the auxiliary input image. In the case of an electrocardiogram-synchronized scan, a medical image of a slice with the same or temporally close cardiac phase as the slice of the input image to be processed is selected as the auxiliary input image. Candidate auxiliary input images are stored in advance in the memory 13.

[0089] The number of auxiliary input images selected can be one or multiple. The number of selected auxiliary input images is set to the number of auxiliary input images input during the training of the DNN being used. That is, if the DNN being used was trained with one medical image as an auxiliary input image, then one medical image will be selected as an auxiliary input image when the DNN is restored. For example, if two medical images were input as auxiliary input images during the training of the DNN, then two medical images will be selected as auxiliary input images when the DNN is restored.

[0090] Other arbitrary constraints may be imposed on the auxiliary input images to be selected. For example, the processing circuit 11 may select only medical images with the same FOV as the input image to be processed as auxiliary input images. In addition, the processing circuit 11 may exclude medical images taken more than a predetermined number of days (for example, two months) before the date the input image to be processed was taken from the list of candidate auxiliary input images.

[0091] When step SA2 is performed, the processing circuit 11 executes the forward propagation function 114 (step SA3). In step SA3, the processing circuit 11 reads the trained DNN 90 to be used from the memory 13. The trained DNN 90 to be read may be specified by the user via an input device. The processing circuit 11 then applies the read trained DNN 90 to the input image and auxiliary input image selected in step SA2 to generate the output image to be processed. For example, the output image y to be processed is calculated by performing calculations (4-1) and (4-2) using the input image and auxiliary input image as input x in equation (3-2).

[0092] A trained DNN90 may be generated for each imaging site and stored in memory 13. By generating a trained DNN90 for each imaging site, the reconstruction accuracy of the output image to be processed can be improved. The processing circuit 11 switches the trained DNN90 according to the selected imaging site. The imaging site may be selected by the user via an input device when the DNN reconstruction process is executed. In this case, the processing circuit 11 reads the trained DNN90 associated with the selected imaging site from memory 13. If an imaging site has already been selected for medical imaging, the processing circuit 11 may automatically read the trained model 90 associated with the selected imaging site from memory 13. If the medical imaging device 3 uses different equipment depending on the imaging site, the imaging site corresponding to that equipment may be automatically identified, and the trained model 90 associated with the identified imaging site may be automatically read from memory 13. For example, in magnetic resonance imaging, the type of coil used, such as a head coil or an abdominal coil, differs depending on the imaging site. In this case, the processing circuit 11 identifies the imaging area corresponding to the type of coil used, based on the coil identifier, etc., and reads the trained model 90 associated with the identified imaging area from the memory 13.

[0093] When step SA3 is performed, the processing circuit 11 executes the display control function 116 (step SA4). In step SA4, the processing circuit 11 displays the output image to be processed, which was generated in step S3, on the display 19.

[0094] This concludes the explanation of the DNN reconstruction process flow shown in Figure 7. Note that the above DNN reconstruction process flow is an example, and the DNN reconstruction process flow according to this embodiment is not limited thereto. For example, if the medical data processing device 1 is independent of the medical image diagnostic device 9, both candidate input images to be processed and candidate auxiliary input images may be stored in memory 13 in advance, and the input image to be processed and the auxiliary input image may be selected from among these candidates. Furthermore, pre-processing such as resolution decomposition and resolution synthesis may be performed on the raw data, and post-processing such as volume rendering and image analysis may be performed on the output image to be processed. Also, the processing circuit 11 may align the input image to be processed and the auxiliary input image in the step prior to the forward propagation process in step SA3. Furthermore, the input image to be processed and the auxiliary input image may be selected after reading out the trained DNN 90 to be used.

[0095] Furthermore, the processing circuit 11 should manage the output image to be processed in such a way that the user can understand that the output image to be processed was generated using data other than raw data. For example, if the processing circuit 11 manages the images in accordance with the DICOM (Digital Imaging and Communications in Medicine) standard, it should associate the output image to be processed with the input image to be processed and the auxiliary input image, thereby managing these three images as a single image set. For example, the output image to be processed, the input image to be processed, and the auxiliary input image are stored in the image file of the output image to be processed or the input image to be processed. In addition to or in combination with the above method, the processing circuit 11 may assign identification information of the auxiliary input image to the DICOM tag of the output image to be processed. These methods allow evidence that the output image to be processed was generated using the auxiliary input image to be left in the output image to be processed. The processing circuit 11 may also assign identification information of the output image to be processed to the DICOM tag of the input image to be processed. This makes the existence of the output image to be processed clear.

[0096] Next, we will explain a specific example of the DNN restoration process.

[0097] As described above, the input image to be processed and the auxiliary input image relate to the same subject but under different imaging conditions. The imaging conditions consist of the imaging principle and multiple types of imaging parameters. The imaging parameters are classified into common parameters and individual parameters. Common parameters are imaging parameters that are set to the same value for the input image to be processed and the auxiliary input image. Individual parameters are imaging parameters that are set to different values ​​for the input image to be processed and the auxiliary input image. Note that the common parameters and individual parameters are the same value for the input image to be processed and the output image to be processed. The input image to be processed and the output image to be processed differ in the amount of data loss or image quality. That is, the output image to be processed has less data loss or higher image quality compared to the input image to be processed. The amount of data loss and image quality can be evaluated using image quality parameters such as image SD.

[0098] Figure 8 schematically shows the relationship between the input and output of a trained DNN in the forward propagation process according to this embodiment. The trained DNN shown in Figure 8 takes a medical image as an auxiliary input image, which has a different slice position from the input image to be processed. As the output image to be processed, it outputs a medical image with the same slice position as the input image to be processed, in which the data loss portion contained in the input image to be processed has been restored. For example, as shown in Figure 8, the slice of the input image to be processed is at position sA, the slice of the auxiliary input image is at position sB which is different from position sA, and the slice of the output image to be processed is at position sA. The distance and angle of position sB with respect to position sA are not particularly limited, but it is preferable that they are always the same value and do not differ with each DNN reconstruction. In the case of Figure 8, the individual parameter is the slice position, and the common parameters are the type of acquisition sequence, the type of k-space filling trajectory, the time resolution, etc.

[0099] Figure 9 is another diagram schematically showing the relationship between the input and output of the trained DNN in the forward propagation process according to this embodiment. The trained DNN shown in Figure 9 takes a medical image as an auxiliary input image, which has a different frame from the input image to be processed. In this embodiment, the frame corresponds to the acquisition time of the medical image or raw data. As the output image to be processed, a medical image is output with the same frame as the input image to be processed, but with the data loss portion contained in the input image to be processed restored. For example, as shown in Figure 9, the frame of the input image to be processed is time phase tA, the frame of the auxiliary input image is time phase tB, and the frame of the output image to be processed is time phase tA. The time difference between time phase tA and time phase tB is not particularly limited, but it is preferable that it is always the same value and does not differ with each DNN restoration. In the case of Figure 9, the individual parameter is the frame (acquisition time), and the common parameters are the slice position, the type of acquisition sequence, the type of k-space filling trajectory, the time resolution, etc.

[0100] Figure 10 is another diagram schematically showing the relationship between the input and output of a trained DNN in the forward propagation process according to this embodiment. The trained DNN shown in Figure 10 takes a medical image as an auxiliary input image, in which both the slice position and frame differ from the target input image. As the target output image, a medical image is output with the same slice position and frame as the target input image, but with the data loss portion contained in the target input image restored. For example, as shown in Figure 10, the slice of the target input image is at position sA and the frame is at time phase tA, the slice of the auxiliary input image is at position sB and the frame is at time phase tB, and the slice of the target output image is at position sA and the frame is at time phase tA. In the case of Figure 10, the individual parameters are either the slice position or the frame, and the common parameters are the type of acquisition sequence, the type of k-space filling trajectory, the time resolution, etc. Note that if multiple images are selected as auxiliary input images, either only one of the slice position and frame may be set to a different value for the multiple auxiliary input images, or both may be set to different values.

[0101] Figure 11 is another diagram schematically showing the relationship between the input and output of the trained DNN in the forward propagation process according to this embodiment. In the trained DNN shown in Figure 11, the k-space filling trajectory in magnetic resonance imaging differs from that of the target input image as the auxiliary input image. The target output image has the same k-space filling trajectory as the target input image, and outputs a medical image in which the data loss portion contained in the target input image has been restored. Examples of k-space filling trajectories include Cartesian scan, Radial scan, PROPELLAR (Periodically Rotated Overlapping Parallel Lines with Enhanced Reconstruction), Spinal scan, and Stack-of-Stars. For example, as shown in Figure 11, the k-space filling trajectory of the target input image is a Radial scan, the k-space filling trajectory of the auxiliary input image is a Cartesian scan, and the k-space filling trajectory of the target output image is a Radial scan. Note that the k-space filling trajectory according to this embodiment also includes the acquisition order of the acquisition lines. The acquisition lines in this embodiment correspond to the phase encoding step in a Cartesian scan and the spokes in a radial scan. The acquisition order in a Cartesian scan can be centric or sequential. For example, the k-space filling trajectory of the target input image may be centric, while the k-space filling trajectory of the auxiliary input image may be sequential. In Figure 11, the individual parameters are the types of k-space filling trajectories, while the common parameters are the types of acquisition sequences and slice positions.

[0102] Figure 12 is another diagram schematically showing the relationship between the input and output of the trained DNN in the forward propagation process according to this embodiment. The trained DNN shown in Figure 12 takes a medical image as an auxiliary input image, which has a different acquisition sequence in magnetic resonance imaging from the input image to be processed. As the output image to be processed, a medical image is output with the same acquisition sequence as the input image to be processed, but with the data loss portion contained in the input image to be processed restored. Examples of acquisition sequences according to this embodiment include gradient echo (GRE) sequences and spin echo (SE) sequences. Furthermore, preparation pulses such as inversion recovery (IR) pulses and fat saturation pulses may be inserted into any pulse sequence, or echo planar imaging (EPI) may be used. In addition, factors determining the acquisition sequence according to this embodiment include not only the type of acquisition sequence, but also various parameters of the acquisition sequence such as the repetition time TR, echo time TE, longitudinal relaxation time T1, transverse relaxation time T2, b value, and doubling rate. For example, as shown in Figure 12, the acquisition sequence for the input image to be processed is EPI acquisition, the acquisition sequence for the auxiliary input image is T1W acquisition, and the acquisition sequence for the output image to be processed is EPI acquisition. In Figure 12, the individual parameters are the types of acquisition sequences, and the common parameters are the types of k-space filling trajectories, slice positions, and temporal resolution.

[0103] Figure 13 is another diagram schematically showing the relationship between the input and output of a trained DNN in the forward propagation process according to this embodiment. In the trained DNN shown in Figure 13, the target input image and two types of auxiliary input images are input to the trained DNN. The target output image is a medical image in which the data loss portion contained in the target input image has been restored, with the same acquisition sequence as the target input image. Each of the two types of auxiliary input images has a different acquisition sequence from the target input image. For example, the acquisition sequence for the target input image is EPI acquisition with a b value of 500, the acquisition sequence for the first auxiliary input image is EPI acquisition with a b value of 0, and the acquisition sequence for the second auxiliary input image is T1W acquisition. In this case, the acquisition sequence for the target output image is EPI acquisition with a b value of 500. In the case of Figure 13, the individual parameters are either the type of acquisition sequence or the b value, while the common parameters are the slice position, the type of k-space filling trajectory, the time resolution, etc. Furthermore, if multiple images are selected as auxiliary input images, the type of acquisition sequence and the b-value may be set to different values ​​for each of the auxiliary input images, or both may be set to different values.

[0104] The above-mentioned types of input images to be processed and auxiliary input images are merely examples, and this embodiment is not limited thereto.

[0105] For example, the following types can be considered as two input images with different k-space filling trajectories: an MR image collected by random undersampling as the input image to be processed, and an MR image collected by regular undersampling as the auxiliary input image. Regular undersampling is a decimation method using Cartesian scanning. Random undersampling, also known as pseudo-radial scanning, is a method of collecting k-space data along a pseudo-radial acquisition line using Cartesian scanning.

[0106] In the above embodiment, the medical data processing device 1 was assumed to be a computer included in the medical image diagnostic device 9. However, the medical data processing device according to this embodiment is not limited to this.

[0107] Figure 14 shows the configuration of another medical data processing device 2 according to this embodiment. The medical data processing device 2 is a dedicated device for the forward propagation function 114. The medical data processing device 2 may be implemented by a computer not included in the medical imaging diagnostic device 9. The medical imaging diagnostic device 2 may also be implemented by an integrated circuit such as an ASIC or FPGA, which may or may not be included in the medical imaging diagnostic device 9. Hereinafter, the medical data processing device 2 will be assumed to be an ASIC.

[0108] As shown in Figure 14, the medical data processing device 2 includes a processing circuit 21, a memory 23, an input interface 25, and an output interface 27. The processing circuit 21, memory 23, input interface 25, and output interface 27 are connected to each other via a bus.

[0109] The processing circuit 21 is a combination of circuit elements or logic circuits designed to perform the forward propagation function 114. The processing circuit 21 applies a trained model to the input image to be processed and the auxiliary input image input via the input interface 25 to generate an output image to be processed, and outputs the output image to be processed via the output interface 27.

[0110] Memory 23 is a circuit element that stores arbitrary information such as ROM or RAM. For example, memory 23 stores calculation results obtained when the forward propagation function 114 is executed.

[0111] The input interface 25 is an interface for input to the processing circuit 21. The input interface 25 inputs, for example, the input image to be processed and the auxiliary input image to the processing circuit 21. The input image to be processed and the auxiliary input image are selected, for example, by a computer equipped with the medical data processing device 2.

[0112] The output interface 27 is an interface for output from the processing circuit 21. For example, the output interface 27 outputs the processed output image output from the processing circuit 21 to a computer, network, or storage device.

[0113] With the above configuration, the medical data processing device 2 can perform the forward propagation function 114 in a form other than the computer included in the medical image diagnostic device 9. Note that the configuration of the medical data processing device 2 in Figure 14 is an example and is not limited thereto. For example, the medical data processing device 2 does not have to have a memory 23. Also, the processing circuit 21 may be equipped with functions other than the forward propagation function 114.

[0114] Figure 15 shows the configuration of the model learning device 5. As shown in Figure 15, the model learning device 5 has a processing circuit 51, memory 53, input interface 55, communication interface 57, and display 59 as hardware resources.

[0115] The processing circuit 51 has a processor such as a CPU or GPU. The processor starts a DNN restoration program installed in memory 53, etc., and executes the forward propagation function 511, the backward propagation function 512, the update function 513, the judgment function 514, and the display control function 515, etc. Note that each of the functions 511 to 515 is not limited to being implemented by a single processing circuit. A processing circuit may be configured by combining multiple independent processors, and each of the functions 511 to 515 may be implemented by each processor executing a program.

[0116] In the forward propagation function 511, the processing circuit 51 forward propagates the input medical data to the multilayer network and calculates estimated output data corresponding to the input medical data.

[0117] In the backpropagation function 512, the processing circuit 51 backpropagates the error through the multilayer network and calculates a gradient vector. The error is defined as the difference between the estimated output data calculated by the forward propagation function 511 and the correct output data.

[0118] In the update function 513, the processing circuit 51 updates the parameters of the multilayer network based on the gradient vector calculated by the backpropagation function 512. Specifically, the processing circuit 51 updates the parameters so that the estimated output medical data and the ground truth output medical data approximate each other.

[0119] In the determination function 514, the processing circuit 51 determines whether or not the termination condition for the learning process is met. The termination condition can be arbitrarily set by the user via an input device or the like.

[0120] In the display control function 515, the processing circuit 51 displays the learning data and learning results on the display 59.

[0121] Memory 53 is a storage device such as ROM, RAM, HDD, SSD, or integrated circuit storage device that stores various types of information. For example, memory 53 stores a model learning program 50 for learning a multilayer network. In addition to the above-mentioned storage devices, memory 53 may also be a drive device that reads and writes various types of information to portable storage media such as CDs, DVDs, or flash memory, or to semiconductor memory elements such as RAM. Furthermore, memory 53 may be located in another computer connected to the model learning device 5 via a network.

[0122] The input interface 55 receives various input operations from the user, converts the received input operations into electrical signals, and outputs them to the processing circuit 51. Specifically, the input interface 15 is connected to input devices such as a mouse, keyboard, trackball, switch, button, joystick, touchpad, and touch panel display. The input interface 55 outputs electrical signals to the processing circuit 51 corresponding to the input operations to the input device. Furthermore, the input device connected to the input interface 55 may be an input device provided on another computer connected via a network or the like.

[0123] The communication interface 57 is an interface for data communication between the medical data processing device 1, the medical imaging device 3, the learning data storage device 7, and other computers.

[0124] The display 59 displays various information according to the display control function 515 of the processing circuit 51. For example, the display 59 displays learning data and learning results. The display 59 also outputs a GUI for accepting various operations from the user. For example, the display 19 can be an LCD display, a CRT display, an organic EL display, a plasma display, or any other display as appropriate.

[0125] Next, the model learning process executed by the processing circuit 51 of the model learning device 5 according to the model learning program 50 will be described. Figure 16 is a diagram showing a typical flow of the model learning process executed by the processing circuit 51 of the model learning device 5 according to the model learning program 50. The process in Figure 16 is started when the processing circuit 51 executes the model learning program 50, triggered by the input of a start command for the model learning process from the user or the like.

[0126] First, the processing circuit 51 executes the forward propagation function 511 (step SB1). In step SB1, the processing circuit 51 receives training data containing multiple training samples. The training data is supplied from the training data storage device 7. The training samples are a combination of main input data, auxiliary input data, and correct output data. The main input data and auxiliary input data are medical data collected from the same subject under different imaging conditions. The main input data corresponds to the input data to be processed. The correct output data is the desired output data (correct output or training data) output from the DNN after receiving the main input data.

[0127] As described above, imaging parameters include common parameters and individual parameters. For main input data and auxiliary input data, common parameters are set to the same value, but individual parameters are set to different values. For main input data and ground truth output data, common parameters and individual parameters are set to the same value. Ground truth output data is data with fewer data loss or higher image quality than main input data. For example, if the imaging parameters are MRI imaging parameters, the main input data is an MR image based on k-space data with a small amount of data, and the ground truth output data is an MR image based on k-space data with a larger amount of data than the main input data. For example, if the main input data is data collected by sparse sampling, the ground truth output data should be data collected by full sampling. Individual parameters of auxiliary input data and individual parameters of ground truth output data are set to different values ​​from each other. If there are multiple sets of auxiliary input data, the individual parameters for different sets of auxiliary input data may be the same value or different values. Ground truth output data originates from the main input data but is unrelated to the auxiliary input data. When the data input and output to a DNN is k-space data, typically the ground truth output k-space data includes the primary input k-space data but does not include the auxiliary input k-space data. When the data input and output to a DNN is an image, typically the k-space data used to reconstruct the ground truth output image includes the k-space data used to reconstruct the primary input image but does not include the k-space data used to reconstruct the auxiliary input image.

[0128] Across multiple training samples input to the same DNN, the combination of imaging conditions for the primary input data and the secondary input data remains the same. More specifically, the combination of imaging conditions for the primary input data and the secondary input data is fixed to the combination of imaging conditions for the primary input data and the secondary input data related to the DNN being generated. For example, when generating the DNN in Figure 8, a medical image with a different slice position relative to the primary input medical image is selected as the secondary input image. When generating the DNN in Figure 9, a medical image with a different frame relative to the primary input medical image is selected as the secondary input image. When generating the DNN in Figure 10, a medical image with a different slice position and a medical image with a different frame relative to the primary input medical image are selected as secondary input images. When generating the DNN in Figure 11, a medical image with a different k-space filling trajectory relative to the primary input medical image is selected as the secondary input image. When generating the DNN in Figure 12, a medical image with a different acquisition sequence relative to the primary input medical image is selected as the secondary input image. When generating the DNN shown in Figure 13, two medical images with different acquisition sequences are selected as auxiliary input images for the primary input medical image.

[0129] Furthermore, the primary input medical image and auxiliary input image during training are not limited to medical images generated by imaging a patient. For example, medical images generated by imaging any phantom may be used as the primary input medical image and auxiliary input image.

[0130] When step SB1 is performed, the processing circuit 51 generates output data through forward propagation of the DNN based on the main input data and auxiliary input data (step SB2). The output data generated in step SB2 is called estimated output data. Note that the DNN parameters are set to their initial values ​​during the first forward propagation. For example, by taking the main input data and auxiliary input data as inputs and performing the above equations (4-1) and (4-2), the estimated output data z (L) This is generated. Furthermore, to improve learning efficiency and accuracy, it is desirable to align the primary input data with the auxiliary input data in the pre-forward propagation stage.

[0131] When step SB2 is performed, the processing circuit 51 executes the backpropagation function 512 (step SB3). In step SB3, the processing circuit 51 calculates the error between the estimated output data generated in step SB2 and the correct output data input in step SB1. Specifically, the correct output data d n Estimated output data z (L) By subtracting this, the error δ defined by equation (13-1) is obtained. j (L) This is calculated.

[0132] When step SB3 is performed, the processing circuit 51 calculates the gradient vector by backpropagation of the DNN based on the error calculated in step SB3 (step SB4). Specifically, the error δ j (L) Based on this, the gradient vector of equation (6-2) or equation (6-4) is calculated.

[0133] When step SB4 is performed, the processing circuit 51 executes the update function 513 (step SB5). In step SB5, the processing circuit 51 updates the parameters based on the gradient vector calculated in step SB4. Specifically, the parameter p is updated based on the gradient vector using equation (6-1) or equation (6-3).

[0134] When step SB5 is performed, the processing circuit 51 executes the determination function 514 (step SB6). In step SB6, the processing circuit 51 determines whether or not the termination condition is met. The termination condition may be set, for example, when the number of iterations reaches a specified number. Alternatively, the termination condition may be set when the gradient vector reaches a threshold.

[0135] If it is determined that the termination condition is not met in step SB6 (step SB6: NO), the processing circuit 11 repeats steps SB1 to SB6 using the same learning sample or a different learning sample.

[0136] Then, if it is determined in step SB6 that the termination condition is met (step SB6: YES), the processing circuit 11 outputs the updated DNN as the trained DNN 90 (step SB7). The trained DNN is stored in the memory 13 of the medical data processing device 1 along with the type and number of auxiliary input images. Specifically, the type indicates the type of imaging condition of the auxiliary input image that is different from the imaging condition of the main input image. For example, in the case of the DNN in Figure 8, the type is slice position and the number is 1; in the case of the DNN in Figure 9, the type is frame and the number is 1; in the case of the DNN in Figure 10, the type is slice position and frame and the number is 2; in the case of the DNN in Figure 11, the type is k-space filling trajectory and the number is 1; in the case of the DNN in Figure 12, the type is acquisition sequence and the number is 1; and in the case of the DNN in Figure 13, the type is acquisition sequence and the number is 2.

[0137] This concludes the explanation of the model learning process by the model learning device 5 according to this embodiment. Note that the learning process flow described above is just one example, and this embodiment is not limited thereto.

[0138] As described above, the model learning program 50 according to this embodiment causes the model learning device 5 to execute at least a forward propagation function 511 and an update function 513. The forward propagation function 511 generates estimated output data by applying the main input data and auxiliary input data to a multilayer network having an input layer that inputs main input data and auxiliary input data relating to the same object but with different imaging conditions, an output layer that outputs output data corresponding to the main input data, and at least one intermediate layer provided between the input layer and the output layer. The update function 513 updates the parameters of the multilayer network so that the estimated output data and the ground truth output data approximate each other.

[0139] With the above configuration, the model learning program 50 according to this embodiment outputs output data in which missing data portions of the main input data have been restored. Therefore, it uses not only the main input data but also the auxiliary input data as input data to learn the parameters of the multilayer network. As a result, the model learning program 50 according to this embodiment can utilize more information not included in the main input data when learning parameters. This improves the accuracy of medical data restoration by the trained model 90 compared to when only the main input data is used as input data.

[0140] Furthermore, as described above, the medical data processing device 1 according to this embodiment includes a memory 13 and a processing circuit 11. The memory 13 stores a trained model 90. The trained model 90 has an input layer that inputs main input data and auxiliary input data relating to the same subject but with different imaging conditions, an output layer that outputs output data corresponding to the main input data, and at least one intermediate layer provided between the input layer and the output layer. The processing circuit 11 applies the trained model to the input data to be processed relating to the subject and auxiliary input data relating to the same subject but with different imaging conditions than the input data to be processed, and generates output data to be processed relating to the subject.

[0141] With the above configuration, the medical data processing device 1 in this embodiment can compensate for missing portions of the input data to be processed using auxiliary input data. Therefore, compared to the case where only the input data to be processed is used as input data, the accuracy of medical data reconstruction by the trained model 90 can be improved.

[0142] Furthermore, the model learning program 50 stored in the memory 53 of the model learning device 5 may also be stored in the memory 13 of the medical data processing device 1. In other words, the learning functions of the model learning device 5, namely the forward propagation function 511, the backward propagation function 512, the update function 513, the judgment function 514, and the display control function 515, may be implemented by the processing circuit 11 of the medical data processing device 1.

[0143] (Examples of application) In the above description, no particular limitations were placed on the timing of the generation of auxiliary input images. In the following application example, multiple medical images are generated sequentially during a single examination, and the medical image to be processed and the auxiliary input image are selected sequentially from among the generated medical images. Hereafter, the process of restoring raw data to medical image data is assumed to be a reconstruction process. In the following description, components having substantially the same function as those in the above embodiment are denoted by the same reference numerals and are described redundantly only when necessary.

[0144] Application Example 1: Magnetic Resonance Imaging Magnetic resonance imaging (MRI) is performed using a magnetic resonance imaging device. In Application Example 1, medical imaging device 3 is assumed to be the MR stand of a magnetic resonance imaging device.

[0145] Figure 17 shows the configuration of the magnetic resonance imaging apparatus 9-1 according to Application Example 1. As shown in Figure 17, the magnetic resonance imaging apparatus 9-1 includes a stand 20-1, a patient bed 31-1, a gradient magnetic field power supply 21-1, a transmitting circuit 23-1, a receiving circuit 25-1, a patient bed drive device 27-1, a sequence control circuit 29-1, and a medical data processing device 1-1.

[0146] The mount 20-1 includes a static magnetic field magnet 41-1 and a gradient magnetic field coil 43-1. The static magnetic field magnet 41-1 and the gradient magnetic field coil 43-1 are housed in the casing of the mount 20-1. The casing of the mount 20-1 has a hollow bore. A transmitting coil 45-1 and a receiving coil 47-1 are arranged inside the bore of the mount 20-1.

[0147] The static magnetic field magnet 41-1 has a hollow, approximately cylindrical shape and generates a static magnetic field inside the approximately cylindrical interior. For example, a permanent magnet, a superconducting magnet, or a normal conducting magnet can be used as the static magnetic field magnet 41-1. Here, the central axis of the static magnetic field magnet 41-1 is defined as the Z-axis, the axis perpendicular to the Z-axis is defined as the Y-axis, and the axis perpendicular to the Z-axis horizontally is defined as the X-axis. The X-axis, Y-axis, and Z-axis constitute an orthogonal three-dimensional coordinate system.

[0148] The gradient coil 43-1 is mounted inside the static magnetic field magnet 41-1 and is a hollow, substantially cylindrical coil unit. The gradient coil 43-1 generates a gradient magnetic field by receiving current from the gradient power supply 21-1. More specifically, the gradient coil 43-1 has three coils corresponding to the mutually orthogonal X, Y, and Z axes. These three coils form a gradient magnetic field in which the magnetic field strength changes along each of the X, Y, and Z axes. The gradient magnetic fields along each of the X, Y, and Z axes are combined to form mutually orthogonal slice-selective gradient magnetic field Gs, phase-encoded gradient magnetic field Gp, and frequency-encoded gradient magnetic field Gr in the desired direction. The slice-selective gradient magnetic field Gs is used to arbitrarily determine the imaging cross-section. The phase-encoded gradient magnetic field Gp is ​​used to change the phase of the MR signal according to the spatial position. The frequency-encoded gradient magnetic field Gr is used to change the frequency of the MR signal according to the spatial position. In the following explanation, the gradient direction of the slice selection gradient magnetic field Gs is assumed to be the Z-axis, the gradient direction of the phase encoding gradient magnetic field Gp is ​​assumed to be the Y-axis, and the gradient direction of the frequency encoding gradient magnetic field Gr is assumed to be the X-axis.

[0149] The gradient power supply 21-1 supplies current to the gradient coil 43-1 according to the sequence control signal from the sequence control circuit 29-1. By supplying current to the gradient coil 43-1, the gradient power supply 21-1 generates gradient magnetic fields along the X, Y, and Z axes using the gradient coil 43-1. These gradient magnetic fields are superimposed on the static magnetic field formed by the static magnetic field magnet 41-1 and applied to the subject P.

[0150] The transmitting coil 45-1 is, for example, positioned inside the gradient magnetic field coil 43-1 and receives current from the transmitting circuit 23-1 to generate a high-frequency magnetic field pulse (hereinafter referred to as an RF magnetic field pulse).

[0151] The transmitting circuit 23-1 supplies current to the transmitting coil 45-1 in order to apply an RF magnetic field pulse to the subject P via the transmitting coil 45-1 in order to excite the target proton present in the subject P. The RF magnetic field pulse oscillates at a resonance frequency unique to the target proton, exciting the target proton. A magnetic resonance signal (hereinafter referred to as the MR signal) is generated from the excited target proton and detected by the receiving coil 47-1. The transmitting coil 45-1 is, for example, a whole-body coil (WB coil). A whole-body coil may be used as a transmitting and receiving coil.

[0152] The receiving coil 47-1 receives MR signals emitted from target protons present in the subject P in response to the action of an RF magnetic field pulse. The receiving coil 47-1 has multiple receiving coil elements capable of receiving MR signals. The received MR signals are supplied to the receiving circuit 25-1 via wired or wireless connection. Although not shown in Figure 1, the receiving coil 47-1 has multiple receiving channels implemented in parallel. Each receiving channel has a receiving coil element that receives the MR signal and an amplifier that amplifies the MR signal. The MR signal is output for each receiving channel. The total number of receiving channels and the total number of receiving coil elements may be the same, or the total number of receiving channels may be greater than or less than the total number of receiving coil elements.

[0153] The receiving circuit 25-1 receives the MR signal generated from the excited target proton via the receiving coil 47-1. The receiving circuit 25-1 processes the received MR signal to generate a digital MR signal. The digital MR signal can be represented in k-space, which is defined by the spatial frequency. Therefore, the digital MR signal will be referred to as k-space data below. k-space data is a type of raw data used for image reconstruction. The k-space data is supplied to the signal data processing device 50-1 via wired or wireless connection.

[0154] The transmitting coil 45-1 and receiving coil 47-1 described above are merely examples. Instead of the transmitting coil 45-1 and receiving coil 47-1, a transmitting and receiving coil equipped with both transmitting and receiving functions may be used. Furthermore, the transmitting coil 45-1, receiving coil 47-1, and the transmitting and receiving coil may be combined.

[0155] A bed 31-1 is installed adjacent to the frame 20-1. The bed 31-1 has a top plate 33-1 and a base 35-1. The subject P is placed on the top plate 33-1. The base 35-1 supports the top plate 33-1 so that it can slide along the X, Y, and Z axes. The bed drive unit 27-1 is housed in the base 35-1. The bed drive unit 27-1 moves the top plate 33-1 under control from the sequence control circuit 29-1. The bed drive unit 27-1 may include any motor, such as a servo motor or a stepping motor.

[0156] The sequence control circuit 29-1 has a CPU or MPU (processor) and memory such as ROM or RAM as hardware resources. The sequence control circuit 29-1 synchronously controls the gradient magnetic field power supply 21-1, the transmission circuit 23-1 and the reception circuit 25-1 based on the imaging protocol determined by the imaging control function 111 of the processing circuit 11, executes a pulse sequence according to the imaging protocol to perform MR imaging of the subject P, and collects k-space data related to the subject P.

[0157] As shown in Figure 17, the medical data processing device 1-1 is a computer having a processing circuit 11, memory 13, input interface 15, communication interface 17, and display 19. Since the medical data processing device 1-1 is equivalent to the medical data processing device 1 described above, its explanation is omitted.

[0158] Figure 18 schematically shows the processing of the medical data processing device 1-1 according to Application Example 1. In the examination shown in Figure 18, first, EPI acquisition with b value = 0 is performed (step SP1). In step SP1, the processing circuit 11 first executes the imaging control function 111 to cause the medical imaging device (magnetic resonance imaging device stand) 3 to perform EPI acquisition with b value = 0. In step SP1, which is the first stage of the examination, complete acquisition of k-space data (full acquisition) is performed. Complete acquisition means acquiring k-space data for all acquisition lines in k-space. Once the k-space data has been acquired, the processing circuit 11 executes the normal reconstruction function 112. In the normal reconstruction function 112, the processing circuit 11 applies FFT (Fast Fourier Transfer) to the k-space data acquired in step SP1 to generate an FFT reconstructed image for b value = 0.

[0159] When step SP1 is performed, the processing circuit 11 executes the imaging control function 111 to cause the mount 20 to perform EPI acquisition with a b value of 500 (step SP2). In step SP2, sparse acquisition of k-space data is performed. Sparse acquisition is the acquisition of k-space data by limiting it to some of the acquisition lines out of all the acquisition lines in k-space. Sparse acquisition is performed by methods such as parallel imaging, half-Fourier method, or compressed sensing. Sparse acquisition can shorten the imaging time. Also, the imaging area or slice position is the same in step SP2 and step SP1. Once the k-space data is acquired, the processing circuit 11 executes the normal reconstruction function 112. In the normal reconstruction function 112, the processing circuit 11 applies an FFT to the k-space data acquired in step SP2 to generate an FFT reconstructed image with a b value of 500. The FFT reconstructed image in step SP2 includes an increase in the b value and image quality degradation due to sparse acquisition.

[0160] When an FFT reconstructed image with a b value of 500 is generated, the processing circuit 11 executes the input selection function 113. As shown in Figure 18, in the input selection function 113, the processing circuit 11 selects the FFT reconstructed image with a b value of 0 obtained by complete acquisition as the auxiliary input image, and selects the FFT reconstructed image with a b value of 500 obtained by decimation acquisition as the input image to be processed. Since the input image to be processed is an image based on decimation acquisition, the data is decimated compared to the auxiliary input image based on complete acquisition.

[0161] Next, the processing circuit 11 executes the forward propagation function 114. In the forward propagation function 114, the processing circuit 11 first reads a trained DNN (Forb=500) from memory 13, which uses an FFT reconstruction image for b=500 obtained by decimation as the input image to be processed, an FFT reconstruction image for b=0 obtained by complete acquisition as the auxiliary input image, and a DNN reconstruction image for b=500 as the output image to be processed. Next, the processing circuit 11 inputs the FFT reconstruction image for b=500 obtained by decimation into the processing range 921 of the input layer 91 of the read-out trained DNN (Forb=500), and inputs the FFT reconstruction image for b=0 obtained by complete acquisition into the auxiliary range 922. Then, the processing circuit 11 forward propagates the FFT reconstruction image for b=500 obtained by decimation and the FFT reconstruction image for b=0 obtained by complete acquisition to the trained DNN (Forb=500). This generates a DNN-reconstructed image for a b-value of 500, which reduces image quality degradation associated with increasing the b-value and thinning.

[0162] Furthermore, when step SP2 is performed, the processing circuit 11 executes the imaging control function 111 to perform T1W acquisition on the rig 20 (step SP3). In step SP3 as well, sparse acquisition of k-space data is performed to shorten the imaging time. The imaging area or slice position is the same in step SP3 and step SP1. Once the k-space data is acquired, the processing circuit 11 executes the normal reconstruction function 112. In the normal reconstruction function 112, the processing circuit 11 applies an FFT to the k-space data acquired in step SP3 to generate a T1W image. The generated T1W image is reconstructed from sparsely acquired k-space data, so it contains some degradation in image quality.

[0163] When an FFT reconstructed image for T1W is generated, the processing circuit 11 executes the input selection function 113. As shown in Figure 18, in the input selection function 113, the processing circuit 11 selects the FFT reconstructed image for b=0 obtained by complete acquisition as the auxiliary input image, and selects the FFT reconstructed image for T1W obtained by decimation acquisition as the input image to be processed.

[0164] Next, the processing circuit 11 executes the forward propagation function 114. In the forward propagation function 114, the processing circuit 11 first reads a trained DNN (ForT1W) from memory 13, which uses the FFT reconstructed image of T1W obtained by decimation as the input image to be processed, the FFT reconstructed image of b-value=0 obtained by complete acquisition as the auxiliary input image, and the DNN reconstructed image of T1W as the output image to be processed. Next, the processing circuit 11 inputs the FFT reconstructed image of T1W obtained by decimation into the processing range 921 of the input layer 91 of the trained DNN (ForT1W), and inputs the FFT reconstructed image of b-value=0 obtained by complete acquisition into the auxiliary range 922. Then, the processing circuit 11 forward propagates the FFT reconstructed image of T1W obtained by decimation and the FFT reconstructed image of b-value=0 obtained by complete acquisition to the trained DNN (ForT1W). As a result, a DNN reconstructed image of T1W with reduced image quality degradation due to decimation is generated.

[0165] As described above, according to Application Example 1, the target input image and auxiliary input image are selected from a series of MR images collected by performing a magnetic resonance imaging (MRI) examination. Specifically, the first step of the examination is complete acquisition, and subsequent steps are decimal acquisition. The MR image based on complete acquisition is used as the auxiliary input image, and each MR image based on decimal acquisition is used as the target input image for forward propagation of the trained DNN. This makes it possible to generate and display high-resolution MR images based on decimal acquisition while shortening the examination time. Furthermore, since it is possible to generate high-quality MR images even with decimal acquisition, it is possible to further increase the decimation rate of k-space data and further shorten the examination time. In addition, according to Application Example 1, image quality degradation specific to each acquisition sequence, such as image quality degradation due to imaging with a relatively large b value, can also be reduced by a trained DNN using two input data: main input data and auxiliary input data.

[0166] Next, we will discuss the application of magnetic resonance imaging to motion imaging. In magnetic resonance imaging, motion imaging can be performed with any type of k-space filling trajectory, but in the following example, we will assume motion imaging based on a radial scan, which has relatively good temporal resolution. MR images may be two-dimensional or three-dimensional. For motion imaging of three-dimensional images based on a radial scan, for example, three-dimensional radial scans and stacks of stars can be applied.

[0167] The processing circuit 11, through the implementation of the imaging control function 111, controls the sequence control circuit 29-1 to perform video imaging on the subject P. During video imaging, the processing circuit 11 collects k-space data of multiple time-series frames. Through the implementation of the normal reconstruction function 112, the processing circuit 11 immediately generates MR images of multiple time-series frames based on the k-space data of multiple time-series frames. Any method can be used to reconstruct the MR images, but to improve the responsiveness of MR image generation, a reconstruction method with a short processing time is preferable. Examples of such reconstruction methods include the Jackson method, the gridding method, and the AUTOMAP method (Bo Zhu et al. Nature 22 March 2018, doi:10.1038 / nature25988). In radial scanning, sample points are arranged at non-equal intervals in the k-space data. The processing circuit 11 reconstructs the non-equal-interval k-space data into equally-interval k-space data using the Jackson method. The processing circuit 11 then applies FFT to the equally spaced k-space data to generate an MR image.

[0168] The implementation of the input selection function 113 allows the processing circuit 11 to select the input image to be processed and the auxiliary input image from multiple frames of MR images in a time series. The k-space data or MR images in multiple frames of a time series have common parameters such as slice position, k-space filling trajectory type (radial scan), and time resolution, while the individual parameters are the acquisition time or frame. For example, the processing circuit 11 selects the latest frame's MR image as the input image to be processed and an MR image from a predetermined frame prior to the latest frame as the auxiliary input image. The predetermined frame may be one frame prior to the latest frame, or two or more frames prior, and is not particularly limited.

[0169] With the implementation of the forward propagation function 114, the processing circuit 11 inputs the selected input image to be processed (MR image of the latest frame) and the auxiliary input image (MR image of a past frame) to the trained DNN 90 for image denoising, and immediately outputs the output image to be processed (DNN reconstructed image of the latest frame). The DNN reconstructed image of the latest frame is displayed on the display 19 with the implementation of the display control function 116. The processing circuit 11 executes the input selection function 113 and the forward propagation function 114 each time the MR image of the latest frame is generated, generates the DNN reconstructed image of the latest frame, and displays it on the display 19 with the implementation of the display control function 116. This makes it possible to display the DNN reconstructed image as a video in real time.

[0170] Although the processing circuit 11 uses a pre-trained DNN90 for image denoising, it may also use a pre-trained DNN90 for k-space data denoising. In this case, the processing circuit 11 selects the target input k-space data and auxiliary input k-space data from the k-space data of multiple time-series frames by implementing the input selection function 113. By implementing the forward propagation function 114, the processing circuit 11 inputs the selected target input k-space data (k-space data of the latest frame) and auxiliary input k-space data (k-space data of past frames) to the pre-trained DNN90 for k-space data denoising, and immediately outputs the target output k-space data (DNN k-space data of the latest frame). Subsequently, the processing circuit 11 generates a DNN-reconstructed image by applying an FFT or the like to the DNN k-space data of the latest frame by implementing the normal reconstruction function 112, and immediately displays the DNN-reconstructed image as a video by implementing the display control function 116.

[0171] Application Example 2: X-ray computed tomography (X-ray) scan X-ray computed tomography (X-ray) imaging is performed using an X-ray computed tomography (X-ray) scanner. In Application Example 2, medical imaging device 3 is assumed to be the CT stand of an X-ray computed tomography scanner.

[0172] Figure 19 shows the configuration of the X-ray computed tomography (X-ray) scanner 9-2. Although multiple CT stands 3-2 are depicted in Figure 19 for illustrative purposes, typically the X-ray computed tomography scanner 9-2 is equipped with only one stand 3-2.

[0173] As shown in Figure 19, the X-ray computed tomography (XCT) scanner 9-2 comprises a stand 3-2, a patient table 30-2, and a medical data processing unit (console) 1-2. The stand 3-2 is a scanning device configured for X-ray CT imaging of a subject P. The patient table 30-2 is a transport device for placing and positioning the subject P to be X-ray CT imaging. The medical data processing unit 1-2 is a computer that controls the stand 3-2. For example, the stand 3-2 and patient table 30-2 are installed in the examination room, and the medical data processing unit 1-2 is installed in a control room adjacent to the examination room. The stand 3-2, patient table 30-2, and medical data processing unit 1-2 are connected to each other by wired or wireless means so that they can communicate with one another.

[0174] As shown in Figure 19, the rig 3-2 includes an X-ray tube 21-2, an X-ray detector 12-2, a rotating frame 13-2, an X-ray high-voltage device 24-2, a control device 25-2, a wedge filter 26-2, a collimator 27-2, and a data acquisition circuit (DAS: Data Acquisition System) 28-2.

[0175] The X-ray tube 21-2 generates X-rays. Specifically, the X-ray tube 21-2 includes a cathode that generates thermionic electrons, an anode that receives thermionic electrons flying from the cathode and generates X-rays, and a vacuum tube that holds the cathode and anode. The X-ray tube 21-2 is connected to the X-ray high-voltage device 24-2 via a high-voltage cable. The X-ray high-voltage device 24-2 supplies a filament current to the cathode. The supply of filament current generates thermionic electrons from the cathode. The X-ray high-voltage device 24-2 applies a tube voltage between the cathode and anode. The application of the tube voltage causes thermionic electrons to fly from the cathode towards the anode, collide with the anode, and generate X-rays. The generated X-rays are irradiated onto the subject P. A tube current flows as thermionic electrons fly from the cathode towards the anode.

[0176] The X-ray detector 22-2 detects X-rays generated from the X-ray tube 21-2 and passing through the subject P, and outputs an electrical signal corresponding to the detected X-ray dose to the DAS 28-2. The X-ray detector 22-2 has a structure in which multiple rows of X-ray detection elements, each arranged in the channel direction, are arranged in the slice direction (column direction). The X-ray detector 22-2 is an indirect conversion type detector having, for example, a grid, a scintillator array, and a photosensor array. The scintillator array has multiple scintillators. The scintillators output light with an amount of light corresponding to the incident X-ray dose. The grid is positioned on the X-ray incident surface side of the scintillator array and has an X-ray shielding plate that absorbs scattered X-rays. The grid is sometimes called a collimator (one-dimensional collimator or two-dimensional collimator). The photosensor array converts the amount of light from the scintillators into an electrical signal. For example, a photodiode is used as the photosensor.

[0177] The rotating frame 23-2 is an annular frame that supports the X-ray tube 21-2 and the X-ray detector 22-2 so that they can rotate around the rotation axis Z. Specifically, the rotating frame 23-2 supports the X-ray tube 21-2 and the X-ray detector 22-2 in opposition to each other. The rotating frame 23-2 is supported by a fixed frame (not shown) so that it can rotate around the rotation axis Z. The control device 25-2 rotates the rotating frame 23-2 around the rotation axis Z, thereby rotating the X-ray tube 21-2 and the X-ray detector 22-2 around the rotation axis Z. The opening 29-2 of the rotating frame 23-2 is set to define the field of view (FOV).

[0178] In this embodiment, the rotation axis of the rotating frame 23-2 or the longitudinal direction of the top plate 33-2 of the bed 30-2 in the non-tilted state is defined as the Z direction, the direction perpendicular to the Z direction and horizontal to the floor surface is defined as the X direction, and the direction perpendicular to the Z direction and perpendicular to the floor surface is defined as the Y direction.

[0179] The X-ray high-voltage device 24-2 comprises a high-voltage generator and an X-ray control device. The high-voltage generator has an electrical circuit including a transformer and a rectifier, and generates the high voltage applied to the X-ray tube 21-2 and the filament current supplied to the X-ray tube 21-2. The X-ray control device controls the high voltage applied to the X-ray tube 21-2 and the filament current supplied to the X-ray tube 21-2. The high-voltage generator may be of the transformer type or the inverter type. The X-ray high-voltage device 24-2 may be installed on the rotating frame 23-2 within the stand 3-2, or on the fixed frame (not shown) within the stand 3-2.

[0180] The wedge filter 26-2 adjusts the dose of X-rays irradiated onto the subject P. Specifically, the wedge 26-2 attenuates the X-rays so that the dose of X-rays irradiated from the X-ray tube 11 to the subject P has a predetermined distribution. For example, the wedge 26-2 is a metal filter formed by processing a metal such as aluminum. The wedge filter 26-2 is processed to have a predetermined target angle and a predetermined thickness. The wedge filter 26-2 is also called a bow-tie filter.

[0181] The collimator 27-2 limits the irradiation range of X-rays that have passed through the wedge filter 26-2. The collimator 27-2 slidably supports multiple lead plates that shield the X-rays and adjusts the shape of the slit formed by the multiple lead plates. The collimator 27-2 is also called an X-ray diaphragm.

[0182] DAS28-2 reads an electrical signal corresponding to the X-ray dose detected by the X-ray detector 22-2 from the X-ray detector 22-2, amplifies the read electrical signal, and integrates the electrical signal over the viewing period to collect detection data having a digital value corresponding to the X-ray dose over the viewing period. The detection data is also called projection data. DAS28-2 is implemented, for example, by an ASIC equipped with circuit elements capable of generating projection data. The projection data (detection data) generated by DAS28-2 is transmitted by optical communication from a transmitter having a light-emitting diode (LED) provided on the rotating frame 13 to a receiver having a light-emitting diode (LED) provided on the non-rotating part (e.g., the fixed frame) of the pedestrian stand 3-2, and then transmitted from the receiver to the medical data processing device 3-2. Note that the method of transmitting projection data from the rotating frame 23-2 to the non-rotating part of the pedestrian stand 3-2 is not limited to the aforementioned optical communication, but any non-contact data transmission method may be used.

[0183] The examination bed 30-2 comprises a base 35-2 and a top plate 33-2. The base 35-2 is installed on the floor. The base 35-2 is a structure that supports a support frame so that it can move perpendicular to the floor (Y direction). The support frame is a frame provided on the upper part of the base 35-2. The support frame supports the top plate 33-2 so that it can slide along the central axis Z. The top plate 33-2 is a flexible plate-like structure on which the subject P is placed. The examination bed drive device is housed in the examination bed 30-2. The examination bed drive device is a motor or actuator that generates power to move the top plate 33-2 on which the subject P is placed. The examination bed drive device operates according to control by a control device 25-2 or a medical data processing device 1-2, etc.

[0184] The control device 25-2 controls the X-ray high-voltage device 24-2, DAS 28-2, and patient table 30-2 in order to perform X-ray CT imaging according to the imaging control function 111 of the processing circuit 11. The control device 25-2 has a processing circuit having a CPU, etc., and drive devices such as motors and actuators. The processing circuit has a processor such as a CPU and memory such as ROM or RAM as hardware resources. The control device 25-2 controls the cradle 3-2 and patient table 30-2 according to operation signals from input interfaces 15 provided on, for example, the medical data processing device 1-2, the cradle 3-2, and the patient table 30-2. For example, the control device 25-2 controls the rotation of the rotating frame 23-2, the tilt of the cradle 3-2, and the operation of the tabletop 33-2 and patient table 30-2.

[0185] The medical data processing device (console) 1-2 is a computer having a processing circuit 11, memory 13, input interface 15, communication interface 17, and display 19. Since the medical data processing device 1-2 is equivalent to the medical data processing device 1 described above, its description is omitted.

[0186] Figure 20 schematically shows the processing of the medical data processing device 1-2 according to Application Example 2. The examination shown in Figure 20 is a time-series imaging procedure in which X-ray CT imaging is repeated over multiple cycles. In the first cycle, X-ray CT imaging is performed using a high dose of X-rays (step SQ1). The processing circuit 11 applies FBP to the projection data collected in step SQ1 to generate an FBP reconstructed image for the first cycle.

[0187] As shown in Figure 20, from the second pass onward, X-ray CT imaging with low-dose X-rays is repeatedly performed (steps SQ2 and SQ3). When projection data for the second pass is collected, the processing circuit 11 applies FBP to the projection data to generate an FBP reconstructed image for the second pass. Low-dose X-ray CT imaging may be performed by reducing the tube current compared to the first pass, by intermittently exposing the X-rays, by rotating the rotating frame at high speed, or by a combination of at least one of the aforementioned low tube current, intermittent X-ray exposure, and high-speed rotation. Since the FBP reconstructed images from the second pass onward are collected by low-dose X-ray CT imaging, they contain a degradation in image quality compared to the FBP reconstructed image of the first pass.

[0188] If an FBP reconstruction image for the second pass is generated, the processing circuit 11 executes the input selection function 113. As shown in Figure 20, in the input selection function 113, the processing circuit 11 selects the FBP reconstruction image for the first pass at a high dose as the auxiliary input image, and selects the FBP reconstruction image for the second pass at a low dose as the input image to be processed.

[0189] Next, the processing circuit 11 executes the forward propagation function 114. In the forward propagation function 114, the processing circuit 11 first reads a trained DNN (For Low Dose) from the memory 13, which uses the low-dose FBP reconstructed image as the input image to be processed, the high-dose FBP reconstructed image as the auxiliary input image, and the high-quality FBP reconstructed image as the output image to be processed. Next, the processing circuit 11 inputs the low-dose FBP reconstructed image for the second pass into the processing range 921 of the input layer 91 of the trained DNN (For Low Dose), and inputs the high-dose FBP reconstructed image for the first pass into the auxiliary range 922. Then, the processing circuit 11 forward propagates the low-dose FBP reconstructed image for the second pass and the high-dose FBP reconstructed image for the first pass to the trained DNN (For Low Dose). As a result, a DNN reconstructed image for the second pass is generated with reduced image quality degradation due to low dose.

[0190] Similarly, when projection data for the third pass is collected, the processing circuit 11 applies FBP to the projection data to generate an FBP reconstructed image for the third pass. Next, the processing circuit 11 executes the input selection function 113. As shown in Figure 20, in the input selection function 113, the processing circuit 11 selects the FBP reconstructed image for the first pass at high dose as the auxiliary input image, and selects the FBP reconstructed image for the third pass at low dose as the input image to be processed.

[0191] Next, the processing circuit 11 executes the forward propagation function 114. In the forward propagation function 114, the processing circuit 11 inputs the FBP reconstructed image for the third pass at low dose to the processing range 921 of the input layer 91 of the trained DNN, and inputs the FBP reconstructed image for the first pass at high dose to the auxiliary range 922 of the input layer 91 of the trained DNN. The processing circuit 11 then forward propagates the FBP reconstructed image for the third pass at low dose and the FBP reconstructed image for the first pass at high dose to the trained DNN. As a result, a DNN reconstructed image for the third pass is generated with reduced image quality degradation due to low dose.

[0192] As described above, according to Application Example 2, the input image to be processed and the auxiliary input image are selected from a series of CT images collected during an X-ray computed tomography (X-ray) examination. Specifically, high-dose X-ray CT imaging is performed in the first pass, and low-dose X-ray CT imaging is performed in the second pass and beyond. The high-dose CT image is used as the auxiliary input image, and each low-dose CT image is used as the input image to be processed, and forward propagation of the trained DNN is performed. This makes it possible to generate and display CT images with less image quality degradation associated with low doses while reducing the radiation dose. Furthermore, since high-quality CT images can be generated even with low-dose X-ray CT imaging, it becomes possible to further reduce the X-ray dose.

[0193] Application Example 3: PET / CT Scan PET / CT scans are performed using a PET / CT scanner. In Application Example 3, the medical imaging device 3 is assumed to be the stand for the PET / CT scanner. In this case, the medical imaging device 3 is equipped with a CT stand for performing X-ray CT imaging and a PET stand for performing PET imaging.

[0194] Figure 21 shows the configuration of the PET / CT system 9-3. As shown in Figure 21, the PET / CT system 9-3 includes a PET gantry 20-3, a CT gantry 30-3, a patient table 40-3, and a medical data processing unit 1-3. Typically, the PET gantry 20-3, CT gantry 30-3, and patient table 40-3 are installed in a common examination room, while the medical data processing unit 1-3 is installed in a control room adjacent to the examination room. The PET gantry 20-3 is an imaging device that performs PET imaging of the subject P. The CT gantry 30-3 is an imaging device that performs X-ray CT imaging of the subject P. The patient table 40-3 movably supports a tabletop 43-3 on which the subject P to be imaged is placed. The medical data processing unit 1-3 is a computer that controls the PET gantry 10, CT gantry 30, and patient table 50.

[0195] As shown in Figure 21, the PET gantry 20-3 includes a detector ring 21-3, a signal processing circuit 23-3, and a coincidence counting circuit 25-3.

[0196] The detector ring 21-3 has a plurality of gamma-ray detectors 27-3 arranged on a circumference around the central axis Z. An image field of view (FOV) is set at the opening of the detector ring 21-3. The subject P is positioned so that the imaging area of ​​the subject P is included in the image field of view. The subject P is administered a drug labeled with a positron-emitting nuclide. The positron emitted from the positron-emitting nuclide annihilates with surrounding electrons, generating a pair of annihilation gamma rays. The gamma-ray detectors 27-3 detect the annihilation gamma rays emitted from the body of the subject P and generate an electrical signal corresponding to the light intensity of the detected annihilation gamma rays. For example, the gamma-ray detector 27-3 has a plurality of scintillators and a plurality of photomultiplier tubes. The scintillators receive annihilation gamma rays originating from radioactive isotopes in the subject P and generate light. The photomultiplier tubes generate an electrical signal corresponding to the light intensity. The generated electrical signal is supplied to the signal processing circuit 23-3.

[0197] The signal processing circuit 23-3 generates single-event data based on the electrical signal from the gamma-ray detector 27-3. Specifically, the signal processing circuit 23-3 performs detection time measurement processing, position calculation processing, and energy calculation processing. The signal processing circuit 23-3 is implemented by a processor configured to perform detection time measurement processing, position calculation processing, and energy calculation processing.

[0198] In the detection time measurement process, the signal processing circuit 23-3 measures the detection time of gamma rays by the gamma-ray detector 27-3. Specifically, the signal processing circuit 23-3 monitors the pulse height of the electrical signal from the gamma-ray detector 27-3 and measures the time when the pulse height exceeds a preset threshold as the detection time. In other words, the signal processing circuit 23-3 electrically detects annihilation gamma rays by detecting that the pulse height exceeds the threshold. In the position calculation process, the signal processing circuit 23-3 calculates the incident position of the annihilation gamma ray based on the electrical signal from the gamma-ray detector 27-3. The incident position of the annihilation gamma ray corresponds to the position coordinates of the scintillator into which the annihilation gamma ray was incident. In the energy calculation process, the signal processing circuit 23-3 calculates the energy value of the detected annihilation gamma ray based on the electrical signal from the gamma-ray detector 17. The detection time data, position coordinate data, and energy value data for a single event are associated. The combination of energy value data, position coordinate data, and detection time data for a single event is called single-event data. Single-event data is generated successively each time an extinction gamma ray is detected. The generated single-event data is supplied to the coincidence counting circuit 25-3.

[0199] The coincidence counting circuit 25-3 performs coincidence counting on single event data from the signal processing circuit 23-3. In terms of hardware resources, the coincidence counting circuit 25-3 is implemented by a processor configured to perform coincidence counting. During coincidence counting, the coincidence counting circuit 25-3 repeatedly identifies single event data for two single events that fall within a predetermined time frame from the repeatedly supplied single event data. This pair of single events is presumed to originate from annihilation gamma rays generated from the same annihilation point. The pair of single events is collectively called a coincidence count event. The line connecting the pair of gamma-ray detectors 27-3 (more specifically, scintillators) that detected these annihilation gamma rays is called the LOR (line of response). The event data for the pair of events constituting the LOR is called coincidence count event data. The coincidence count event data and single event data are transmitted to the medical data processing device 1-3. When there is no particular distinction between coincidence count event data and single event data, they will be referred to as PET event data.

[0200] In the above configuration, the signal processing circuit 23-3 and the same-counting circuit 25-3 are assumed to be included in the PET gantry 20-3, but this is not limited to that configuration. For example, the same-counting circuit 25-3, or both the signal processing circuit 23-3 and the same-counting circuit 25-3, may be included in a separate device from the PET gantry 20-3. Furthermore, one same-counting circuit 25-3 may be provided for each of the multiple signal processing circuits 23-3 mounted on the PET gantry 20-3, or the multiple signal processing circuits 23-3 mounted on the PET gantry 20-3 may be divided into multiple groups, and one same-counting circuit 25-3 may be provided for each group.

[0201] As shown in Figure 21, the CT gantry 30-3 includes an X-ray tube 31-3, an X-ray detector 32-3, a rotating frame 33-3, an X-ray high-voltage device 34-3, a CT control device 35-3, a wedge filter 36-3, a collimator 37-3, and a DAS 38-3.

[0202] The X-ray tube 31-3 generates X-rays. Specifically, the X-ray tube 31-3 includes a vacuum tube that holds a cathode which generates thermionic electrons and an anode which receives thermionic electrons flying from the cathode and generates X-rays. The X-ray tube 31-3 is connected to the X-ray high-voltage device 34-3 via a high-voltage cable. The tube voltage is applied between the cathode and the anode by the X-ray high-voltage device 34-3. The application of the tube voltage causes thermionic electrons to fly from the cathode to the anode. The flight of thermionic electrons from the cathode to the anode causes a tube current to flow. The application of high voltage from the X-ray high-voltage device 34-3 and the supply of filament current cause thermionic electrons to fly from the cathode to the anode, and X-rays are generated when the thermionic electrons collide with the anode.

[0203] The X-ray detector 32-3 detects X-rays generated from the X-ray tube 31-3 and passing through the subject P, and outputs an electrical signal corresponding to the detected X-ray dose to the DAS 38-3. The X-ray detector 32-3 has a structure in which multiple rows of X-ray detection elements, each arranged in the channel direction, are arranged in the slice direction (column direction or row direction). The X-ray detector 32-3 is, for example, an indirect conversion type detector having a grid, a scintillator array, and a photosensor array. The scintillator array has multiple scintillators. The scintillators output light with an intensity corresponding to the incident X-ray dose. The grid is positioned on the X-ray incident surface side of the scintillator array and has an X-ray shielding plate that absorbs scattered X-rays. The photosensor array converts the light from the scintillators into an electrical signal corresponding to the intensity of the light. For example, a photodiode or a photomultiplier tube can be used as the photosensor. The X-ray detector 32 may also be a direct conversion type detector (semiconductor detector) having semiconductor elements that convert incident X-rays into electrical signals.

[0204] The rotating frame 33-3 is an annular frame that supports the X-ray tube 31-3 and the X-ray detector 32-3 so that they can rotate around the rotation axis Z. Specifically, the rotating frame 33-3 supports the X-ray tube 31-3 and the X-ray detector 32-3 in opposition to each other. The rotating frame 33-3 is supported by a fixed frame (not shown) so that it can rotate around the rotation axis Z. The CT control device 35-3 rotates the rotating frame 33-3 around the rotation axis Z, thereby rotating the X-ray tube 31-3 and the X-ray detector 32-3 around the rotation axis Z. The rotating frame 33-3 rotates at a constant angular velocity around the rotation axis Z, powered by the drive mechanism of the CT control device 35-3. An image field of view (FOV) is set at the opening of the rotating frame 33-3.

[0205] In this embodiment, the rotation axis of the rotating frame 33-3 or the longitudinal direction of the top plate 43-3 of the bed 40-3 in the non-tilted state is defined as the Z-axis direction, the axis perpendicular to the Z-axis direction and horizontal to the floor surface is defined as the X-axis direction, the axis perpendicular to the Z-axis direction and perpendicular to the floor surface is defined as the Y-axis direction.

[0206] The X-ray high-voltage device 34-3 has an electrical circuit including a transformer and a rectifier, and includes a high-voltage generator that generates a high voltage to be applied to the X-ray tube 31-3 and a filament current to be supplied to the X-ray tube 31-3, and an X-ray control device that controls the output voltage according to the X-rays irradiated by the X-ray tube 31-3. The high-voltage generator may be of the transformer type or the inverter type. The X-ray high-voltage device 34-3 may be installed on the rotating frame 33-3 or on a fixed frame (not shown) inside the CT gantry 30-3.

[0207] The wedge filter 36-3 adjusts the dose of X-rays irradiated onto the subject P. Specifically, the wedge filter 36-3 attenuates the X-rays so that the dose of X-rays irradiated from the X-ray tube 31-3 to the subject P has a predetermined distribution. For example, a metal plate such as aluminum is used as the wedge filter 36-3. The wedge filter 36-3 is also called a bowtie filter.

[0208] The collimator 37-3 limits the irradiation range of X-rays that have passed through the wedge filter 36-3. The collimator 37-3 slidably supports multiple lead plates that shield the X-rays and adjusts the shape of the slit formed by the multiple lead plates.

[0209] DAS38-3 reads an electrical signal corresponding to the X-ray dose detected by the X-ray detector 32-3 from the X-ray detector 32, amplifies the read electrical signal with a variable amplification factor, and integrates the electrical signal over the viewing period to collect raw CT data having a digital value corresponding to the X-ray dose over that viewing period. DAS38-3 is implemented, for example, by an ASIC equipped with circuit elements capable of generating raw CT data. The raw CT data is transmitted to the medical data processing device 1-3 via a non-contact data transmission device or the like.

[0210] The CT control device 35-3 controls the X-ray high-voltage device 34-3 and DAS 38-3 to perform X-ray CT imaging according to the imaging control function 111 of the processing circuit 11 of the medical data processing device 1-3. The CT control device 35-3 has a processing circuit with a CPU and the like, and a drive mechanism with a motor and actuator. The processing circuit 11 has a processor such as a CPU or MPU and memory such as ROM or RAM as hardware resources.

[0211] Furthermore, the CT gantry 30-3 comes in various types, including Rotate / Rotate-Type (third-generation CT) in which the X-ray generation unit and X-ray detection unit rotate together around the subject, and Stationary / Rotate-Type (fourth-generation CT) in which a large number of X-ray detection elements are fixed in a ring-shaped array and only the X-ray generation unit rotates around the subject. Any of these types can be applied to one embodiment.

[0212] As shown in Figure 21, the patient table 40-3 is used to place the subject P to be scanned and to move the placed subject. The patient table 40-3 is shared by the PET gantry 20-3 and the CT gantry 30-3.

[0213] The bed 40-3 comprises a base 45-3 and a top plate 43-3. The base 45-3 is installed on the floor. The base 45-3 is a housing that supports a support frame so that it can move perpendicular to the floor (in the Y-axis direction). The support frame is a frame provided on the upper part of the base 45-3. The support frame supports the top plate 43-3 so that it can slide along the central axis Z. The top plate 43-3 is a flexible plate on which the subject P is placed. The bed drive device is housed within the housing of the bed 40-3. The bed drive device is a motor or actuator that generates power to move the support frame on which the subject P is placed and the top plate 43-3. The bed drive device operates according to control by a medical data processing device 1-3, etc.

[0214] The PET gantry 20-3 and CT gantry 30-3 are positioned such that the central axis Z of the opening of the PET gantry 20-3 and the central axis Z of the opening of the CT gantry 30-3 are approximately coincident. The bed 40-3 is positioned so that the long axis of the top plate 43-3 is parallel to the central axis Z of the openings of the PET gantry 20-3 and the CT gantry 30-3. The CT gantry 30-3 and PET gantry 20-3 are installed in that order, starting from the bed 40-3.

[0215] The medical data processing device (console) 1-3 is a computer having a processing circuit 11, memory 13, input interface 15, communication interface 17, and display 19. Since the medical data processing device 1-3 is equivalent to the medical data processing device 1 described above, its description is omitted.

[0216] Figure 22 shows a typical processing flow of the medical data processing device 1 according to Application Example 3. Figure 23 is a schematic diagram showing the processing of the medical data processing device 1 according to Application Example 3.

[0217] As shown in Figures 22 and 23, the processing circuit 11 first executes the imaging control function 111 (step SC1). In step SC1, the processing circuit 11 controls the medical imaging device 3 to perform X-ray CT imaging on the subject. The projection data of the subject collected by the medical imaging device 3 is transmitted to the medical data processing device 1.

[0218] When step SC1 is performed, the processing circuit 11 executes the normal restoration function 112 (step SC2). In step SC2, the processing circuit 11 applies an analytical image reconstruction method such as FBP or a sequential approximation image reconstruction method to the projection data collected in step SC1 to reconstruct a CT image of the subject.

[0219] When step SC2 is performed, the processing circuit 11 executes the imaging control function 111 (step SC3). In step SC3, the processing circuit 11 controls the medical imaging device 3 to perform PET imaging on the subject. The coincidence data of the subject collected by the medical imaging device 3 is transmitted to the medical data processing device 1.

[0220] When step SC3 is performed, the processing circuit 11 executes the normal restoration function 112 (step SC4). In step SC4, the processing circuit 11 applies an analytical image reconstruction method such as FBP or a sequential approximation image reconstruction method to the coincidence data collected in step SC3 to reconstruct a PET image of the subject. The PET image has a degraded image quality compared to the CT image because the gamma ray detection efficiency by PET imaging is lower than the X-ray detection efficiency by X-ray CT imaging, etc.

[0221] When steps SC2 and SC4 are performed, the processing circuit 11 executes the image processing function 115 (step SC5). In step SC5, the processing circuit 11 aligns the CT image reconstructed in step SB2 and the PET image reconstructed in step SC4. The alignment may be performed by any method such as rigid body alignment or non-rigid body alignment.

[0222] When step SC5 is performed, the processing circuit 11 executes the input selection function 113 (step SC6). In step SC6, the processing circuit 11 selects the PET image aligned in step SC5 as the processing target input image and selects the CT image aligned in step SC5 as the auxiliary input image (step SC6).

[0223] When step SCB6 is performed, the processing circuit 11 executes the forward propagation function 114 (step SC7). In step SC7, the processing circuit 11 applies the trained DNN to the PET image and the CT image to generate a PET-derived DNN-reconstructed image. Specifically, in the forward propagation function 114, the processing circuit 11 first reads a trained DNN (ForPET) from memory 13, which uses the PET image as the input image to be processed, the CT image as the auxiliary input image, and the PET image with data loss removed as the output image to be processed. Next, the processing circuit 11 inputs the PET image to the processing range 921 of the input layer 91 of the trained DNN (ForPET) and the CT image to the auxiliary range 922. Then, the processing circuit 11 forward propagates the input PET image and CT image to the trained DNN (ForPET). This generates a PET-derived DNN-reconstructed image.

[0224] When step SC7 is performed, the processing circuit 11 executes the display control function 116 (step SC8). In step SC8, the processing circuit 11 displays the PET-derived DNN reconstructed image generated in step SC7 and the CT image aligned in step SC5 on the display 19. For example, the processing circuit 11 displays a composite image of the PET / CT-derived DNN reconstructed image and the CT image.

[0225] This concludes the explanation of the processing performed by the medical data processing device 1-3 in Application Example 3.

[0226] Note that the processing flow shown in Figure 22 is just one example, and the processing related to Application Example 3 is not limited to the flow shown in Figure 22. For example, the order of X-ray CT imaging in step SC1 and PET imaging in step SC3 may be reversed.

[0227] Furthermore, the order of the alignment process in step SC5 and the selection process in step SC6 may be reversed. Also, although the alignment process in step SC5 and the forward propagation process in step SC7 are described as separate processes, the alignment process in step SC5 may be incorporated into the trained DNN (ForPET).

[0228] As described above, according to Application Example 3, the target input image and auxiliary input image are selected from the PET and CT images collected during the execution of a PET / CT examination. A relatively high-resolution CT image is used as the auxiliary input image, and a relatively low-resolution PET image is used as the target input image, and the trained DNN is forward-propagated. This makes it possible to generate and display high-resolution PET images using CT images.

[0229] Application Example 4: Ultrasound Examination Ultrasound examinations are performed using an ultrasound diagnostic device. In Application Example 4, the medical imaging device 3 is assumed to be the ultrasound probe of the ultrasound diagnostic device.

[0230] Figure 24 shows the configuration of the ultrasound diagnostic device 9-4 according to application example 4. As shown in Figure 24, the ultrasound diagnostic device 9-4 includes an ultrasound probe 3-4 and a medical data processing device (main unit) 1-4.

[0231] The ultrasonic probe 3-4 performs an ultrasonic scan on a scan area within a patient or other living body, for example, according to the imaging control function 111 of the medical data processing device 1-4. The ultrasonic probe 3-4 has, for example, a plurality of piezoelectric transducers, a matching layer, and a backing material. In this embodiment, the ultrasonic probe 3-4 has, for example, a plurality of ultrasonic transducers arranged along a predetermined direction. The ultrasonic probe 3-4 is detachably connected to the device body 1-4.

[0232] Multiple piezoelectric transducers generate ultrasound according to drive signals supplied from the ultrasound transmission circuit 31-4 of the medical data processing device 1-4. This transmits ultrasound from the ultrasound probe 3-4 to the living body. Once ultrasound is transmitted from the ultrasound probe 3-4 to the living body, it is reflected one after another by discontinuities in acoustic impedance within the body's internal tissues and received as reflected wave signals by the multiple piezoelectric transducers. The amplitude of the received reflected wave signals depends on the difference in acoustic impedance at the discontinuities where the ultrasound is reflected. Furthermore, when the transmitted ultrasound pulse is reflected by a moving blood flow or radiation-absorbing tissue spacer, the reflected wave signal undergoes a frequency shift due to the Doppler effect, depending on the velocity component of the ultrasound transmission direction of the moving object. The ultrasound probe 3-4 receives the reflected wave signals from the living body and converts them into electrical signals. These electrical signals are supplied to the medical data processing device 1-4.

[0233] The medical data processing device 1-4 shown in Figure 24 is a computer that generates and displays ultrasound images based on reflected wave signals received by an ultrasound probe 3-4. As shown in Figure 24, the medical data processing device 1-4 includes an ultrasound transmission circuit 31-4, an ultrasound reception circuit 32-4, a processing circuit 11, a memory 13, an input interface 15, a communication interface 17, and a display 19.

[0234] The ultrasonic transmitting circuit 31-4 is a processor that supplies drive signals to the ultrasonic probe 3-4. The ultrasonic transmitting circuit 31-4 is implemented, for example, by a trigger generation circuit, a delay circuit, and a pulser circuit. The trigger generation circuit repeatedly generates rate pulses at a predetermined rate frequency to form the transmitted ultrasonic waves. The delay circuit provides a delay time for each piezoelectric transducer necessary to focus the ultrasonic waves generated from the ultrasonic probe 3-4 into a beam and determine the transmission directivity, to each rate pulse generated by the trigger generation circuit. The pulser circuit applies drive signals (drive pulses) to the multiple ultrasonic transducers provided on the ultrasonic probe 3-4 at timings based on the rate pulses. By arbitrarily changing the delay time provided to each rate pulse by the delay circuit, the transmission direction from the piezoelectric transducer surface can be arbitrarily adjusted.

[0235] The ultrasonic receiving circuit 32-4 is a processor that performs various processes on the reflected wave signal received by the ultrasonic probe 3-4 to generate a received signal. The ultrasonic receiving circuit 32-4 is implemented, for example, by an amplifier circuit, an A / D converter, a receiving delay circuit, and an adder. The amplifier circuit amplifies the reflected wave signal received by the ultrasonic probe 3-4 for each channel and performs gain correction processing. The A / D converter converts the gain-corrected reflected wave signal into a digital signal. The receiving delay circuit gives the digital signal the delay time necessary to determine the receiving directivity. The adder adds multiple digital signals to which the delay time has been given. The addition process of the adder generates a received signal in which the reflected component from the direction corresponding to the receiving directivity is emphasized.

[0236] The processing circuit 11 is, for example, a processor that functions as the central hub of the ultrasound diagnostic device 1-4. The processing circuit 11 realizes functions corresponding to the program by executing the program stored in the memory 13. The processing circuit 11 has, for example, an imaging control function 111, a normal restoration function 112, an input selection function 113, a forward propagation function 114, an image processing function 115, a display control function 116, a B-mode processing function 332, and a Doppler mode processing function 333.

[0237] In the B-mode processing function 332, the processing circuit 11 generates B-mode data based on the received signal from the ultrasonic receiving circuit 31-4. Specifically, the processing circuit 11 performs, for example, envelope detection and logarithmic amplification on the received signal from the ultrasonic receiving circuit 31-4 to generate data (B-mode data) in which the signal strength is expressed as brightness. The generated B-mode data is stored as B-mode RAW data on a two-dimensional ultrasonic scan line (raster) in a RAW data memory (not shown).

[0238] In the Doppler mode processing function 333, the processing circuit 11 performs frequency analysis on the received signal from the ultrasound receiving circuit 31-4 to generate data (Doppler data) that extracts motion information based on the Doppler effect of blood flow within the ROI (Region of Interest) set in the scan area. Specifically, the processing circuit 11 generates Doppler data that estimates the average velocity, mean variance, average power value, etc., as blood flow motion information for each of multiple sample points. The generated Doppler data is stored as Doppler RAW data on a two-dimensional ultrasound scan line in a RAW data memory (not shown).

[0239] The processing circuit 11 is capable of performing a color Doppler method called Color Flow Mapping (CFM) in its Doppler mode processing function 333. In the CFM method, ultrasound is transmitted and received multiple times on multiple scan lines. The processing circuit 11 applies an MTI (Moving Target Indicator) filter to the data sequence at the same location to suppress signals originating from stationary or slow-moving tissue (clutter signals) and extract signals originating from blood flow. The processing circuit 11 then estimates information such as blood flow velocity, dispersion, or power from the extracted signals.

[0240] The memory 13, input interface 15, communication interface 17, display 19, imaging control function 111, normal restoration function 112, input selection function 113, forward propagation function 114, image processing function 115, and display control function 116 will not be explained.

[0241] FIG. 25 is a diagram schematically showing the processing of the medical data processing apparatus 1-4 according to Application Example 4. In an ultrasonic examination, the processing circuit 11 generates ultrasonic images of a plurality of frames in time series. Specifically, the processing circuit 11 executes ultrasonic imaging via the ultrasonic probe 3-4 by means of the imaging control function 11, and collects echo data of a plurality of frames in time series in real time. Next, the processing circuit 11 executes the B-mode processing function 332 to generate B-mode data of a plurality of frames in time series based on the echo data of the plurality of frames in time series. Then, the processing circuit 11 executes the normal restoration function 112 to generate a B-mode image IU of a plurality of frames in time series based on the B-mode data of the plurality of frames in time series. The B-mode images IU of the plurality of frames in time series are dynamically displayed on the display 19.

[0242] As shown in FIG. 25, assume that the current frame is "t". At the current frame t, the B-mode image IUt is displayed on the display 19. Here, "n" in t-n indicates an integer number of the frame n frames before the current frame t. The B-mode image IU-n displayed on the display 19 is generated by the normal restoration function 112 of the processing circuit 11, for example, by means of a scan conversion technique. Therefore, it can be said that the image quality of the B-mode image IU-n is relatively rough.

[0243] When the frame that the user wants to observe in detail is displayed, the user presses the freeze button (still button) via the input interface 15. In FIG. 25, assume that the freeze button is pressed for the current frame t. Triggered by the pressing of the freeze button, the processing circuit 11 displays the B-mode image IUt of the current frame t as a still image on the display 19.

[0244] Furthermore, upon pressing the freeze button, the processing circuit 11 sequentially executes the input selection function 113 and the forward propagation function 114. In the input selection function 113, the processing circuit 11 selects the B-mode image IUt of the current frame t as the input image to be processed, and selects the B-mode image IUt-n of a frame tn that is earlier than the current frame t as the auxiliary input image. As the past B-mode image IUt-n, for example, the B-mode image IUt-1 of the most recent past frame t-1, which is presumed to have a similar morphology to the B-mode image IUt of the current frame t, may be selected. Alternatively, as the past B-mode image IUt-n, for example, a B-mode image of a past frame that belongs to the same respiratory or cardiac phase as the B-mode image IUt of the current frame t may be selected. Note that the past B-mode image to be selected may be a B-mode image from a future frame than the current frame t. Hereinafter, let's assume that the B-mode image IUt-1 of frame t-1 is selected as the auxiliary input image.

[0245] Next, the processing circuit 11 executes the forward propagation function 114. In the forward propagation function 114, the processing circuit 11 applies the trained DNN to the B-mode image (input image to be processed) IUt of the current frame t and the B-mode image (auxiliary input image) IUt-1 of the past frame t-1 to generate the DNN reconstructed image IU0 of the current frame t. Specifically, the processing circuit 11 first reads from the memory 13 a trained DNN (For ultrasound images) in which the B-mode image of the first frame is used as the input image to be processed, the B-mode image of the second frame is used as the auxiliary input image, and the B-mode image with data loss portions restored is used as the output image to be processed. Next, the processing circuit 11 inputs the B-mode image IUt of the current frame t into the processing range 921 of the input layer 91 of the trained DNN (For ultrasound images), and inputs the B-mode image IUt-1 of the past frame t-1 into the auxiliary range 922. The processing circuit 11 then forward-propagates the input B-mode image IUt and B-mode image IUt-1 to the trained DNN. This allows the generation of a B-mode image IU0 with data loss portions restored for the current frame t.

[0246] The B-mode image IU0 generated by the forward propagation function 114 is displayed as a still image on the display 19. The B-mode image IU0 may be displayed adjacent to the B-mode image IUt, or it may be displayed as a replacement for the B-mode image IUt.

[0247] When the freeze button is pressed again, the processing circuit 11 generates multiple frames of B-mode images in a time series by ultrasonic imaging of the subject P via the ultrasonic probe 30-4, and can display them on the display 10 in real time.

[0248] This concludes the explanation of the processing performed by the medical data processing device 1-4 in Application Example 4.

[0249] In the above description, the ultrasound image was assumed to be a B-mode image, but this embodiment is not limited to this. For example, the ultrasound image may be a Doppler-mode image.

[0250] As described above, according to Application Example 4, the target input image and auxiliary input image are selected from the ultrasound images collected during the ultrasound examination. The auxiliary input image is selected from ultrasound images that are morphologically similar to the target input image. This makes it possible to generate and display relatively high-quality ultrasound images immediately.

[0251] Application Example 5: X-ray Fluoroscopy X-ray fluoroscopy is performed using an X-ray diagnostic device. In Application Example 5, the medical imaging device 3 is assumed to be a C-arm to which an X-ray tube and an X-ray detector are attached. Note that Application Example 5 is equivalent to Application Example 4.

[0252] In X-ray fluoroscopy, the processing circuit 11 generates high-dose radiographic images and low-dose fluoroscopic images via the C-arm, and displays both the radiographic and fluoroscopic images on the display 19. The radiographic images are of higher quality than the fluoroscopic images. The radiographic and fluoroscopic images are displayed side by side, or a narrow-field fluoroscopic image is superimposed on a wide-field radiographic image. Radiographic images are typically still images, while fluoroscopic images are typically moving images. The radiographic images depict contrast-enhanced blood vessels in high quality, and therefore function as a roadmap for the contrast-enhanced blood vessels. The fluoroscopic images depict blood vessels in real time. The operator observes both the radiographic and fluoroscopic images while guiding surgical instruments such as catheters to the target site.

[0253] The user presses the freeze button via the input interface 15 when the perspective image of the frame they wish to examine in detail is displayed. Upon pressing the freeze button, the processing circuit 11 displays the perspective image of the current frame as a still image on the display 19.

[0254] Furthermore, when the freeze button is pressed, the processing circuit 11 sequentially executes the input selection function 113 and the feedforward function 114. In the input selection function 113, the processing circuit 11 selects the fluoroscopic image of the current frame as the input image to be processed, and selects a captured image of a past or future frame that meets predetermined criteria as an auxiliary input image. In the feedforward function 114, the processing circuit 11 reads a trained model (For X-ray image) from the memory 13, which uses the fluoroscopic image as the input image to be processed, the captured image as the auxiliary input image, and the DNN fluoroscopic image with data loss restored as the output image to be processed. Next, the processing circuit 11 inputs the fluoroscopic image of the current frame into the processing range 921 of the input layer 91 of the trained model (For X-ray image), and inputs a captured image of a past or future frame that meets predetermined criteria into the auxiliary range 922. Then, the processing circuit 11 feedforward the input fluoroscopic image and captured image to the trained DNN. This makes it possible to generate a DNN perspective image with data loss portions restored for the current frame t. The generated DNN perspective image is displayed as a still image on the display 19.

[0255] As described above, according to Application Example 5, an input image to be processed is selected from the fluoroscopic images collected during the execution of an X-ray fluoroscopy examination, and a high-quality image that is similar in form to the input image to be processed is selected as an auxiliary input image. This makes it possible to instantly generate and display high-quality X-ray images at a low dose.

[0256] (Example 1) The processing circuit 11 according to Example 1 reconstructs a video image from radial k-space data that has been undersampled with a high decimation rate. To obtain spatial correlation, temporal correlation, and a reduction in reconstruction time delay, a deep neural network is trained using an additional input image that displays the assisted correlation. The image quality obtained with the method of this example is higher than that of a normal DNN reconstruction with one input and one output.

[0257] Clinical applications of dynamic MRI require both high-speed data acquisition and low-latency reconstruction. Methods for achieving this include image reconstruction from k-space data undersampled with a high decimation rate, including stacks of stars trajectories (i.e., bundles of 2D golden angle radial trajectories), and time-constrained compressed sensing. However, evaluating these time constraints requires a large number of frames (e.g., all frames of a dynamic image). This method is not suitable for low-latency reconstruction. For low-latency reconstruction, single-image reconstruction methods are preferred. An example of such a method is the use of a DNN from a single input image to a single output image. However, because this method does not use temporal correlation, it requires a larger number of spokes compared to time-constrained methods.

[0258] Figure 26 is a schematic diagram showing the processing of the medical data processing device 1 according to Embodiment 1. As shown in Figure 26, the processing circuit 11 according to Embodiment 1 performs a dynamic reconstruction method using a DNN that takes M consecutive frames from neighboring slices of N slices as input and outputs a single reconstructed image in order to integrate spatial correlation, temporal correlation and reduction of reconstruction time delay. The processing circuit 11 according to the embodiment reconstructs the image frame by frame. Since each frame depends only on (M-1) / 2 future frames, the reconstruction delay time is reduced compared to methods based on compressed sensing. The method according to the embodiment can be considered an extension of a normal reconstruction method using a DNN. In a normal reconstruction method, M=N=1. The DNN according to the embodiment consists of multiple convolution layers, multiple ReLUs and multiple residual connections.

[0259] Figure 27 shows an overview of the DNN according to Example 1. As shown in Figure 27, as a preprocessing step, the processing circuit 11 applies a Fast Fourier Transform (FFT) in the bundle direction. For each 2D radial k-space frame, the processing circuit 11 reconstructs an initial image using an unequal-interval FFT (NUFFT: Non-Uniform FFT) and parallel imaging (PI). The initial image reconstruction method does not depend on spatial and temporal correlations with other images. Spatial and temporal correlations are integrated into the DNN. The hidden layers include a 3x3 kernel convolution layer and a ReLU. Residual connections are inserted for each pair of bundled hidden layers. The number of hidden layers is 10. The first layer and the final layer each consist only of a 1x1 kernel convolution layer. The number of channels in each hidden layer is 128.

[0260] A training dataset was collected from two volunteers. A dataset from one volunteer was collected for validation. All datasets contain Cartesian images with a matrix size of 256 × 160 × 16. In addition, the validation dataset contains actual k-space data using Stack of Stars trajectories. 21 spokes were assigned to each radial k-space frame. The number of input images was set to M=5 and N=3. Ten frames were generated from the Cartesian images by simulating radial sampling to generate the input for the DNN. The DNN was trained using Adam (Adaptive moment estimation) with an error function of mean squared error.

[0261] Figures 28, 29, 30, and 31 show the results using simulated radial data (21 spokes per frame). Figure 28 shows the reconstructed image using NUFFT+PI. Figure 29 shows the reconstructed image using the normal reconstruction method (M=N=1). Figure 30 shows the reconstructed image using the reconstruction method according to Example 1 (M=5 and N=3). Figure 31 shows the reconstructed image of Truth. Figures 32, 33, and 34 show the results using actual stack of stars data (21 spokes per frame). Figure 32 shows the reconstructed image using NUFFT+PI. Figure 33 shows the reconstructed image using the normal reconstruction method (M=N=1). Figure 34 shows the reconstructed image using the reconstruction method according to Example 1 (M=5 and N=3). The image quality of the images reconstructed using the M=5 and N=3 reconstruction method is clearly improved compared to the images using NUFFT+PI and the images using the M=N=1 reconstruction method. In particular, actual stack-of-stars data, by using spatial and temporal correlations, provide a clearer picture of the structure within the liver.

[0262] The method according to Example 1 can reconstruct moving images from 21 spokes per frame, which is significantly fewer than the 45 spokes used in projection reconstruction methods using DNNs. This result shows that the image quality from actual stack of stars data is lower than that from simulated radial data. To improve image quality, the sampling imperfections of the MRI system should be accounted for in the radial sampling simulation. Improving the simulation is a future challenge.

[0263] As described above, the method according to Example 1 integrates spatial and temporal correlations into the DNN. Unlike methods based on compressed sensing, the method according to Example 1 reconstructs the image frame by frame and can be used for low-latency reconstruction. Experimental results show that the method according to Example 1 is effective for reconstructing radial motion images that are undersampled with a high decimation rate.

[0264] (Example 2) The auxiliary input images in Example 2 are two or more images with different acquisition times and temporal resolutions for the same slice position. The medical data processing device 1 in Example 2 will now be described. The medical data processing device 1 in Example 2 can be implemented in any medical imaging diagnostic device capable of capturing multiple time-series frames (video imaging). However, in order to provide a concrete explanation below, the medical data processing device 1 will be assumed to be implemented in the magnetic resonance imaging device 9-1. In magnetic resonance imaging, video imaging can be performed with any type of k-space filling trajectory, but in the following example, it will be assumed to be video imaging based on a radial scan, which has relatively good temporal resolution. The MR image may be a two-dimensional image or a three-dimensional image. For example, 3D radial scanning and stack of stars can be applied as video imaging of a three-dimensional image based on a radial scan.

[0265] The processing circuit 11 according to Example 2 performs dense input when inputting the auxiliary input image to the DNN. Dense input is a method of inputting image data from multiple frames that are temporally surrounding the frame of the input image to be processed to the DNN with hierarchical temporal resolution. Dense input is expected to further improve the image quality of the output image to be processed.

[0266] Figure 35 is a schematic diagram illustrating dense input. In Figure 35, the horizontal axis is defined by time, and the vertical axis is defined by temporal resolution. Temporal resolution is expressed in levels. Level 1 is the temporal resolution corresponding to one imaging frame. One imaging frame is the time required to collect a predetermined amount of k-spatial data used for image reconstruction of one frame. For example, in a radial scan, if one frame has 10 spokes, one imaging frame is the time required to collect k-spatial data for 10 spokes. In Figure 35, the temporal resolution levels are set from level 1 to level 5. Level 2 is 1 / 2 the temporal resolution of level 1, level 3 is 1 / 3 the temporal resolution of level 1, level 4 is 1 / 4 the temporal resolution of level 1, and level 5 is 1 / 5 the temporal resolution of level 1. When dense input is applied to N imaging frames (level 1 frames), N(N+1) / 2 frames are generated. The number of frames for the nth level is N+1-2. Lower temporal resolution means more k-space data is included, resulting in better image quality.

[0267] Specifically, through the implementation of the imaging control function 111, the processing circuit 11 collects k-space data of frames FR1-1, FR1-2, FR1-3, FR1-4, and FR1-5 related to multiple time-series Level 1s. Through the implementation of the normal reconstruction function 11, MR images of frames FR1-1, FR1-2, FR1-3, FR1-4, and FR1-5 are generated based on the k-space data of multiple time-series frames FR1-1, FR1-2, FR1-3, FR1-4, and FR1-5.

[0268] For Level 2 processing, for example, the following is performed: Processing circuit 11 integrates the k-space data of frame FR1-1 and frame FR1-2, which are time-sequential, to generate the k-space data of frame FR2-1. Similarly, processing circuit 11 integrates the k-space data of frame FR1-2 and frame FR1-3, which are time-sequential, to generate the k-space data of frame FR2-2, integrates the k-space data of frame FR1-3 and frame FR1-4, which are time-sequential, to generate the k-space data of frame FR2-3, and integrates the k-space data of frame FR1-4 and frame FR1-5, which are time-sequential, to generate the k-space data of frame FR2-4.

[0269] Levels 3, 4, and 5 are processed similarly. For example, for level 3, the processing circuit 11 integrates the k-space data of frames FR1-1, FR1-2, and FR1-3, which are time-sequential, to generate the k-space data of frame FR3-1. For level 4, the processing circuit 11 integrates the k-space data of frames FR1-1, FR1-2, FR1-3, and FR1-4, which are time-sequential, to generate the k-space data of frame FR4-1. For level 5, the processing circuit 11 integrates the k-space data of frames FR1-1, FR1-2, FR1-3, FR1-4, and FR1-5, which are time-sequential, to generate the k-space data of frame FR5-1.

[0270] The above method of integrating frames is merely an example and is not limited to it. For example, in the above example, k-space data from two or more frames that are consecutive in time are integrated. However, k-space data from two or more frames that are not consecutive in time may also be integrated. For example, the k-space data of frame FR1-1 and the k-space data of frame FR1-3 may be integrated. Also, in the above example, multiple frames belonging to the same level are assumed not to overlap in time. However, multiple frames belonging to the same level may be set to overlap in time. Also, although multiple frames belonging to the same level are assumed to be set without gaps in time, two frames that are adjacent in time may be set to be separated in time.

[0271] Figure 36 shows an example of DNN reconstruction using dense input according to Example 2. As shown in Figure 36, the time resolution of Level 1 is the time resolution of one imaging frame, for example, 5 seconds / f (frame), the time resolution of Level 2 is for example, 10 seconds / f, and the time resolution of Level 3 is for example, 15 seconds / f.

[0272] The processing shown in Figure 36 is preferably performed in real time during MR imaging. The processing circuit 11 generates k-space data of multiple time-series frames by radial scanning or the like through the implementation of the imaging control function 111. For example, as shown in Figure 36, k-space data of frames FR1-1, FR1-2, FR1-3, and FR1-4 related to multiple level 1 frames in the time series are generated. Based on the k-space data of each frame FR1-n (where n is an integer), the processing circuit 11 immediately generates MR images of each frame FR1-n. Any method can be used to reconstruct the MR image, but to improve the responsiveness of MR image generation, a simple reconstruction method such as the Jackson method (or gridding method) is preferable.

[0273] Furthermore, each time an MR image of the latest Level 1 frame FR1-4 is generated, the processing circuit 11 generates MR images of Level 2 frames FR2-1, FR1-2, FR1-3, and FR2-3, and Level 3 frames FR3-1 and FR3-2, based on the k-space data of multiple Level 1 frames FR1-1, FR1-2, FR1-3, and FR1-4. The processing circuit 11 then selects the latest Level 1 frame FR1-4 MR image as the input image to be processed.

[0274] Next, auxiliary input images are selected. In Embodiment 2, the number of imaging frames to be selected as auxiliary input images (hereinafter referred to as the number of auxiliary input images) is set in advance. The number of auxiliary input images may be set as a total number, or the number may be set for each level. The processing circuit 11 selects the latest frame of MR images that matches the conditions from among the generated MR images of multiple levels and multiple frames as the auxiliary input image. For example, as shown in Figure 36, the number of frames for level 1 is set to 4, the number of frames for level 2 is set to 3, and the number of frames for level 3 is set to 2. In this case, the four frames FR1-4, FR1-3, FR1-2, and FR1-1 of level 1 prior to the latest frame FR1-4, and the multiple frames FR2-1, FR2-2, FR2-3, FR3-1, and FR3-2 of level 2 or higher based on those four frames are selected as auxiliary input images.

[0275] Thus, an MR image at the same level and frame as the input image to be processed may be selected as an auxiliary input image. If not necessary, an MR image at the same level and frame as the input image to be processed may be excluded from the auxiliary input images.

[0276] When a target input image and an auxiliary input image are selected, the processing circuit 11 inputs the selected target input image and auxiliary input image to the trained DNN (Fordense Input) and generates a target output image. For example, as shown in Figure 36, if the MR image of the current frame FR1-4 is input to the trained DNN (Fordense Input) as the target input image, and the MR images of frames FR1-4, FR1-3, FR1-2, FR1-1, FR2-3, FR2-2, FR2-1, FR3-2, and FR3-1 are input as auxiliary input images, the MR image of the current frame FR1-4 is output as the target output image. According to Embodiment 2, a group of MR images with hierarchical temporal resolution is input to the trained DNN as auxiliary input images. This allows the trained DNN to capture only the temporal continuity of a few frames that are temporally surrounding the current frame FR1-4. This further improves the image quality of the DNN-reconstructed image.

[0277] The trained DNN (For Dense Input) according to Example 2 can be generated by the model learning device 5 using the same method as in the above embodiment. The trained DNN (For Dense Input) is designed to accept multiple frames of MR images as auxiliary input images. A trained DNN (For Dense Input) is generated for each number of auxiliary input images. The ground truth output image can be any MR image that is of the same subject as the main input image and has higher image quality than the main input image. As such a ground truth output image, for example, an MR image with a higher temporal resolution or a larger amount of k-space data (in other words, a larger number of spokes) than the main input image should be selected. For example, if the temporal resolution of the main input image is level 1, an MR image with a temporal resolution of level 2 or higher should be selected as the ground truth output image. As auxiliary input images, a group of MR images for dense input is selected. That is, multiple MR images relating to multiple frames that are temporally surrounding the frame of the main input image and having a hierarchical temporal resolution are selected as auxiliary input images.

[0278] During training, the processing circuit 51 should use the dropout technique because the DNN for dense input requires a relatively large number of auxiliary input images. Dropout is a technique in which one or more units are randomly (or pseudo-randomly) selected from the multiple units included in the DNN to be deactivated, and training is performed using the DNN with the selected one or more units deactivated. By using the dropout technique, training can be performed accurately and efficiently.

[0279] As described above, in Example 2, using the dense input method, a group of MR images with hierarchical temporal resolution located temporally around the target input image (or main input image) are input to the DNN as auxiliary input images. Dense input is expected to improve the image quality of the output image and the learning efficiency of the DNN. Furthermore, since the auxiliary input images are limited to MR images from the past few frames of the current frame, MR images can be output in near real-time without waiting for the completion of video capture. In addition, the time until the output of the first image can be shortened.

[0280] (Example 3) Example 3 is an application of Example 2. In Example 2, MR images of level 1 frames having the same time resolution as the imaging frame were input to the DNN as auxiliary input images. However, this embodiment is not limited to this. The processing circuit 11 in Example 3 inputs only MR images of level 2 or higher frames to the DNN as auxiliary input images.

[0281] Figure 37 shows an example of DNN reconstruction using dense input according to Example 3. As shown in Figure 37, the time resolution of Level 1 is the time resolution of one imaging frame, for example, 2.5 seconds / f (frame), the time resolution of Level 2 is for example, 5 seconds / f, and the time resolution of Level 3 is for example, 7.5 seconds / f. In the case of Example 3, since the time resolution of Level 1 is relatively high, it is difficult to reconstruct an image using only the k-space data of one frame belonging to Level 1. Difficulty in reconstruction using only the data means that the amount of k-space data contained in one frame is small, resulting in an image quality unsuitable for interpretation. That is, it is possible to generate an MR image by applying a reconstruction method such as FFT to the k-space data of one frame.

[0282] As shown in Figure 37, k-space data for Level 1 frames FR1-1, FR1-2, FR1-3, FR1-4, and FR1-5 are generated in real time. The current frame is frame FR1-5. Based on the k-space data of each frame FR1-n (where n is an integer), the processing circuit 11 immediately generates MR images for each frame FR1-n using a simple reconstruction method such as the Jackson method. Furthermore, each time the latest MR image for Level 1 frame FR1-5 is generated, the processing circuit 11 generates MR images for Level 2 frames FR2-1, FR1-2, FR1-3, and FR1-4, and MR images for Level 3 frames FR3-1, FR3-2, and FR3-3, based on the k-space data of the current Level 1 frame FR1-5 and the k-space data of past frames FR1-1, FR1-2, FR1-3, and FR1-4. The processing circuit 11 then selects the latest frame FR1-5 MR images of level 1 as the input images to be processed.

[0283] Next, auxiliary input images are selected. In Embodiment 3, the processing circuit 11 selects MR images from among the MR images of level 2 or higher frames that match the number of auxiliary input images. For example, as shown in Figure 37, the number of auxiliary input images is set to 4 for level 2 frames and 4 for level 3 frames. In this case, the four frames FR1-4, FR1-3, FR1-2, and FR1-1 prior to the latest level 1 frame FR1-4, and the multiple frames FR2-1, FR2-2, FR2-3, FR3-1, and FR3-2 of level 2 or higher are selected as auxiliary input images.

[0284] When a target input image and an auxiliary input image are selected, the processing circuit 11 inputs the selected target input image and auxiliary input image to the trained DNN (Fordense Input) and generates a target output image. For example, as shown in Figure 37, when the MR image of the current frame FR1-5 is input to the trained DNN (Fordense Input) as the target input image, and the MR images of frames FR2-4, FR2-3, FR2-2, FR2-1, FR3-3, FR3-2, and FR3-1 are input as auxiliary input images, the DNN-reconstructed image of the current frame FR1-5 is output as the target output image.

[0285] Furthermore, as auxiliary input images, MR images from two or more frames at any level or higher may be selected, or only MR images from frames belonging to any level of level 2 or higher may be selected. For example, only MR images from frames belonging to the reconstruction level may be selected. Also, although the above embodiment states that the MR image of the current frame at level 1 is selected as the input image to be processed, it is not limited to this. For example, an MR image of a past frame at level 1 may be selected as the input image to be processed, or an MR image of any frame at level 2 or higher may be selected as the input image to be processed.

[0286] The trained DNN (For dense input) according to Example 3 can be generated by the model learning device 5 using the same method as in Example 2. For example, if the time resolution of the main input image is level 1, it is preferable to select an MR image with a time resolution of level 2 or higher as the ground truth output image. A group of MR images for dense input is selected as the auxiliary input image. That is, multiple MR images with hierarchical time resolutions that are temporally surrounding the frame of the main input image are selected as the auxiliary input image.

[0287] According to Example 3, when the time resolution of a Level 1 frame is high and standalone reconstruction is difficult, MR images from Level 2 or higher frames are input to the trained DNN as auxiliary input images using a dense input method. This makes it possible to output high-quality MR images while maintaining a high imaging frame rate.

[0288] (Other embodiments) In the embodiments and application examples described above, an example is given in which the parameters of a parameterized composite function are derived using a deep neural network, a machine learning technique. However, the embodiments are not limited to this. The machine learning technique is not limited to a technique that mimics the neural circuits of a biological brain; other techniques may be applied. Furthermore, the embodiments are not limited to deriving the parameters of a parameterized composite function using machine learning. For example, the parameters may be derived by a person adjusting them as appropriate without using machine learning. In other words, the trained model 90 used by the medical data processing device 1 is not limited to a machine learning model generated by the model learning device 5. For example, the medical data processing device 1 may use a machine learning model with parameters set according to user instructions as the trained model 90.

[0289] According to at least one embodiment described above, the accuracy of medical data recovery can be improved.

[0290] The term "processor" used in the above explanation can refer to, for example, CPUs, GPUs, or Application Specific Integrated Circuits (ASICs), programmable logic devices (such as Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and Field Programmable Gate Arrays (GATE Arrays)). Array (FPGA) refers to a circuit such as an array. The processor performs its functions by reading and executing a program stored in a memory circuit. Alternatively, instead of storing the program in a memory circuit, the processor may be configured to directly incorporate the program into its circuitry. In this case, the processor performs its functions by reading and executing the program incorporated into the circuitry. Furthermore, instead of executing a program, the processor may perform functions corresponding to the program by combining logic circuits. In this embodiment, each processor is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and perform its functions. Moreover, multiple components shown in Figures 1, 3, 14, 15, 17, 19, 21, and 24 may be integrated into a single processor to perform its functions.

[0291] The following is a summary of some of the inventions disclosed in this specification.

[0292] [Note 1-1] A storage unit for storing a trained model having an input layer that inputs first MR data and second MR data relating to the same imaging target as the first MR data but with different imaging parameters than the first MR data, an output layer that outputs third MR data in which the missing portion of the first MR data has been restored, and at least one intermediate layer provided between the input layer and the output layer, A medical data processing device comprising: a processing unit that generates third MR data relating to a subject from first MR data to be processed relating to a subject and second MR data relating to the subject and collected with imaging parameters different from those of the first MR data to be processed relating to the subject, according to the trained model;

[0293] [Appendix 1-2] The medical data processing device described in [Appendix 1-1], wherein the first MR data and the second MR data are k-space data or MR image data generated by performing a restoration process on the k-space data.

[0294] [Appendix 1-3] The medical data processing device described in [Appendix 1-2], wherein the restoration process is a denoising type restoration or a data error feedback type restoration.

[0295] [Appendix 1-4] The medical data processing device according to [Appendix 1-1], wherein the imaging parameters include at least one of slice position, acquisition time, acquisition sequence, k-space filling trajectory, and temporal resolution. [Appendix 1-5] The first MR data has a larger data reduction rate compared to the second MR data. (Medical data processing device as described in [Appendix 1-4])

[0296] [Appendix 1-6] The medical data processing device described in [Appendix 1-1] inputs the first MR data and the second MR data as a single input vector to the trained model.

[0297] [Appendix 1-7] The first MR data is set to a first range of the input vector, The second MR data is set to the second range of the input vector, The positions of the first range and the second range are fixed. [Appendix 1-6] Medical data processing device as described above.

[0298] [Appendix 1-8] The second MR data comprises multiple sets of MR data with different imaging parameters, and each of the multiple sets of the second MR data is set to a fixed range within the second range of the input vector. [Appendix 1-7] Medical data processing device as described above.

[0299] [Appendix 1-9] The medical data processing device according to [Appendix 1-1] further comprises a learning unit that generates estimated output data by applying first MR data and second MR data to a parameterized composite function obtained by combining multiple functions, and generates the trained model by updating the parameters of the parameterized composite function so that the estimated output data approximates the ground truth output data.

[0300] [Appendix 1-10] It further includes a selection unit that selects the imaging area according to user instructions, The processing unit switches the learned model according to the selected imaging area. [Appendix 1-1] The medical data processing device described above.

[0301] [Appendix 1-11] The imaging parameters include a first parameter and a second parameter. The first MR data and the second MR data share the same first parameter, but differ in the second parameter. The first MR data and the third MR data share the same first parameter and second parameter. The second MR data and the third MR data share the same first parameter, but differ in the second parameter. The MR data of the third mentioned above has fewer data loss or higher image quality compared to the first MR data. [Appendix 1-1] The medical data processing device described above.

[0302] [Appendix 1-12] The first parameter mentioned above is the slice position, The second parameters are the acquisition time, acquisition sequence, k-space filling trajectory, and temporal resolution. [Appendix 1-11] The medical data processing device described.

[0303] [Appendix 1-13] The first parameter is at least one of the acquisition sequence, the k-space filling trajectory, and the temporal resolution. The second parameter mentioned above is the slice position. [Appendix 1-11] The medical data processing device described.

[0304] [Appendix 1-14] The first parameter is the slice position and the EPI in the acquisition sequence, and the second parameter is the b value in the acquisition sequence. [Appendix 1-11] The medical data processing device described.

[0305] [Appendix 1-15] A storage unit for storing a trained model having: an input layer for inputting first k-space data or MR image data and second k-space data or MR image data relating to the same imaging target as the first k-space data or MR image data but with different imaging parameters than the first k-space data or MR image data; an output layer for outputting third k-space data or MR image data in which missing portions of the first k-space data or MR image data have been restored; and at least one intermediate layer provided between the input layer and the output layer. An acquisition unit that performs MR imaging on a subject and collects first k-space data relating to a first imaging parameter and second k-space data relating to a second imaging parameter different from the first imaging parameter, A processing unit that generates third k-space data or MR image data relating to the subject from the collected k-space data or MR image data based on the collected k-space data and the collected second k-space data or MR image data based on the collected k-space data, according to the trained model,

[0306] [Appendix 1-16] The collection unit collects k-space data of multiple frames in a time series, including the first k-space data and the second k-space data. The processing unit selects the k-space data of one first frame as the first k-space data from among the k-space data of the multiple frames, and selects the k-space data of one or more second frames as the second k-space data. [Appendix 1-15] Magnetic resonance imaging apparatus as described.

[0307] [Appendix 1-17] The collection unit collects k-space data of multiple frames in a time series, including the first k-space data and the second k-space data. The processing unit selects one k-space data from the k-space data of the multiple frames as the first k-space data, generates multiple k-space data of second frames with different acquisition times and / or temporal resolutions based on the k-space data of the multiple frames, and selects the k-space data of the multiple second frames as the second k-space data. [Appendix 1-15] Magnetic resonance imaging apparatus as described.

[0308] [Appendix 1-18] The processing unit generates input MR image data for the first frame based on the k-space data of the first frame, generates k-space data for the plurality of second frames based on the k-space data of the plurality of frames, and generates input MR image data for the plurality of second frames based on the k-space data of the plurality of second frames. According to the trained model, the output MR image data of the first frame is generated as the third MR data from the input MR image data of the first frame and the input MR image data of the plurality of second frames. [Appendix 1-17] Magnetic resonance imaging apparatus as described.

[0309] [Appendix 1-19] The plurality of frames and the first frame have a first time resolution level, The plurality of second frames have a second time resolution level lower than the first time resolution level. [Appendix 1-17] Magnetic resonance imaging apparatus as described.

[0310] [Appendix 1-20] The first temporal resolution level described above has a temporal resolution corresponding to one imaging frame, The second temporal resolution level has a temporal resolution corresponding to two or more imaging frames, and the plurality of second frames have N(N+1) / 2 frames. The plurality of second frames have (N+1-n) frames for each nth time resolution level, from the time resolution level corresponding to N imaging frames to the time resolution level corresponding to 2 imaging frames. [Appendix 1-19] Magnetic resonance imaging apparatus as described.

[0311] [Appendix 1-21] The first MR data is included in the second MR data, and the magnetic resonance imaging apparatus is as described in [1-16].

[0312] [Appendix 1-22] A step of generating estimated output data by applying a first MR data and a second MR data relating to the same target as the first MR data but with different imaging parameters to a parameterized composite function obtained by combining multiple functions, A step of generating a trained model by updating the parameters of the parameterized composite function so that the estimated output data and the ground truth output data from which the missing parts of the first MR data have been restored approximate each other. A method for generating a trained model that includes the following features.

[0313] [Note 2-1] A storage unit for storing a trained model having an input layer for inputting first CT data and second CT data relating to the same imaging target as the first CT data but with different imaging parameters than the first CT data, an output layer for outputting third CT data in which the missing portion of the first CT data has been restored, and at least one intermediate layer provided between the input layer and the output layer, A medical data processing device comprising: a processing unit that generates a third CT data relating to a subject from a first CT data to be processed relating to the subject and a second CT data relating to the subject but with imaging parameters different from the first CT data to be processed relating to the subject, according to the trained model;

[0314] [Note 2-2] The medical data processing device described in [Appendix 2-1], wherein the first CT data and the second CT data are projection data or CT image data generated by applying a restoration process to the projection data.

[0315] [Appendix 2-3] The medical data processing device described in [Appendix 2-2], wherein the restoration process is a denoising type restoration or a data error feedback type restoration.

[0316] [Appendix 2-4] The medical data processing device described in [Appendix 2-1] includes at least one of the imaging parameters: slice position, acquisition time, tube current, tube voltage, focal size, detector spatial resolution, number of views, reconstruction function, gantry rotation speed, and temporal resolution.

[0317] [Appendix 2-5] The first CT data has a larger data reduction amount compared to the second CT data. (Medical data processing device as described in [Appendix 2-4].)

[0318] [Appendix 2-6] The medical data processing device described in [Appendix 2-1] inputs the first CT data and the second CT data as a single input vector to the trained model.

[0319] [Appendix 2-7] The first CT data is set to a first range of the input vector, The second CT data is set to the second range of the input vector, The positions of the first range and the second range are fixed. [Appendix 2-6] Medical data processing device as described.

[0320] [Appendix 2-8] The second CT data comprises multiple sets of CT data with different imaging parameters, and each of the multiple sets of the second CT data is set to a fixed range within the second range of the input vector. [Appendix 2-7] Medical data processing device as described above.

[0321] [Appendix 2-9] The medical data processing device according to [Appendix 2-1] further comprises a learning unit that generates estimated output data by applying first CT data and second CT data to a parameterized composite function obtained by combining multiple functions, and generates the trained model by updating the parameters of the parameterized composite function so that the estimated output data approximates the ground truth output data.

[0322] [Appendix 2-10] It further includes a selection unit that selects the imaging area according to user instructions, The processing unit switches the learned model according to the selected imaging area. [Appendix 2-1] The medical data processing device described therein.

[0323] [Appendix 2-11] The imaging parameters include a first parameter and a second parameter. The first CT data and the second CT data share the same first parameter, but differ in the second parameter. The first CT data and the third CT data share the same first and second parameters. The second CT data and the third CT data share the same first parameter, but differ in the second parameter. The CT data of the third mentioned above has fewer data loss or higher image quality compared to the first CT data. [Appendix 2-1] The medical data processing device described therein.

[0324] [Appendix 2-12] The first parameter mentioned above is the slice position, The second parameters are the acquisition time and the tube current. [Appendix 2-11] The medical data processing device described.

[0325] [Appendix 2-13] A storage unit for storing a trained model having: an input layer for inputting first projection data or CT image data and second projection data or CT image data relating to the same imaging target as the first projection data or CT image data but with different imaging parameters than the first projection data or CT image data; an output layer for outputting third projection data or CT image data in which missing portions of the first projection data or CT image data have been restored; and at least one intermediate layer provided between the input layer and the output layer. A collection unit that performs CT imaging on the subject and collects first projection data relating to a first imaging parameter and second projection data relating to a second imaging parameter different from the first imaging parameter, A processing unit that generates third projection data or CT image data relating to the subject from the collected first projection data or CT image data and the collected second projection data or CT image data according to the trained model, An X-ray computed tomography (X-ray) scanner equipped with the following features.

[0326] [Appendix 2-14] The aforementioned acquisition unit acquires projection data for multiple rotations of the rotating frame, The processing unit generates CT image data of the first lap as the first CT image data based on the projection data of the first lap from among the projection data of multiple laps, and generates CT image data of the second lap as the second CT image data based on the projection data of the second lap from among the projection data of multiple laps. [Appendix 2-13] X-ray computed tomography equipment as described.

[0327] [Appendix 2-15] The projection data for the first rotation is generated by high-dose CT imaging. The projection data for the second rotation is generated by low-dose CT imaging. [Appendix 2-14] X-ray computed tomography equipment as described.

[0328] [Appendix 2-16] A step of generating estimated output data by applying a first CT data and a second CT data relating to the same imaging target as the first CT data but with different imaging parameters to a parameterized composite function obtained by combining multiple functions, A step of generating a trained model by updating the parameters of the parameterized composite function so that the estimated output data and the ground truth output data from which the missing parts of the first CT data have been restored approximate each other. A method for generating a trained model that includes the following features.

[0329] [Note 3-1] A storage unit for storing a trained model, comprising: an input layer that inputs first US data and second US data relating to the same imaging target as the first US data but with different imaging parameters; an output layer that outputs third US data from which missing portions of the first US data have been restored; and at least one intermediate layer provided between the input layer and the output layer. A medical data processing device comprising: a processing unit that generates a third US data relating to a subject from a first US data to be processed relating to the subject and a second US data relating to the subject but with imaging parameters different from the first US data to be processed relating to the subject, according to the trained model;

[0330] [Note 3-2] The medical data processing device described in [Appendix 3-1], wherein the first US data and the second US data are US raw data or US image data generated by performing a restoration process on the US raw data.

[0331] [Appendix 3-3] The medical data processing device described in [Appendix 3-2], wherein the restoration process is a denoising type restoration or a data error feedback type restoration.

[0332] [Appendix 3-4] The medical data processing device described in [Appendix 3-1], wherein the imaging parameters include at least one of slice position, acquisition time, focal position, gain, transmission intensity, reception intensity, PRF, beam scanning method, scanning mode, and temporal resolution.

[0333] [Appendix 3-5] The first US data has a larger amount of data decimation compared to the second US data. (Medical data processing device as described in [Appendix 3-4])

[0334] [Appendix 3-6] The medical data processing device described in [Appendix 3-1], wherein the first US data and the second US data are input to the trained model as a single input vector.

[0335] [Appendix 3-7] The first US data is set to a first range of the input vector, The second US data is set to the second range of the input vector, The positions of the first range and the second range are fixed. [Appendix 3-6] Medical data processing device as described.

[0336] [Appendix 3-8] The second US data comprises multiple sets of US data with different imaging parameters, and each of the multiple sets of the second US data is set to a fixed range within the second range of the input vector. [Appendix 3-7] Medical data processing device as described.

[0337] [Appendix 3-9] The medical data processing device according to [Appendix 3-1] further comprises a learning unit that generates estimated output data by applying first US data and second US data to a parameterized composite function obtained by combining multiple functions, and generates the trained model by updating the parameters of the parameterized composite function so that the estimated output data approximates the ground truth output data.

[0338] [Appendix 3-10] It further includes a selection unit that selects the imaging area according to user instructions, The processing unit switches the learned model according to the selected imaging area. [Appendix 3-1] The medical data processing device described therein.

[0339] [Appendix 3-11] The imaging parameters include a first parameter and a second parameter. The first US data and the second US data share the same first parameter, but differ in the second parameter. The first US data and the third US data share the same first parameter and second parameter. The second US data and the third US data share the same first parameter, but differ in the second parameter. The US data of the third type described above has fewer data loss or higher image quality compared to the first US data described above. [Appendix 3-1] The medical data processing device described therein.

[0340] [Appendix 3-12] The first parameter mentioned above is the slice position, The second parameter mentioned above is the collection time. [Appendix 3-11] The medical data processing device described therein.

[0341] [Appendix 3-13] A storage unit for storing a trained model having: an input layer that inputs first US raw data or US image data and second US raw data or US image data relating to the same imaging target as the first US raw data or US image data but with different imaging parameters than the first US raw data or US image data; an output layer that outputs third US raw data or US image data from which missing portions of the first US raw data or US image data have been restored; and at least one intermediate layer provided between the input layer and the output layer. A data collection unit that performs US imaging on the subject and collects first raw US data relating to a first imaging parameter and second raw US data relating to a second imaging parameter different from the first imaging parameter. A processing unit that generates a third US raw data or US image data relating to the subject from the collected first US raw data or US image data based on the collected first US raw data and the collected second US raw data or US image data based on the collected second US raw data, according to the trained model, An ultrasound diagnostic device equipped with the following features.

[0342] [Appendix 3-14] The aforementioned collection unit collects raw US data from multiple time-series frames. The processing unit selects from the US raw data of the multiple frames one US raw data of a first frame as the first US raw data, and selects one or more US raw data of second frames as the second US raw data. [Appendix 3-13] The ultrasound diagnostic device described.

[0343] [Appendix 3-15] The aforementioned processing unit, Based on the raw US data of the first frame, input US image data for the first frame is generated, and based on the raw US data of the second frame, input US image data for the second frame is generated. According to the trained model, the output US image data of the first frame is generated as the third US data from the input US image data of the first frame and the input US image data of the plurality of second frames. [Appendix 3-14] The ultrasound diagnostic device described.

[0344] [Appendix 3-16] The system further includes a display unit that displays the US image data of the multiple frames in real time, based on the US raw data of the multiple frames. The aforementioned processing unit, When the user issues an image freeze command, the US image data that was displayed on the display device at the time the image freeze command was issued is selected as the US image data for the first frame. The US image data of a frame a predetermined frame before the time the image freeze command was issued, or a frame a predetermined frame after the time the image freeze command was issued, is selected as the US image data for the second frame. [Appendix 3-15] The ultrasound diagnostic device described.

[0345] [Appendix 3-17] A step of generating estimated output data by applying a first US data and a second US data relating to the same target as the first US data but with different imaging parameters to a parameterized composite function obtained by combining multiple functions, A step of generating a trained model by updating the parameters of the parameterized composite function so that the estimated output data and the ground truth output data from which the missing parts of the first US data have been restored approximate each other. A method for generating a trained model that includes the following features.

[0346] [Note 4-1] A storage unit for storing a trained model having an input layer that inputs first medical data and second medical data relating to the same imaging target as the first medical data but with different imaging parameters than the first medical data, an output layer that outputs third medical data in which the missing portion of the first medical data has been restored, and at least one intermediate layer provided between the input layer and the output layer, A medical data processing device comprising: a processing unit that generates third medical data relating to a subject from first medical data to be processed relating to a subject and second medical data relating to the subject but with imaging parameters different from those of the first medical data to be processed, according to the trained model;

[0347] [Note 4-2] The first medical data and the second medical data are raw data or medical image data generated by performing a restoration process on the raw data, as described in [Appendix 4-1], in the medical data processing device.

[0348] [Note 4-3] The medical data processing device described in [Appendix 4-2], wherein the restoration process is a denoising type restoration or a data error feedback type restoration.

[0349] [Appendix 4-4] The medical data processing device according to [Appendix 4-1], wherein the imaging parameters include at least one of slice position, medical data imaging principle, and temporal resolution.

[0350] [Appendix 4-5] The first medical data is processed using a method described in [Appendix 4-4] which involves a larger amount of data decimation compared to the second medical data.

[0351] [Appendix 4-6] The medical data processing device described in [Appendix 4-1] inputs the first medical data and the second medical data as a single input vector to the trained model.

[0352] [Appendix 4-7] The first medical data is set to a first range of the input vector, The second medical data is set to the second range of the input vector, The positions of the first range and the second range are fixed. [Appendix 4-6] Medical data processing device as described.

[0353] [Appendix 4-8] The second medical data comprises multiple sets of medical data with different imaging parameters, and each of the multiple sets of the second medical data is set to a fixed range within the second range of the input vector. [Appendix 4-7] Medical data processing device as described above.

[0354] [Appendix 4-9] The medical data processing device according to [Appendix 4-1] further comprises a learning unit that generates estimated output data by applying first medical data and second medical data to a parameterized composite function obtained by combining multiple functions, and generates the trained model by updating the parameters of the parameterized composite function so that the estimated output data approximates the ground truth output data.

[0355] [Appendix 4-10] It further includes a selection unit that selects the imaging area according to user instructions, The processing unit switches the learned model according to the selected imaging area. [Appendix 4-1] The medical data processing device described therein.

[0356] [Appendix 4-11] The imaging parameters include a first parameter and a second parameter. The first medical data and the second medical data share the same first parameter, but differ in the second parameter. The first medical data and the third medical data share the same first parameter and second parameter. The second medical data and the third medical data share the same first parameter, but differ in the second parameter. The medical data in the third instance has fewer data loss or higher image quality compared to the first medical data. [Appendix 4-1] The medical data processing device described therein.

[0357] [Appendix 4-12] The first parameter mentioned above is the slice position, The second parameter described above is the imaging principle for medical data. [Appendix 4-11] The medical data processing device described therein.

[0358] [Appendix 4-13] The second parameter relating to the first medical data is PET imaging as the imaging principle, The second parameter relating to the second medical data is X-ray CT imaging as the imaging principle. [Appendix 4-12] The medical data processing device described therein.

[0359] [Appendix 4-14] A storage unit for storing a trained model having an input layer that inputs first medical data and second medical data relating to the same imaging target as the first medical data but with different imaging parameters than the first medical data, an output layer that outputs third medical data in which the missing portion of the first medical data has been restored, and at least one intermediate layer provided between the input layer and the output layer, A data collection unit that performs imaging on the subject and collects first medical data relating to a first imaging parameter and second medical data relating to a second imaging parameter different from the first imaging parameter. A processing unit that generates third medical data relating to the subject from the collected first medical data and the collected second medical data according to the trained model, A medical imaging diagnostic device equipped with the following features.

[0360] [Appendix 4-15] The medical imaging diagnostic apparatus described in [Appendix 4-14] comprises a PET scanner for performing PET imaging on the subject to collect first medical data and an X-ray CT scanner for performing X-ray CT imaging on the subject to collect second medical data.

[0361] [Appendix 4-16] A step of generating estimated output data by applying first medical data and second medical data relating to the same imaging target as the first medical data but with different imaging parameters to a parameterized composite function obtained by combining multiple functions, A step of generating a trained model by updating the parameters of the parameterized composite function so that the estimated output data and the ground truth output data from which the missing parts of the first medical data have been restored approximate each other. A method for generating a trained model that includes the following features. [Explanation of Symbols]

[0362] 1. Medical data processing device 2. Medical data processing device 3. Medical imaging device 5 Model Learning Devices 7. Learning Data Storage Device 9. Medical imaging diagnostic equipment 11 Processing Circuit 13 memory 15 Input Interfaces 17 Communication Interface 19 displays 21 Processing Circuit 23 memory 25 Input Interfaces 27 Output Interface 50 Learning Programs 51 Processing Circuit 53 memory 55 Input Interfaces 57 Communication Interface 59 displays 90 pre-trained models 100 Medical Signal Processing Systems 111 Imaging control function 112 Normal recovery function 113 Input Selection Function 114 Forward propagation function 115 Image processing functions 116 Display control function 511 Forward propagation function 512 Backpropagation Function 513 Update function 514 Judgment Function 515 Display control function

Claims

1. A processing unit that takes as input first medical data relating to data captured by a first medical imaging device and second medical data relating to the same imaging target as the first medical data but with different imaging parameters relating to data captured by a second medical imaging device that is the same as or different from the first medical imaging device, and outputs third medical data in which the missing portion of the first medical data has been restored, and generates third medical data in which the missing portion of the first medical data has been restored from the first medical data to be processed and the second medical data relating to different imaging parameters relating to the first medical data to be processed, according to a trained model. A medical data processing device equipped with the following features.

2. The medical data processing apparatus according to claim 1, wherein the first medical data and the second medical data are raw data or medical image data generated by performing a restoration process on the raw data.

3. The medical data processing apparatus according to claim 1, wherein the first imaging diagnostic apparatus and the second imaging diagnostic apparatus are the same or different PET apparatuses.

4. The medical data processing apparatus according to claim 1 or 3, wherein the processing unit generates a third PET image, which is the third medical data, from a first PET image, which is the first medical data, and a second PET image, which is the second medical data, which is a slice different from the first PET image, thereby reducing the noise of the first PET image.

5. The medical data processing apparatus according to claim 1, wherein the first imaging diagnostic apparatus and the second imaging diagnostic apparatus are the same X-ray computed tomography apparatus, magnetic resonance imaging apparatus, ultrasound diagnostic apparatus, or X-ray diagnostic apparatus.

6. The first imaging diagnostic device is a PET scanner and the second imaging diagnostic device is an X-ray computed tomography scanner. The first diagnostic imaging device is a SPECT scanner, and the second diagnostic imaging device is an X-ray computed tomography scanner. The first imaging diagnostic device is a PET scanner and the second imaging diagnostic device is a magnetic resonance imaging device, or The first diagnostic imaging device is a SPECT device, and the second diagnostic imaging device is a magnetic resonance imaging device. The medical data processing device according to claim 1.

7. The medical data processing apparatus according to claim 1, wherein the imaging parameters include at least one of slice position and temporal resolution.

8. The medical data processing device according to claim 7, wherein the first medical data undergoes a larger data decimation amount compared to the second medical data.

9. The medical data processing device according to claim 1, wherein the first medical data and the second medical data are input to the trained model as a single input vector.

10. The first medical data is set to a first range of the input vector, The second medical data is set to the second range of the input vector, The positions of the first range and the second range are fixed. The medical data processing device according to claim 9.

11. The second medical data comprises multiple sets of medical data with different imaging parameters. The second medical data for each of the multiple sets is set to a fixed range within the second range of the input vector. The medical data processing device according to claim 10.

12. The medical data processing device according to claim 1, further comprising a learning unit that generates a trained model by applying first medical data and second medical data to a parameterized composite function obtained by combining multiple functions to generate estimated output data, and updating the parameters of the parameterized composite function so that the estimated output data approximates the ground truth output data.

13. It further includes a selection unit that selects the imaging area according to user instructions, The processing unit switches the learned model according to the selected imaging area. The medical data processing device according to claim 1.

14. The imaging parameters include a first parameter and a second parameter, The first medical data and the second medical data share the same first parameter, but differ in the second parameter. The first medical data and the third medical data share the same first parameter and second parameter. The second medical data and the third medical data share the same first parameter, but differ in the second parameter. The medical data in the third instance has fewer data loss or higher image quality compared to the first medical data. The medical data processing device according to claim 1.

15. The first parameter is the slice position, The first and second diagnostic imaging devices are diagnostic imaging devices based on different imaging principles. The medical data processing device according to claim 14.

16. The first imaging diagnostic device is a PET device, The second imaging diagnostic device mentioned above is an X-ray CT scanner. The medical data processing device according to claim 15.

17. A collection unit that performs imaging on the subject and collects first medical data obtained using a first imaging principle and relating to first imaging parameters, and second medical data obtained using a second imaging principle different from the first imaging principle and relating to second imaging parameters different from the first imaging parameters, A processing unit that takes as input first medical data relating to data captured using the first imaging principle and second medical data relating to data captured using a second imaging principle that is the same as or different from the first imaging principle, and captures the same target as the first medical data and captures the same subject as the first medical data, and outputs third medical data in which the missing portion of the first medical data has been restored, and generates third medical data in which the missing portion of the first medical data has been restored from the collected first medical data and the collected second medical data, according to a trained model; A medical imaging diagnostic device equipped with the following features.

18. The medical imaging diagnostic apparatus according to claim 17, wherein the collection unit comprises a PET scanner for performing PET imaging on the subject to collect the first medical data and an X-ray CT scanner for performing X-ray CT imaging on the subject to collect the second medical data.

19. A step of generating estimated output data by applying to a parameterized composite function obtained by combining multiple functions first medical data relating to data captured by a first medical imaging device and second medical data relating to data captured by a second medical imaging device that is the same as or different from the first medical imaging device, for the same target as the first medical data but with different imaging parameters than the first medical data, A step of generating a trained model by updating the parameters of the parameterized composite function so that the estimated output data and the correct output data from which the missing parts of the first medical data have been restored approximate each other. A method for generating a trained model that includes the following features.

Images