Inference device, medical image diagnostic device, inference method, and trained neural network generation method

Abstract

To improve inference performance of a trained neural network using an activation function.SOLUTION: An inference device includes an acquisition section and an inference section. The acquisition section acquires processing target data. The inference section calculates inference data by applying a trained neural network to the processing target data. The trained neural network has an ensemble activation function that executes calculation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the plurality of activation functions, for each of a plurality of unit network structures which converts an input vector element into an output vector element.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to an inference device, a medical image diagnostic device, an inference method, and a method for generating a trained neural network. [Background technology]

[0002] In neural networks that perform desired inference on medical data such as medical image data and raw data thereof, there are methods that apply deep neural networks (DNNs) and convolutional neural networks (CNNs).

[0003] In such neural networks, various types of activation functions have been considered as functions used in activation processing. Examples of activation functions include the logistic sigmoid function (logistic function), hyperbolic tangent function (tanh), rectified linear unit (ReLU), linear map, identity map, max-out function, ELU, LeakyReLU, and ComplexReLU. However, the inference performance when using any of these activation functions has its advantages and disadvantages.

[0004] Conventionally, the type of activation function used in a neural network is determined by the designer. Therefore, the designer needs to know in advance the type of activation function that is optimal for the intended use and optimization method. On the other hand, it is known that the selection of the type of activation function to be used has a significant impact on the inference performance of the neural network. For this reason, it is desirable to use an appropriate activation function that will improve the inference performance of the neural network. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Hidenori Takeshima, "Deep Learning and Its Application to Function Approximation for MR in Medicine: An Overview", [Online], September 17, 2021, Magnetic Resonance in Medical Sciences, [Retrieved October 15, 2021], Internet<URL:https: / / doi.org / 10.2463 / mrms.rev.2021-0040> Summary of the Invention [Problem to be solved by the invention]

[0006] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the inference performance of a trained neural network using an activation function. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0007] An inference device according to an embodiment includes an acquisition unit and an inference unit. The acquisition unit acquires processing target data. The inference unit calculates inference data by applying a trained neural network to the processing target data. The trained neural network has an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients corresponding to the plurality of activation functions, for each of a plurality of unit network structures that converts input vector elements into output vector elements. [Brief explanation of the drawings]

[0008] [Figure 1]FIG. 1 is a diagram showing an overview of the configuration and processing of a medical data processing system to which a medical data processing apparatus according to an embodiment belongs. [Figure 2] FIG. 2 is a diagram showing the structure of a neural network according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an outline of a calculation using an ensemble activation function according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating the configuration of a convolutional layer according to the embodiment. [Figure 5] FIG. 5 is a diagram showing the configuration of a medical image diagnostic apparatus according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a combination of inputs and outputs of a trained neural network according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating a configuration of a model learning device according to the embodiment. [Figure 8] FIG. 8 is a diagram schematically illustrating a calculation using an ensemble activation function according to a first modified example of the embodiment. [Figure 9] FIG. 9 is a diagram schematically illustrating a calculation using an ensemble activation function according to a second modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the drawings, embodiments of an inference device and a trained neural network generation method will be described in detail. The inference device is a device that calculates inference data by applying a trained neural network to data to be processed. The inference device is typically a medical data processing device that receives input medical data to be processed and performs processing to output corresponding output medical data. The input medical data corresponds to the data to be processed. Note that the inference device is not limited to the medical field and may also be used for inference processing of data outside the medical field. In the following embodiments, a case will be described in which the inference device processes medical data. In the following description, components having approximately the same functions and configurations will be given the same symbols, and repeated description will be provided only when necessary.

[0010] (Embodiment) 1 is a diagram showing an overview of the configuration and processing of a medical data processing system 100 to which a medical data processing device 1, which is an inference device according to this embodiment, belongs. As shown in FIG. 1, the medical data processing system 100 according to this embodiment includes the medical data processing device 1, a medical imaging device 3, a model learning device 5, and a learning data storage device 7.

[0011] The training data storage device 7 stores training data including a plurality of training samples. For example, the training data storage device 7 is a computer with a built-in storage device. Alternatively, the training data storage device 7 may be a large-capacity storage device communicatively connected to the computer via a cable or a communication network. As the storage device, a hard disk drive (HDD), a solid state drive (SSD), an integrated circuit storage device, or the like can be used as appropriate.

[0012] The model learning device 5 causes a machine learning model to perform machine learning according to a model learning program based on the learning data stored in the learning data storage device 7, thereby generating a trained machine learning model (hereinafter referred to as a trained model). The model learning device 5 is a computer such as a workstation having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The model learning device 5 and the learning data storage device 7 may be communicatively connected via a cable or a communication network, or the learning data storage device 7 may be mounted on the model learning device 5. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via a cable, a communication network, or the like. The model learning device 5 and the learning data storage device 7 do not need to be communicatively connected. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via a portable storage medium on which the learning data is stored. The model learning device 5 is an example of a learning device.

[0013] The machine learning model according to this embodiment is a parameterized composite function obtained by combining multiple functions. The parameterized composite function is defined by a combination of multiple adjustable functions and parameters. The machine learning model according to this embodiment may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multi-layer network model (hereinafter referred to as a multi-layer network). As the multi-layer network, for example, a neural network such as a deep neural network (DNN) or a convolutional neural network (CNN) having a convolutional layer is used. In the following description, the multi-layer network according to this embodiment is assumed to be a neural network. The neural network according to this embodiment receives input medical data to be processed, which is collected by a medical imaging device 3, as input, and outputs corresponding output medical data.

[0014] The medical imaging apparatus 3 generates medical data to be processed. Conceptually, the medical data according to this embodiment includes raw data collected by performing medical imaging on a subject using the medical imaging apparatus 3 or another medical imaging apparatus, and medical image data generated by performing restoration processing on the raw data. The medical imaging apparatus 3 may be any modality apparatus capable of generating medical data. For example, the medical imaging apparatus 3 according to this embodiment may be a single-modality apparatus such as a magnetic resonance imaging apparatus (MRI apparatus), an X-ray computed tomography apparatus (CT apparatus), an X-ray diagnostic apparatus, a PET (Positron Emission Tomography) apparatus, a SPECT (Single Photon Emission CT) apparatus, or an ultrasound diagnostic apparatus, or may be a multi-modality apparatus such as a PET / CT apparatus, a SPECT / CT apparatus, a PET / MRI apparatus, or a SPECT / MRI apparatus.

[0015] The medical data processing device 1 generates output medical data corresponding to input medical data to be processed, which is collected by the medical imaging device 3, using a trained model trained by the model learning device 5 according to a model learning program. The medical data processing device 1 and the model learning device 5 may be communicatively connected via a cable or a communication network, or the medical data processing device 1 and the model learning device 5 may be implemented on a single computer. In this case, the trained model is supplied from the model learning device 5 to the medical data processing device 1 via a cable, a communication network, or the like. The medical data processing device 1 and the model learning device 5 do not necessarily have to be communicatively connected. In this case, the trained model is supplied from the model learning device 5 to the medical data processing device 1 via a portable storage medium or the like on which the trained model is stored. The trained model may be supplied at any time between the manufacture of the medical data processing device 1 and its installation in a medical facility or the like, or during maintenance, or at any other time. The supplied trained model is stored in the medical data processing device 1. Furthermore, the medical data processing device 1 may be a computer installed in a medical image diagnostic device that is equipped with a medical imaging device 3, or may be a computer that is communicatively connected to the medical image diagnostic device via a cable, a network, etc., or may be a computer independent of the medical image diagnostic device.

[0016] A typical configuration of a neural network according to this embodiment will be described below. FIG. 2 is a diagram showing a typical configuration of a neural network according to this embodiment. Here, a neural network is a network that has a structure in which connections are made only between adjacent layers arranged in layers, and in which information propagates in one direction from the input layer side to the output layer side. The neural network according to this embodiment is a forward propagation network in which image data input to the input layer propagates from the input layer side to the output layer side while connecting only between adjacent layers.

[0017] The neural network according to this embodiment is assumed to be composed of L layers, namely, an input layer (l=1), intermediate layers (l=2, 3, . . . , L-1), and an output layer (l=L), as shown in Fig. 2. Note that the following is an example, and the configuration of the neural network is not limited to the following description.

[0018] Input data is input to the input layer (first layer). The input data can be, for example, image data such as medical image data, or raw data (RAW data) such as k-space data or projection data. In the input layer, the input data becomes output data as is.

[0019] The intermediate layer (l=2,3,...,L-1) following the input layer outputs calculated values ​​by sequentially executing calculations based on the weighting matrices between each layer, the bias of each layer, and the activation process of each layer.

[0020] In the output layer (Lth layer) following the intermediate layer, the data input from the intermediate layer becomes the output data as is.

[0021] The neural network according to this embodiment is a forward propagation network in which data input to an input layer propagates from the input layer to the output layer, with connections only between adjacent layers. Such a forward propagation network is defined as a composite function that combines linear relationships between layers using a weighting matrix W, nonlinear relationships (or linear relationships) using activation processing in each layer, and biases. In particular, the weighting matrix and bias are called parameters p of the network. The form of the composite function defined in this way changes depending on how the parameter p is selected. Therefore, the neural network according to this embodiment can be defined as a function that enables the output layer to output desirable results by appropriately selecting the parameter p of the composite function.

[0022] A neural network is composed of multiple unit network structures. Each unit network structure is a unit of elements that make up a network, converting input vector elements into output vector elements. Unit network structures correspond to, for example, nodes, layers, channels, units, etc. For example, each unit network structure receives as input an input vector element that includes the output values ​​of multiple other unit network structures, calculates a value by multiplying each value of the input vector element by a different weight and adding a bias b, and applies activation processing to the calculated value. In the activation processing, a single value is output after applying a nonlinear transformation (or linear transformation) to the calculated value. The calculated value is then output as the output of the unit network structure.

[0023] In this embodiment, a new function called an ensemble activation function is used for the activation process. The ensemble activation function is generated by preparing multiple activation functions and incorporating a part that integrates them into the learning process. Specifically, the ensemble activation function applies multiple activation functions to the same data and outputs an output value calculated based on each activation function and a mixture coefficient.

[0024] An ensemble activation function is, for example, a function that applies multiple activation functions to the same data, integrates the output values ​​of the multiple activation functions according to a mixing coefficient, and outputs the integrated output value.

[0025] As the activation function, various functions can be selected depending on the purpose, such as the logistic sigmoid function (logistic function), hyperbolic tangent function (tanh), rectified linear unit (ReLU), linear mapping, identity mapping, max-out function, etc. Alternatively, ELU, LeakyReLU, Complex ReLU, etc. may be used as the activation function.

[0026] The mixing coefficients are weighting parameters assigned to each activation function to be integrated, and are determined, for example, by optimizing the mixing coefficients together with other weighting parameters included in the neural network during the process of training the neural network so that it can receive predetermined input data and output desired output data.

[0027] The ensemble activation function according to this embodiment will be described below with reference to FIG. 3. FIG. 3 is a diagram schematically illustrating the operation of the ensemble activation function according to this embodiment. As shown in FIG. 3, the ensemble activation function is composed of three activation functions A1-A3 and a mixing function M. The activation functions A1-A3 are different types of activation functions. In FIG. 3, the activation functions A1-A3 are represented as "Act.1"-"Act.3." The number of activation functions used in the ensemble activation function may be two, four, or more.

[0028] The ensemble activation function accepts an input vector element x_i, which represents a vector composed of multiple input values ​​x output from a specific channel i constituting a CNN.

[0029] The input vector element x_i is applied to each activation function A1-A3. The activation function A1 performs an operation using each value x of the input vector element x_i as input and outputs the result as output value z1. Similarly, the activation functions A2 and A3 perform an operation using each value x of the input vector element x_i as input and output values ​​z2 and z3.

[0030] The mixing function M receives output values ​​from multiple types of activation functions prepared in advance as inputs and outputs a single output value by integrating the acquired output values. In the example shown in Figure 3, the mixing function M integrates three output values ​​z1-z3 acquired from the activation functions A1-A3 according to a mixing coefficient to calculate the output value y.

[0031] The mixing function M may be a function that calculates a weighted linear sum of multiple inputs, or a function that calculates a weighted nonlinear sum of multiple inputs. Furthermore, the mixing function M may be a linear function with each of the multiple inputs as a variable, a quadratic function, or a function expressed as a polynomial with a degree of three or more. Furthermore, a kernel function used in the kernel trick technique may be used as the mixing function M. Furthermore, a neural network expressed as a parameterized composite function in which multiple functions are combined may be used as the mixing function M.

[0032] For example, when there are two types of activation functions forming an ensemble activation function, the mixing function M integrates the output values ​​z1 and z2 of two types of activation functions A1 and A2 prepared in advance to calculate the output value y. Equation (1) shows an example of a calculation formula for the output value y when a linear function is used as the mixing function M. In equation (1), each of the coefficients a1 to a3 is a mixing coefficient in this embodiment. The coefficients a1 to a3 are determined by optimizing the entire neural network together with other parameters. In equation (1), the mixing function M is a function that calculates the value y obtained by adding together the output values ​​z1 and z2 of the activation functions A1 and A2 multiplied by different coefficients.

number

number

[0033] The activation function and mixing function M are applied sequentially to each value x included in the input vector element x_i, and an output value y for each value x included in the input vector element x_i is output. The ensemble activation function generates an output vector element y_i including the output value y output for each value x included in the input vector element x_i, and the generated output vector element y_i is the output of the ensemble activation function.

[0034] Next, a case where CNN is used as an example of a neural network will be described. CNN includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. In CNN, for example, a processing block in which a pooling layer is arranged after two convolutional layers is repeatedly arranged multiple times before the fully connected layer. The number of connections and the order of connections in the convolutional layer and the pooling layer are set appropriately.

[0035] Input data is provided to the input layer. The input data is typically vector data. Depending on the implementation, the input data may be read out for each block of input data in memory (for example, for each channel), or for each element containing input data for multiple channels. In the convolutional layer, convolution processing is performed on the input data from the input layer. In the pooling layer, max pooling processing, for example, is performed on the convolutionally processed data. In the fully connected layer, the data processed in the processing block and the channels of the fully connected layer are fully connected between layers. In the output layer, output data, which is the final output from the CNN, is generated.

[0036] In the convolution layer, convolution processing, regularization processing, and activation processing are performed. Note that regularization processing and activation processing are not essential and may not be performed. If activation processing is not performed in the convolution layer, activation processing is performed in a layer other than the convolution layer. Figure 4 is a diagram schematically illustrating a typical configuration of a convolution layer. In Figure 4, convolution processing is represented as "Conv." and activation processing is represented as "Ensemble Act." The regularization processing is omitted in Figure 4.

[0037] In convolution processing, the input data in the input layer is convolved for each channel. For example, in convolution processing, one kernel (filter) is used for each channel, and the data after convolution processing is generated as a feature map.

[0038] In regularization, a feature map that has undergone convolution processing is input and regularization processing is performed on the feature map. For example, batch normalization or dropout is used as the regularization processing. Any general processing can be used for the regularization processing.

[0039] In the activation process, the ensemble activation function is applied to the convolutional or regularized data to generate the final output data from the convolutional layer, which is then input to the adjacent lower convolutional or pooling layer.

[0040] In the output layer, an activation process using an activation function is performed on the output from the fully connected layer. In the activation process, for example, a softmax function is applied as the activation function. Alternatively, other activation functions may be applied depending on the desired output format. For example, when using a CNN for binary classification, a logistic function is used as the activation function, and when using a CNN for regression problems, a linear map is used as the activation function.

[0041] The ensemble activation function may be used for the activation process in the output layer instead of the activation process in the convolutional layer. Also, the ensemble activation function may be used for the activation process in both the convolutional layer and the output layer.

[0042] An example of the configuration of the medical data processing device 1 according to this embodiment will be described below. In the following description, it is assumed that the medical data processing device 1 is connected to a medical imaging device 3 and incorporated together with the medical imaging device 3 into a medical image diagnostic device 9.

[0043] FIG. 5 is a diagram showing the configuration of a medical image diagnostic apparatus 9 according to this embodiment. As shown in FIG. 5, the medical image diagnostic apparatus 9 includes a medical data processing device 1 and a medical imaging apparatus 3. For example, the medical imaging apparatus 3 corresponds to a gantry, and the medical data processing device 1 corresponds to a console connected to the gantry. The medical data processing device 1 may be provided on the gantry of the medical image diagnostic apparatus 9, or may be realized by a separate component that is neither the console nor the gantry of the medical image diagnostic apparatus 9. An example of the separate component is a computer other than the console or a dedicated computing device installed in an equipment room when the medical image diagnostic apparatus 9 is a magnetic resonance imaging apparatus.

[0044] The medical imaging device 3 performs medical imaging on a subject using an imaging principle corresponding to the modality type of the medical imaging device 3, and collects raw data related to the subject. The collected raw data is transmitted to the medical data processing device 1. For example, the raw data is k-space data when the medical imaging device 3 is a magnetic resonance imaging device, projection data or sinogram data when the medical imaging device 3 is an X-ray computed tomography device, echo data when the medical imaging device 3 is an ultrasound diagnostic device, coincidence data or sinogram data when the medical imaging device 3 is a PET device, or projection data or sinogram data when the medical imaging device 3 is a SPECT device. Furthermore, when the medical imaging device 3 is an X-ray diagnostic device, the raw data is X-ray image data. The medical imaging device 3 is an example of an imaging unit.

[0045] When the medical imaging device 3 is a gantry for a magnetic resonance imaging device, the gantry applies a static magnetic field via a static magnetic field magnet, and repeatedly applies a gradient magnetic field via a gradient magnetic field coil and an RF pulse via a transmission coil. MR signals are emitted from the subject due to the application of RF pulses. The emitted MR signals are received via a receiving coil. The received MR signals are subjected to signal processing such as A / D conversion by a receiving circuit. The MR signals after A / D conversion are called k-space data. The k-space data is transmitted to the medical data processing device 1 as raw data.

[0046] When the medical imaging device 3 is a gantry for an X-ray computed tomography device, the gantry rotates an X-ray tube and an X-ray detector around the subject, irradiating the subject with X-rays from the X-ray tube, and detecting the X-rays that pass through the subject with the X-ray detector. The X-ray detector generates an electrical signal having a peak value corresponding to the detected X-ray dose. The electrical signal undergoes signal processing such as A / D conversion by a data acquisition circuit. The electrical signal after A / D conversion is called projection data or sinogram data. The projection data or sinogram data is transmitted to the medical data processing device 1 as raw data.

[0047] When the medical imaging device 3 is an ultrasound probe of an ultrasound diagnostic device, the ultrasound probe transmits ultrasound beams from multiple ultrasound transducers into the body of a subject and receives ultrasound reflected from within the body of the subject via the ultrasound transducers. The ultrasound transducers generate electrical signals having peak values ​​corresponding to the sound pressure of the received ultrasound waves. The electrical signals are A / D converted by an A / D converter provided in the ultrasound probe or the like. The electrical signals after A / D conversion are called echo data. The echo data are transmitted to the medical data processing device 1 as raw data.

[0048] When the medical imaging device 3 is a PET device gantry, the gantry uses a coincidence measurement circuit to simultaneously measure a pair of 511 keV gamma rays generated in response to the annihilation of a positron emitted from a radionuclide accumulated in the subject with an electron surrounding the radionuclide, thereby generating digital data having digital values ​​related to the energy value and detection position of the pair of gamma rays (LOR (Line Of Response)). This digital data is called coincidence data or sinogram data. The coincidence data or sinogram data is transmitted to the medical data processing device 1 as raw data.

[0049] When the medical imaging device 3 is a C-arm of an X-ray diagnostic device, the C-arm generates X-rays from an X-ray tube provided in the C-arm. An X-ray detector such as an FPD (Flat Panel Display) provided in the C-arm or independently of the C-arm receives the X-rays generated from the X-ray tube and transmitted through the subject. The X-ray detector generates an electrical signal having a peak value corresponding to the detected X-ray dose, and performs signal processing such as A / D conversion on the electrical signal. The electrical signal after A / D conversion is called X-ray image data. The X-ray image data is transmitted to the medical data processing device 1 as raw data.

[0050] As shown in FIG. 5, the medical data processing device 1 has a processing circuit 11, a memory 13, an input interface 15, a communication interface 17, and a display 19 as hardware resources.

[0051] The memory 13 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit that stores various information. The memory 13 may be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a flash memory, in addition to an HDD or SSD. The memory 13 may also be a drive device that reads and writes various information from and to semiconductor memory elements such as a flash memory or a RAM (Random Access Memory). The storage area of ​​the memory 13 may be located within the medical data processing device 1 or in an external storage device connected via a network.

[0052] The memory 13 stores programs executed by the processing circuit 11, various data used in the processing of the processing circuit 11, etc. As the programs, for example, programs that are installed in advance on a computer from a network or a non-transitory computer-readable storage medium and cause the computer to realize each function of the processing circuit 11 are used. Note that the various data handled in this specification are typically digital data. The memory 13 is an example of a storage unit.

[0053] The memory 13 stores the trained model 90 generated by the model learning device 5. The trained model 90 is a neural network in which a parameter p has been trained so that input medical data can be input and desired output medical data can be output. The trained model 90 is an example of a trained neural network.

[0054] FIG. 6 is a diagram showing an example of a combination of inputs and outputs of the trained model 90. The trained model 90 accepts input of input data and outputs output data. For example, as shown in FIG. 6, the trained model 90 accepts input of input medical data and outputs output medical data. The input medical data and the output medical data are, for example, medical image data related to the subject to be processed. Instead of medical image data, raw data related to the subject to be processed may be used as the input medical data and the output medical data. The raw data is raw data related to the subject to be processed. Furthermore, input of a medical image may be accepted as the input medical data, and a classification result related to the medical image may be output as the output medical data.

[0055] Note that the raw data according to this embodiment is not limited to the original raw data collected by the medical imaging device 3. For example, the raw data according to this embodiment may be calculated raw data generated by performing a forward projection process on a medical image generated by the restoration function 112 or the inference function 114. Furthermore, the raw data according to this embodiment may be raw data obtained by performing any data processing, such as data compression, resolution decomposition, data interpolation, or resolution synthesis, on the original raw data. Furthermore, in the case of three-dimensional raw data, the raw data according to this embodiment may be hybrid data obtained by performing a restoration process on only one or two axes. Similarly, the medical image according to this embodiment is not limited to the original medical image generated by the restoration function 112 or the inference function 114. For example, the medical image according to this embodiment may be medical image obtained by performing any image processing, such as image compression, resolution decomposition, image interpolation, or resolution synthesis, on the original medical image.

[0056] The input interface 15 accepts various input operations from an operator, converts the accepted input operations into electrical signals, and outputs them to the processing circuitry 11. For example, the input interface 15 accepts input of medical information, input of various command signals, etc. from the operator. The input interface 15 is realized by a mouse, keyboard, trackball, switch buttons, a touch screen integrating a display screen and a touchpad, a non-contact input circuit using an optical sensor, a voice input circuit, etc., for performing various processes in the processing circuitry 11. The input interface 15 is connected to the processing circuitry 11 and converts input operations received from the operator into electrical signals and outputs them to the control circuit. Note that, in this specification, the input interface is not limited to those having physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs the electrical signals to the processing circuitry 11 is also an example of an input interface. The input interface 15 is an example of an input unit.

[0057] The communication interface 17 is a network interface that controls transmission of communications between the medical data processing device 1 and external devices via a network.

[0058] The display 19 displays various types of information. For example, the display 19 outputs medical information generated by the processing circuitry 11, a GUI (Graphical User Interface) for receiving various operations from an operator, etc. For example, the display 19 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 19 is an example of a display unit.

[0059] The processing circuitry 11 controls the overall operation of the medical data processing device 1. The processing circuitry 11 is a processor that executes an imaging control function 111, a restoration function 112, an acquisition function 113, an inference function 114, an image processing function 115, and a display control function 116 by calling and executing programs stored in the memory 13. While FIG. 5 illustrates the imaging control function 111, the restoration function 112, the acquisition function 113, the inference function 114, the image processing function 115, and the display control function 116 being implemented by a single processing circuitry 11, it is also possible to configure the processing circuitry by combining multiple independent processors, and have each processor execute a program to implement each function. Furthermore, the imaging control function 111, the restoration function 112, the acquisition function 113, the inference function 114, the image processing function 115, and the display control function 116 may each be implemented as a separate hardware circuit. The above description of the functions executed by the processing circuitry 11 also applies to the following embodiments and modifications.

[0060] Although the medical data processing device 1 is described as a single console that executes multiple functions, the multiple functions may be executed by separate devices. For example, the functions of the processing circuitry 11 may be distributed and installed in different devices.

[0061] The term "processor" used in the above description refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC, a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD)), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA). The processor realizes its function by reading and executing a program stored in memory 13. Note that instead of storing the program in memory 13, the program may be configured to be directly embedded in the processor circuit. In this case, the processor realizes its function by reading and executing the program embedded in the circuit. Note that each processor in this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its function. Furthermore, the multiple components in FIG. 5 may be integrated into a single processor to realize its function. The above description of "processor" also applies to the following embodiments and modified examples.

[0062] The processing circuitry 11 controls the medical imaging device 3 using the imaging control function 111 in accordance with imaging conditions to perform medical imaging on the subject. The imaging conditions according to this embodiment include the imaging principle of the medical imaging device 3 and various imaging parameters. The imaging principle corresponds to the type of medical imaging device 3, specifically, a magnetic resonance imaging device, an X-ray computed tomography device, a PET device, a SPECT device, and an ultrasonic diagnostic device. The imaging parameters include, for example, a field of view (FOV), an imaging region, a slice position, a frame (a time phase of a medical image), a time resolution, a matrix size, the presence or absence of a contrast agent, and the like. In the case of magnetic resonance imaging, the imaging parameters further include, for example, the type of imaging sequence, parameters such as time to repeat (TR), echo time (TE), and flip angle (FA), and the type of k-space filling trajectory. In the case of X-ray computed tomography, the imaging parameters further include X-ray conditions (tube current, tube voltage, X-ray exposure duration, etc.), scan type (non-helical scan, helical scan, synchronous scan, etc.), tilt angle, reconstruction function, number of views per rotation of the rotating frame, rotation speed, detector spatial resolution, etc. In the case of ultrasound diagnosis, the imaging parameters further include focal position, gain, transmission intensity, reception intensity, PRF, beam scanning method (sector scan, convex scan, linear scan, etc.), and scanning mode (B-mode scan, Doppler scan, color Doppler scan, M-mode scan, A-mode scan, etc.).

[0063] The processing circuitry 11 performs a restoration process on the raw data transmitted from the medical imaging device 3 using the restoration function 112 to restore a medical image. The restoration process according to this embodiment includes restoration from raw data to raw data, restoration from raw data to image data, and restoration from image data to image data. The restoration process from raw data defined in a coordinate system to two-dimensional image data or three-dimensional image data defined in another coordinate system is also called a reconstruction process or an image reconstruction process. The restoration process according to this embodiment includes, for example, denoising restoration and data error feedback restoration. For example, image reconstruction according to this embodiment can be classified into analytical image reconstruction and iterative image reconstruction. For example, analytical image reconstruction according to MR image reconstruction includes Fourier transform and inverse Fourier transform. Analytical image reconstruction according to CT image reconstruction includes filtered back projection (FBP), convolution back projection (CBP), or applications of these. Iterative image reconstruction includes expectation maximization (EM), algebraic reconstruction technique (ART), or applications of these. The processing circuit 11 that realizes the restoration function 112 is an example of an image generation unit.

[0064] The processing circuitry 11 acquires data to be processed using an acquisition function 113. In this embodiment, the data to be processed is medical data related to the subject to be processed. The medical data to be processed is, for example, medical image data obtained by medical imaging of the subject. The processing circuitry 11 that realizes the acquisition function 113 is an example of an acquisition unit.

[0065] The processing circuitry 11 uses the inference function 114 to apply the learned model 90 to the data to be processed to calculate inference data. In this embodiment, the processing circuitry 11 generates output medical data as inference data by applying the learned model 90 to input medical data related to the subject. The output medical data is, for example, a desired medical image generated by performing image processing on a medical image. The processing circuitry 11 that realizes the inference function 114 is an example of an inference unit.

[0066] The processing circuitry 11 performs various image processing using the image processing function 115 on the medical image generated by the restoration function 112, the output image generated by the inference function 114, etc. For example, the processing circuitry 11 performs three-dimensional image processing such as volume rendering, surface volume rendering, pixel value projection processing, MPR (Multi-Planer Reconstruction) processing, and CPR (Curved MPR) processing. The processing circuitry 11 may also perform registration processing as image processing.

[0067] The processing circuitry 11 displays various information on the display 19 using the display control function 116. For example, the processing circuitry 11 displays a medical image generated by the restoration function 112, an output image generated by the inference function 114, and a medical image processed by the image processing function 115.

[0068] Next, the operation of the inference process executed by the acquisition function 113 and the inference function 114 of the medical data processing device 1 will be described. The inference process is a process of applying the trained model 90 to medical input data and causing the trained model 90 to output desired output medical data. Note that the processing procedures in each process described below are merely examples, and each process can be modified as appropriate as possible. Furthermore, steps in the processing procedures described below can be omitted, replaced, or added as appropriate depending on the embodiment.

[0069] In the inference process, the processing circuitry 11 first acquires the trained model 90 from the memory 13 using the acquisition function 113, and then acquires the input medical data to be processed to which the trained model 90 is to be applied from the medical imaging device 3.

[0070] Next, the processing circuitry 11 applies the trained model 90 to the input medical data using the inference function 114. The trained model 90 accepts the input medical data and generates output medical data. The processing circuitry 11 acquires the output medical data generated by the trained model 90 as an inference result.

[0071] Next, the effects of the medical data processing device 1 according to this embodiment will be described.

[0072] The medical data processing device 1 according to this embodiment acquires processing target data and applies a trained model 90 to the processing target data to calculate inference data. The trained model 90 has an ensemble activation function. The ensemble activation function performs calculations based on multiple activation functions and multiple mixing coefficients corresponding to the multiple activation functions for each of multiple unit network structures that convert input vector elements into output vector elements. The unit network structures are, for example, nodes, layers, channels, etc. of a neural network. Note that one ensemble activation function may be provided for one node, or one ensemble activation function may be provided for multiple nodes. In other words, the unit network structure may be multiple nodes, multiple layers, multiple channels, etc.

[0073] Specifically, the ensemble activation function can apply multiple activation functions to the same input vector element to calculate multiple first output values, apply multiple mixing coefficients to each of the multiple first output values ​​to calculate multiple second output values, and calculate an output vector element based on the multiple second output values. For example, when performing the operation expressed by Equation (1) on output values ​​z1 and z2 of multiple activation functions A1 and A2, the ensemble activation function can perform an operation based on the multiple output values ​​z1 and z2 and multiple coefficients a1, a2, and a3 to obtain an output value y, and calculate an output vector element y_i based on the multiple output values ​​y. Here, coefficients a1, a2, and a3 correspond to the mixing coefficients, output values ​​z1 and z2 correspond to the first output value, and output value y corresponds to the second output value.

[0074] With the above configuration, the medical data processing device 1 of this embodiment prepares multiple activation functions as candidates for use in advance, and performs inference using a trained model 90 having an ensemble activation function that integrates the multiple prepared activation functions, thereby achieving higher inference performance than when the user arbitrarily selects the activation function to be used.

[0075] The medical image diagnostic apparatus 9 according to this embodiment includes the above-described medical data processing apparatus 1 and a medical imaging apparatus 3 that performs medical imaging on a subject. The medical data processing apparatus 1 can acquire medical data acquired by the medical imaging apparatus 3 as data to be processed. Then, the medical data processing apparatus 1 can apply the trained model 90 to the acquired medical data to calculate inference data.

[0076] With the above configuration, the medical image diagnostic device 9 according to this embodiment can apply the trained model 90, which has high inference performance by using an ensemble activation function, to desired inference processing for medical data acquired by the medical image diagnostic device 9. For example, when image processing for medical images acquired by the medical imaging device 3 is performed using the trained model 90, the inference performance can be improved compared to when the activation function used for the activation processing is arbitrarily selected by the user.

[0077] Next, an example of the configuration of the model learning device 5 according to this embodiment will be described. Fig. 7 is a diagram showing the configuration of the model learning device 5. As shown in Fig. 7, the model learning device 5 has, as hardware resources, a processing circuit 51, a memory 53, an input interface 55, a communication interface 57, and a display 59.

[0078] The processing circuitry 51 controls the overall operation of the model learning device 5. The processing circuitry 51 is a processor that executes an acquisition function 511 and a learning function 512 by calling and executing a program in the memory 53.

[0079] The processing circuit 51 acquires training data used for training the neural network using the acquisition function 511. The training data includes input training data and output training data (hereinafter referred to as correct output data). The processing circuit 51 that realizes the acquisition function 511 is an example of an acquisition unit.

[0080] The processing circuit 51 uses the learning function 512 to train the neural network based on the training data to generate a trained neural network. At this time, the processing circuit 51 updates the learning parameters for each unit network structure and the mixing coefficients of the ensemble activation function so as to minimize a loss function based on the error between the output data (hereinafter referred to as estimated output data) of the neural network based on the input training data and the correct output data. The processing circuit 51 that realizes the learning function 512 is an example of a learning unit. The learning parameters are parameters of a synthesis function that defines the neural network. The learning parameters are, for example, a weighting matrix, a bias, etc.

[0081] The memory 53 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit that stores various information. The memory 53 may be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a flash memory, in addition to an HDD or SSD. The memory 53 may also be a drive device that reads and writes various information from and to semiconductor memory elements such as flash memory and RAM (Random Access Memory). The storage area of ​​the memory 53 may be located within the model learning device 5 or in an external storage device connected via a network.

[0082] The memory 53 stores programs executed by the processing circuit 51, various data used in the processing of the processing circuit 51, etc. As the programs, for example, a program that is installed in advance on a computer from a network or a non-transitory computer-readable storage medium and causes the computer to realize each function of the processing circuit 51 is used. Note that the various data handled in this specification are typically digital data. The memory 53 is an example of a storage unit.

[0083] The memory 53 stores, for example, a model learning program 50 for training a neural network. The memory 53 also temporarily stores training data used in training the neural network. The memory 53 also stores multiple activation functions that are candidates for use in an ensemble activation function.

[0084] The input interface 55 accepts various input operations from the operator, converts the accepted input operations into electrical signals, and outputs them to the processing circuitry 51. For example, the input interface 55 accepts input of medical information, input of various command signals, etc. from the operator. The input interface 55 is realized by a mouse, keyboard, trackball, switch buttons, a touch screen integrating a display screen and a touchpad, a non-contact input circuit using an optical sensor, a voice input circuit, etc., for performing various processes in the processing circuitry 11. The input interface 55 is connected to the processing circuitry 51 and converts input operations received from the operator into electrical signals and outputs them to the control circuitry. Note that, in this specification, the input interface is not limited to those having physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs the electrical signals to the processing circuitry 51 is also an example of an input interface. The input interface 55 is an example of an input unit.

[0085] The communication interface 57 is a network interface that controls transmission of communications between the medical data processing device 1 and external devices via a network.

[0086] The display 59 displays various types of information. For example, the display 59 outputs medical information generated by the processing circuitry 51, a GUI (Graphical User Interface) for receiving various operations from an operator, and the like. For example, the display 59 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 59 is an example of a display unit.

[0087] Next, a description will be given of the model learning process that the processing circuitry 51 of the model learning device 5 executes in accordance with the model learning program 50. The model learning process is a process of training a machine learning model based on learning data to generate a trained model 90.

[0088] In the model learning process, the processing circuit 51 first acquires learning data including multiple learning samples from the learning data storage device 7 using the acquisition function 511. The learning samples include a combination of input learning data and correct output data. The correct output data is data that is desired to be output from the neural network when the input learning data is input to the neural network. The correct output data may also be called teacher data. The input learning data and the correct output data are, for example, medical images generated by imaging a patient. The input learning data and the correct output data may also be medical images generated by imaging an arbitrary phantom.

[0089] Next, the processing circuit 51 generates estimated output data by forward propagation of the neural network based on the input training data using the learning function 512. Note that the parameters of the neural network are set to initial values ​​in the first forward propagation. Next, the processing circuit 51 calculates the error between the generated estimated output data and the input correct output data using the learning function 512. Next, the processing circuit 51 calculates a gradient vector by backpropagating the neural network based on the calculated error using the learning function 512. Next, the processing circuit 51 updates the parameters of the entire neural network, including the mixture coefficients, based on the calculated gradient vector using the learning function 512.

[0090] For example, if the ensemble activation function is a function that performs the calculation of the aforementioned formula (1) to integrate the activation functions A1 and A2, training the neural network updates the learning parameters and updates each of the coefficients a1-a3 in formula (1) as mixing coefficients. Also, if the ensemble activation function is a function that performs the calculation of the aforementioned formula (2) to integrate the activation functions A1 and A2, training the neural network updates the learning parameters and updates each of the coefficients a1-a6 in formula (2) as mixing coefficients.

[0091] The processing circuitry 51 determines whether a termination condition is satisfied. The termination condition may be set, for example, when the number of iterations reaches a specified number. Alternatively, the termination condition may be set when the gradient vector becomes less than a threshold. If the termination condition is not satisfied, the processing circuitry 51 repeats the above process using the same training sample or another training sample. Then, if the termination condition is satisfied, the processing circuitry 51 outputs the updated neural network as a trained model 90. The trained model 90 is stored in the memory 13 of the medical data processing device 1.

[0092] This concludes the description of the model learning process by the model learning device 5 according to this embodiment. Note that the flow of the learning process described above is an example, and this embodiment is not limited to this.

[0093] As described above, the model learning program 50 according to this embodiment causes the model learning device 5 to execute the learning function 512. The learning function 512 applies the input training data to a neural network having an input layer that receives the input training data, an output layer that outputs output data corresponding to the input training data, and at least one intermediate layer provided between the input layer and the output layer to generate estimated output data. The learning function 512 then updates the parameters of the neural network, including multiple mixture coefficients, so that the estimated output data approximates the ground truth output data.

[0094] Furthermore, the model learning device 5 according to this embodiment can acquire learning data including input learning data and output learning data, and train a neural network based on the learning data to generate a trained neural network. The neural network has a plurality of unit network structures with learning parameters that convert input vector elements into output vector elements. The neural network also has, for each of the plurality of unit network structures, an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients corresponding to the plurality of activation functions. The model learning device 5 can update the learning parameters and the plurality of mixing coefficients so as to minimize the error between the output data of the neural network based on the input learning data and the output learning data.

[0095] With the above configuration, this embodiment allows the activation function itself to be generated by machine learning, which makes it possible to set optimal parameters as the mixture coefficients of the ensemble activation function, thereby realizing an inference device with further improved inference performance.

[0096] (First Modification) A first modification of the embodiment will be described. This modification is obtained by modifying the configuration of the embodiment as follows. Descriptions of the configuration, operation, and effects that are the same as those of the embodiment will be omitted.

[0097] In this modification, a neural network is used that outputs multiple output data for one input data. The neural network is configured to receive, for example, medical images as input data and output multiple medical images that have been subjected to different image processing.

[0098] Figure 8 is a diagram showing an outline of the operation of the ensemble activation function according to this modification. In the example shown in Figure 8, the ensemble activation function is composed of six activation functions A1-A6 and two mixing functions M1 and M2. In Figure 8, the activation functions A1-A6 are represented as "Act.1"-"Act.6."

[0099] The activation functions A1-A3 are different types of activation functions. The activation functions A4-A6 are different types of activation functions. The types of activation functions used as the activation functions A1-A3 and the types of activation functions used as the activation functions A4-A6 may be the same or different.

[0100] The input vector element x_i input to the ensemble activation function is applied to each activation function A1-A6. Each activation function A1-A6 accepts each value x of the input vector element x_i as input and outputs the result of applying the activation function to each value x as an output value z1-z6.

[0101] The mixing function M1 combines the three output values ​​z1-z3 obtained from the activation functions A1-A3 according to the mixing coefficients to calculate a single output value y1. The mixing function M2 combines the three output values ​​z4-z6 obtained from the activation functions A4-A6 according to the mixing coefficients to calculate a single output value y2.

[0102] The activation functions A1-A6 and mixing functions M1 and M2 are applied sequentially to each value x contained in the input vector element x_i, and output values ​​y1 and y2 for each value x contained in the input vector element x_i are output. The ensemble activation function generates output vector elements y1_i containing the output value y1 and y2_i containing the output value y2, and these are the outputs of the ensemble activation function.

[0103] Alternatively, a neural network that outputs multiple output data for a single input data can be configured by applying multiple activation functions to the input data, using the outputs of each activation function as multiple channels for subsequent inputs, and repeating the convolution and activation processes described above to output multiple output data. For example, if the number of input data channels input to each activation function is 10 and there are three activation functions, the number of channels input to the next convolution process will be 30. In this case, the ensemble calculation in the ensemble activation function becomes part of the coefficients of the next convolution process. Even in this case, the same effects as those of the above-described embodiment can be obtained.

[0104] (Second Modification) A second modification of the embodiment will be described. This modification is obtained by modifying the configuration of the embodiment as follows. Descriptions of the configuration, operation, and effects that are the same as those of the embodiment will be omitted.

[0105] The ensemble activation function according to this modification is a function that applies multiple mixing coefficients to the same input vector element to calculate multiple first output values, applies multiple activation functions to each of the multiple first output values ​​to calculate multiple second output values, and calculates an output vector element based on the multiple second output values.

[0106] FIG. 9 is a diagram illustrating the outline of the operation using the ensemble activation function according to this modification. Here, as an example, we will explain the case where the ensemble activation function is a function that combines activation functions A1, A2, and A3. The ensemble activation function calculates an output value z'1 (= b1·x) by multiplying the input value x input to activation function A1 by a coefficient b1 in advance, calculates an output value z'2 (= b2·x) by multiplying the input value x input to activation function A2 by a coefficient b2 in advance, and calculates an output value z'3 (= b3·x) by multiplying the input value x input to activation function A3 by a coefficient b3 in advance. Each of the output values ​​z'1-z'3 corresponds to a first output value. Each of the coefficients b1-b3 corresponds to a mixing coefficient.

[0107] Next, the ensemble activation function applies activation function A1 to output value z'1, activation function A2 to output value z'2, and activation function A3 to output value z'3. The ensemble activation function then applies mixing function M to output value z1 of activation function A1, output value z2 of activation function A2, and output value z3 of activation function A3 to calculate output value y. Output values ​​z1, z2, and z3 correspond to the second output value. Mixing function M may be a linear function with output values ​​z1, z2, and z3 as variables, or a quadratic function, or a polynomial of degree 3 or higher.

[0108] (Third Modification) A third modification of the embodiment will be described. This modification is obtained by modifying the configuration of the embodiment as follows. Descriptions of the configuration, operation, and effects that are the same as those of the embodiment will be omitted.

[0109] The ensemble activation function according to this modification is a function that calculates output vector elements by applying a function reconstructed based on a plurality of mixing coefficients and a plurality of activation functions to input vector elements.

[0110] Here, as an example, we will explain the case where the ensemble activation function is a function obtained by integrating activation functions A1 and A2. The ensemble activation function is calculated using a function reconstructed in advance by multiplying activation function A1 by coefficient c1, and a function reconstructed in advance by multiplying activation function A2 by coefficient c2. Here, coefficients c1 and c2 each correspond to a mixing coefficient.

[0111] The ensemble activation function applies each of the reconstructed activation functions A1 and A2 to the input value x, and applies the above-mentioned mixing function M to the output value z1 of the reconstructed activation function A1 and the output value z2 of the reconstructed activation function A2 to calculate the output value y. The mixing function M may be a linear function with the output values ​​z1 and z2 as variables, a quadratic function, or a polynomial of degree 3 or higher.

[0112] (Fourth Modification) A fourth modified example of the embodiment will be described. This modified example is obtained by modifying the configuration of the embodiment as follows. Descriptions of the configuration, operation, and effects that are the same as those of the embodiment will be omitted.

[0113] In the neural network according to this modification, a regularization technique is incorporated as a measure against overfitting.

[0114] An example of a regularization method is to set all mixture coefficients to the same value. For example, if the ensemble activation function is a function that integrates the output values ​​z1 and z2 of the activation functions A1 and A2 using the above-mentioned formula (1), overlearning can be suppressed by setting the coefficients a1 to a3 of all ensemble activation functions provided for each unit network structure to the same value.

[0115] Alternatively, the ensemble activation functions present in each network structure may be grouped in advance, and the mixing coefficients of all ensemble activation functions belonging to each group may be set to the same value.

[0116] In addition, the mixing coefficients corresponding to the same activation function may be the same across multiple ensemble activation functions in a neural network. In this case, the mixing coefficients (e.g., a1 in Equation (1)) corresponding to the same type of activation function (e.g., activation function A1) are set to the same value across all ensemble activation functions.

[0117] In this embodiment, an ensemble activation function is generated for each unit network structure by training the entire neural network, and therefore the generated ensemble activation functions may be different or identical for each unit network structure.

[0118] Another example of a regularization technique is to add a cost function to a loss function based on the error between estimated output data and input ground truth output data, the cost function decreasing as the difference between multiple mixing coefficients decreases. For example, the sum of squares of the mixing coefficients is added as the cost function. This allows each parameter to be optimized to minimize the difference between multiple mixing coefficients when updating the learning parameters and mixing coefficients to minimize the error between estimated output data and ground truth output data, thereby suppressing overfitting.

[0119] Another regularization technique is transfer learning using unintended training data. In this method, the learning parameters and mixture coefficients of a trained neural network generated by training the neural network based on unintended training data are used as initial values, and the neural network is trained using normal training data. When the normal training data is medical image data, for example, a trained model is generated by training the neural network using image data from a general field other than the medical field as unintended training data. Then, the learning parameters and mixture coefficients of the generated trained model are used as initial values, and the neural network is trained using the medical image data as training data, thereby suppressing a decline in generalization performance due to overfitting and improving inference performance.

[0120] Other regularization techniques include, for example, data augmentation, a method of adding an L2 regularization term for the learning parameters and / or the mixing coefficients to the loss function, a method of performing sparse regularization by adding an L2 regularization term for the learning parameters and / or the mixing coefficients to the loss function, etc. Furthermore, a method of performing L2 regularization or L1 regularization on the differences between the learning parameters and the mixing coefficients after applying the above-mentioned transfer learning may also be used.

[0121] The regularization method is not limited to the above-described method, and various commonly used regularization methods may be used. Also, a combination of the above methods may be used.

[0122] (Other variations) The neural network according to this embodiment can also be applied to a complex network. Furthermore, the neural network according to this embodiment can also be applied to an unbiased neural network. For example, in an unbiased complex network that denoises complex images such as MRI images, the ensemble activation function according to this embodiment can be applied instead of an activation function.

[0123] The neural network according to this embodiment is also useful for an inference device that processes complex data. Different activation functions or neural networks may be used depending on whether the acquired data to be processed is complex or not. For example, the neural network using the ensemble activation function according to this embodiment may be used to perform inference processing only when the acquired data to be processed is MRI image data acquired by a magnetic resonance imaging device or echo data acquired by an ultrasound diagnostic device.

[0124] In this embodiment, a neural network applied to medical data has been described as an example. However, the ensemble activation function of this embodiment can also be applied to a neural network that performs image recognition on general images. An example of such a neural network is a DNN that outputs whether an object displayed in an image is a "cat," "dog," "horse," "cow," etc. To train this neural network, for example, an image displaying some kind of animal is used as input training data, and a one-hot vector indicating the type of animal, such as "cat," "dog," "horse," or "cow," is used as correct output data.

[0125] According to at least one of the embodiments described above, it is possible to improve the inference performance of a trained neural network that uses an activation function.

[0126] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0127] 100...Medical data processing system 1...Medical data processing device 3...Medical imaging equipment 5...Model learning device 7...Learning data storage device 9...Medical imaging diagnostic equipment 11...Processing circuit 111...Image capture control function 112...Restore function 113...Acquisition function 114...Inference function 115...Image processing function 116...Display control function 13...Memory 15...Input interface 17...Communication interface 19...Display 50...Model Learning Program 51...Processing circuit 511...retrieval function 512…Learning function 53...Memory 55...Input interface 57...Communication interface 59...Display 90...Trained model A1-A6...Activation functions M, M1, M2...Mixing functions a1-a6, b1-b3, c1-c2...coefficients y1-y2, z1-z6, z'1-z'3...output value

Claims

1. an acquisition unit that acquires data to be processed; an inference unit that applies a trained neural network to the processing target data to calculate inference data; the trained neural network has an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the plurality of activation functions for each of a plurality of unit network structures that converts input vector elements into output vector elements; The ensemble activation function applies the plurality of mixing coefficients to the same input vector element to calculate a plurality of first output values, applies the plurality of activation functions to the plurality of first output values ​​respectively to calculate a plurality of second output values, and calculates the output vector element based on the plurality of second output values. Reasoning device.

2. The inference device according to claim 1; a medical imaging device for performing medical imaging on a subject, the acquiring unit acquires medical data acquired by the medical imaging device as the processing target data; Medical imaging diagnostic equipment.

3. an acquisition step of acquiring data to be processed; an inference step of calculating inference data by applying a trained neural network to the processing target data, the trained neural network has an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the plurality of activation functions for each of a plurality of unit network structures that converts input vector elements into output vector elements; The ensemble activation function applies the plurality of mixing coefficients to the same input vector element to calculate a plurality of first output values, applies the plurality of activation functions to the plurality of first output values ​​respectively to calculate a plurality of second output values, and calculates the output vector element based on the plurality of second output values. Reasoning method.

4. an acquisition step of acquiring training data including input training data and output training data; a learning step of training a neural network based on the learning data to generate a trained neural network; The neural network A plurality of unit network structures with learning parameters that convert input vector elements into output vector elements; an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the plurality of activation functions for each of the plurality of unit network structures; the learning step updates the learning parameters and the plurality of mixture coefficients so as to minimize an error between output data of the neural network based on the input learning data and the output learning data; the learning step includes a step of setting mixing coefficients corresponding to the same activation function to the same value in the plurality of ensemble activation functions included in the neural network; How to generate trained neural networks.

5. An acquisition step of acquiring training data including input training data and output training data; a learning step of training a neural network based on the learning data to generate a trained neural network; The neural network A plurality of unit network structures with learning parameters that convert input vector elements into output vector elements; an ensemble activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the plurality of activation functions for each of the plurality of unit network structures; the learning step updates the learning parameters and the plurality of mixture coefficients so as to minimize an error between output data of the neural network based on the input learning data and the output learning data; the learning step includes a step of adding a cost function to a loss function, the cost function having a smaller value as the difference between the plurality of mixing coefficients decreases. How to generate trained neural networks.

6. the learning step includes a step of training the neural network based on unintended learning data using learning parameters and mixture coefficients of a trained neural network generated by training the neural network based on the unintended learning data as initial values, and executing the training of the neural network based on the unintended learning data.

6. The trained neural network generating method according to claim 4 or 5.

7. The ensemble activation function applies the plurality of activation functions to the same input vector element to calculate a plurality of first output values, applies the plurality of mixing coefficients to the plurality of first output values, respectively, to calculate a plurality of second output values, and calculates the output vector element based on the plurality of second output values.

6. The trained neural network generating method according to claim 4 or 5.

8. The ensemble activation function applies the plurality of mixing coefficients to the same input vector element to calculate a plurality of first output values, applies the plurality of activation functions to the plurality of first output values ​​respectively to calculate a plurality of second output values, and calculates the output vector element based on the plurality of second output values.

6. The trained neural network generating method according to claim 4 or 5.

9. The ensemble activation function calculates the output vector elements by applying a function reconstructed based on the plurality of mixing coefficients and the plurality of activation functions to input vector elements.

6. The trained neural network generating method according to claim 4 or 5.

Images