Inference device and method, medical image diagnosis device, and learning complete neural network generation method

Abstract

The present application provides a reasoning device and method, a medical image diagnostic device, and a learning completed neural network generation method. The problem is to improve the reasoning performance of a learning completed neural network using an activation function. The solution is that the reasoning device according to the embodiments includes an acquisition unit and a reasoning unit. The acquisition unit acquires processing target data. The reasoning unit applies a learning completed neural network to the processing target data to calculate reasoning data. The learning completed neural network has a fusion activation function that performs an operation based on a plurality of activation functions and a plurality of mixing coefficients corresponding to the plurality of activation functions, respectively, for each unit network configuration in a plurality of unit network configurations that transform input vector elements into output vector elements.

Description

[0001] Related applications:

[0002] This application claims priority based on Japanese Patent Application 2021-178506, filed on November 1, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments described herein generally relate to inference devices, medical image diagnostic devices, inference methods, and methods for generating neural networks after learning. Background Technology

[0004] Among neural networks that perform desired inference on medical data such as medical image data or its raw data, there are methods that utilize deep neural networks (DNNs) or convolutional neural networks (CNNs).

[0005] In such neural networks, various activation functions (activation functions) are explored as the functions used for activation processing. Examples of activation functions include the logistic sigmoid function, the hyperbolic tangent function (tanh), the normalized linear function (ReLU: Corrected Linear Unit), linear mapping, identity mapping, the Maxout function, ELU, LeakyReLU, Complex ReLU, etc. However, each of these activation functions has its advantages and disadvantages in terms of inference performance.

[0006] Previously, the type of activation function used in neural networks was determined by the designer. Therefore, the designer needed to know in advance the optimal type of activation function corresponding to the intended use or optimization method. On the other hand, it is known that the choice of activation function significantly impacts the inference performance of the neural network. Therefore, it is desirable to use an appropriate activation function that improves the inference performance of the neural network.

[0007] Existing technical documents:

[0008] Non-Patent Literature 1: Hidenori Takeshima, “Deep Learning and Its Application to Function Approximation for MR in Medicine: An Overview”, [Online], September 17, 2021, Magnetic Resonance in Medical Sciences, [Retrieved October 15, 2021], Internet <URL: https: / / doi.org / 10.2463 / mrms.rev.2021-0040> Summary of the Invention

[0009] The problem that the invention aims to solve:

[0010] One of the problems to be solved by the embodiments disclosed in this specification and accompanying drawings is to improve the inference performance of learning-completed neural networks using activation functions. However, the problems to be solved by the embodiments disclosed in this specification and accompanying drawings are not limited to the above-mentioned problems. Problems corresponding to the effects of the various configurations shown in the embodiments described below can also be identified as other problems.

[0011] The inference apparatus according to the embodiment includes an acquisition unit and an inference unit. The acquisition unit acquires processing object data. The inference unit applies a learned neural network to the processing object data to calculate inference data. The learned neural network has an ensemble activity function, which performs calculations based on multiple activation functions and multiple mixing coefficients corresponding to each of the multiple unit network structures that transform input vector elements into output vector elements.

[0012] Invention effects:

[0013] The purpose of this invention is to improve the inference performance of learning-complete neural networks that use activation functions. Attached Figure Description

[0014] Figure 1 This is a diagram showing the general structure and processing of the medical data processing system to which the medical data processing device involved in the embodiment belongs.

[0015] Figure 2 This is a diagram illustrating the construction of the neural network involved in the implementation method.

[0016] Figure 3 This is a diagram that schematically illustrates the case of operations based on fusion activation functions involved in the implementation method.

[0017] Figure 4 This is a diagram illustrating the configuration of the convolutional layer involved in the implementation method.

[0018] Figure 5 This is a diagram illustrating the configuration of the medical image diagnostic device according to the embodiment.

[0019] Figure 6 This is a diagram illustrating an example of the combination of inputs and outputs of a learning-complete neural network involved in the implementation method.

[0020] Figure 7 This is a diagram illustrating the configuration of the model learning device involved in the implementation method.

[0021] Figure 8 This diagram is a schematic representation of the operation based on the fusion activation function involved in the first variation of the implementation.

[0022] Figure 9 This is a diagram that schematically illustrates the case of the operation based on the fusion activation function involved in the second variation of the implementation.

[0023] Explanation of reference numerals in the attached figures:

[0024] 100... Medical Data Processing System

[0025] 1...Medical data processing device

[0026] 3...Medical camera devices

[0027] 5... Model Learning Device

[0028] 7... Learning Data Storage Device

[0029] 9...Medical image diagnostic device

[0030] 11……Processing Circuit

[0031] 111……Camera control function

[0032] 112……Restore function

[0033] 113……Obtain Function

[0034] 114... reasoning function

[0035] 115... Image processing function

[0036] 116……Display control function

[0037] 13... Memory

[0038] 15……Input Interface

[0039] 17……Communication Interface

[0040] 19... Monitor

[0041] 50……Model Learning Program

[0042] 51……Processing Circuit

[0043] 511... Obtain Function

[0044] 512... Learning Function

[0045] 53... Memory

[0046] 55……Input Interface

[0047] 57……Communication Interface

[0048] 59... Monitor

[0049] 90……Complete learning model

[0050] A1-A6……Activation function

[0051] M, M1, M2... mixture functions

[0052] coefficients a1-a6, b1-b3, c1-c2...

[0053] Output values ​​of y1-y2, z1-z6, z'1-z'3, etc. Detailed Implementation

[0054] The inference apparatus according to the embodiment includes an acquisition unit and an inference unit. The acquisition unit acquires processing object data. The inference unit applies a learned neural network to the processing object data to calculate inference data. The learned neural network has a fusion activation function that performs calculations based on multiple activation functions and multiple mixing coefficients corresponding to each of the multiple unit network structures that transform input vector elements into output vector elements.

[0055] Hereinafter, with reference to the accompanying drawings, embodiments of the inference apparatus and the method for generating a learning-complete neural network will be described in detail. The inference apparatus is a device that uses a learning-complete neural network to calculate inference data for data to be processed. Typically, the inference apparatus is a medical data processing device that takes input medical data of the processing object as input and outputs corresponding output medical data. The input medical data corresponds to the processing object data. Furthermore, the inference apparatus is not limited to the medical field and can also be used for inference processing of data outside the medical field. In the following embodiments, the case where the inference apparatus processes medical data will be described. In the following description, the same reference numerals are used for constituent elements having substantially the same function and structure, and descriptions will be repeated only where necessary.

[0056] (Implementation Method)

[0057] Figure 1 This diagram illustrates the configuration and processing of the medical data processing system 100, which is part of the medical data processing device 1 that serves as a reasoning device according to this embodiment. (See diagram for details.) Figure 1 As shown, the medical data processing system 100 involved in this embodiment includes a medical data processing device 1, a medical camera device 3, a model learning device 5, and a learning data storage device 7.

[0058] The learning data storage device 7 stores learning data including multiple learning samples. For example, the learning data storage device 7 is a computer with a built-in storage device. Alternatively, the learning data storage device 7 can also be a large-capacity storage device that can be connected to the computer via a cable or communication network in a communicative manner. As such a storage device, HDD (Hard Disk Drive), SSD (Solid State Drive), integrated circuit storage devices, etc., can be appropriately utilized.

[0059] The model learning device 5, based on the learning data stored in the learning data storage device 7, performs machine learning according to the model learning program, generating a machine learning model that has completed learning (hereinafter referred to as the completed learning model). The model learning device 5 is a computer such as a workstation equipped with processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The model learning device 5 and the learning data storage device 7 can be connected in a communicative manner via cables or a communication network, or the learning data storage device 7 can be mounted on the model learning device 5. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via cables or a communication network. Alternatively, the model learning device 5 and the learning data storage device 7 may not be connected in a communicative manner. In this case, learning data is supplied from the learning data storage device 7 to the model learning device 5 via a portable storage medium storing the learning data. The model learning device 5 is an example of a learning device.

[0060] The machine learning model involved in this embodiment is a parameterized synthesis function obtained by synthesizing multiple functions. The parameterized synthesis function is defined by a combination of multiple adjustable functions and parameters. The machine learning model involved in this embodiment can be any parameterized synthesis function that satisfies the above requirements, but it is set as a multi-layered network model (hereinafter referred to as a multi-layered network). For example, a deep neural network (DNN) or a convolutional neural network (CNN) with convolutional layers can be used as a multi-layered network. In the following description, the multi-layered network in this embodiment is set as a neural network. The neural network involved in this embodiment takes the input medical data of the processing object acquired by the medical imaging device 3 as input and outputs corresponding output medical data.

[0061] The medical imaging device 3 generates medical data of the object being processed. The medical data involved in this embodiment conceptually includes: raw data collected by the medical imaging device 3 or other medical imaging devices through medical imaging of the subject, or medical image data generated by restoring the raw data. The medical imaging device 3 can be any medical device as long as it can generate medical data. For example, the medical imaging device 3 involved in this embodiment can be a single medical device such as a magnetic resonance imaging device (MRI device), a computed tomography (CT) device, an X-ray diagnostic device, a PET (positron emission tomography) device, a SPECT (single photon emission CT) device, or an ultrasound diagnostic device, or it can be a composite medical device such as a PET / CT device, a SPECT / CT device, a PET / MRI device, or a SPECT / MRI device.

[0062] The medical data processing device 1 uses a learned model obtained by the model learning device 5 according to a model learning program to generate output medical data corresponding to the input medical data of the processing object collected by the medical imaging device 3. The medical data processing device 1 and the model learning device 5 can be connected in a communicable manner via cables or a communication network, or they can be installed in a single computer. In this case, the learned model is supplied from the model learning device 5 to the medical data processing device 1 via cables or a communication network. The medical data processing device 1 and the model learning device 5 may not necessarily be connected in a communicable manner. In this case, the learned model is supplied from the model learning device 5 to the medical data processing device 1 via a portable storage medium or the like storing the learned model. The supply of the learned model can occur at any time between the manufacture of the medical data processing device 1 and its installation in a medical facility, or during maintenance; it can be done at any time. The supplied learned model is stored in the medical data processing device 1. In addition, the medical data processing device 1 can be a computer mounted in a medical image diagnostic device equipped with a medical camera device 3, a computer that is connected to the medical image diagnostic device via a cable or network in a communicable manner, or a computer that is independent of the medical image diagnostic device.

[0063] The following describes a typical configuration of the neural network involved in this embodiment. Figure 2This diagram illustrates a typical configuration of the neural network according to this embodiment. Here, the neural network is a network with a hierarchical structure where connections are made only between adjacent layers, and information propagates unidirectionally from the input layer to the output layer. The neural network according to this embodiment is a forward propagation network where image data input to the input layer propagates from the input layer to the output layer only between adjacent layers.

[0064] The neural network involved in this embodiment is as follows: Figure 2 As shown, the neural network is assumed to consist of L layers: an input layer (l=1), intermediate layers (l=2, 3, ..., L-1), and an output layer (l=L). Furthermore, the following is an example; the structure of the neural network is not limited to the description below.

[0065] Input data is input into the input layer (layer 1). Input data may include image data such as medical image data, raw data (RAW data) such as k-space data or projection data, etc. In the input layer, the input data is processed as is and becomes the output data.

[0066] In the intermediate layers (l = 2, 3, ..., L-1) following the input layer, the output is calculated by sequentially performing calculations based on the weighting matrix between each layer, the bias of each layer, and the activation process of each layer.

[0067] In the output layer (Lth layer) that follows the intermediate layer, the data input from the intermediate layer becomes the output data exactly as it is.

[0068] The neural network involved in this embodiment is a forward propagation network in which data input to the input layer propagates from the input layer side to the output layer side only between adjacent layers. This forward propagation network is defined as a synthesis function composed of the linear relationships between layers using a weighting matrix W, the nonlinear relationships (or linear relationships) within each layer using activation processing, and the biases. Specifically, the weighting matrix and the biases are referred to as the network parameters p. The synthesis function defined in this way changes its form as a function depending on how the parameter p is chosen. Therefore, the neural network involved in this embodiment can be defined as a function that enables its output layer to output the desired result by appropriately selecting the parameter p of the synthesis function.

[0069] A neural network consists of multiple unit network structures. Each unit network structure is a unit that transforms input vector elements into output vector elements, constituting the network. Unit network structures are analogous to nodes, layers, channels, and cells. For example, in each unit network structure, input vector elements, including the output values ​​of other unit network structures, are taken as input. A value is calculated by applying different weights to each value of the input vector element, adding a bias 'b', and then applying an activation process to the calculated value. In the activation process, a single value is output, obtained by applying a non-linear (or linear) transformation to the calculated value. This calculated value is then output as the output of the unit network structure.

[0070] In this embodiment, a new function called a fusion activation function is used in the activation process. The fusion activation function is generated by preparing multiple activation functions and incorporating a fusion portion of them into the learning process. Specifically, the fusion activation function applies multiple activation functions to the same data and outputs the values ​​obtained by processing the data according to each activation function and a mixing coefficient.

[0071] A fusion activation function is, for example, a function that applies multiple activation functions to the same data and combines the output values ​​of the multiple activation functions according to the mixing coefficient to output the combined output value.

[0072] As an activation function, various functions can be selected depending on the purpose, such as the logistic sigmoid function, the hyperbolic tangent function (tanh), the normalized linear function (ReLU: Corrected Linear Unit), linear mapping, identity mapping, and the Maxout function. Alternatively, ELU, LeakyReLU, Complex ReLU, etc., can also be used as activation functions.

[0073] Mixing coefficients are weighted parameters set according to each activation function to be integrated. For example, in training a neural network to output target data as input, the mixing coefficients are determined by optimization along with other weighted parameters included in the neural network.

[0074] The following is for reference Figure 3 This describes the fusion activation function involved in this implementation. Figure 3 This is a diagram that roughly illustrates the operation based on the fusion activation function involved in this embodiment. For example... Figure 3 As shown, the fusion activation function consists of three activation functions A1-A3 and a blending function M. Activation functions A1-A3 are different types of activation functions. Figure 3In this context, activation functions A1-A3 are represented as "Act.1"-"Act.3". The number of activation functions used in the fusion activation function can be either two or more.

[0075] The fusion activation function accepts input vector elements x_i. The input vector element x_i, for example, represents a vector composed of multiple input values ​​x from the output of a specific channel i that constitutes the CNN.

[0076] The input vector element x_i is applied to each activation function A1-A3. Activation function A1 outputs the result of the operation performed on each value x of the input vector element x_i as input, as output value z1. Similarly, activation functions A2 and A3 output the results of the operation performed on each value x of the input vector element x_i as input, as output values ​​z2 and z3, respectively.

[0077] The blending function M is a function that takes the output values ​​from multiple pre-prepared activation functions as input and outputs a single output value obtained by combining the multiple output values. Figure 3 In one example, the mixing function M calculates the output value y by combining the three output values ​​z1-z3 obtained from the activation functions A1-A3 according to the mixing coefficients.

[0078] The mixture function M can be either a linearly weighted sum of multiple inputs or a non-linearly weighted sum of multiple inputs. Furthermore, the mixture function M can be a first-order function, a second-order function, or a function expressed as a polynomial of degree 3 or higher, using each of the multiple inputs as variables. Additionally, the kernel function used in kernel tricks can also be used as the mixture function M. Furthermore, the mixture function M can also be a neural network that represents a parameterized synthesis function obtained by combining multiple functions.

[0079] For example, when there are two types of activation functions forming the fusion activation function, the mixing function M combines the output values ​​z1 and z2 of the two pre-prepared activation functions A1 and A2 to calculate the output value y. Equation (1) represents an example of the calculation formula for the output value y when a first-order function is used as the mixing function M. In Equation (1), the coefficients a1-a3 are each the mixing coefficients in this embodiment. The coefficients a1-a3 are determined by being optimized along with other parameters during the optimization of the neural network as a whole. In Equation (1), the mixing function M is a function that calculates the value y obtained by multiplying the output values ​​z1 and z2 of activation functions A1 and A2 by different coefficients and then adding them together.

[0080] y = a1z1 + a2z2 + a3 (1)

[0081] Equation (2) represents an example of the calculation formula for the output value y when a quadratic function is used as the mixing function M. In Equation (2), the coefficients a1-a6 are each the mixing coefficients in this embodiment. The coefficients a1-a6 are determined by being optimized together with other parameters during the optimization of the neural network as a whole.

[0082] y = a1z1 2 +a2z1z2+a3z2 2 +a4z1+a5z2+a6 (2)

[0083] The activation function and blending function M described above are applied sequentially to each value x included in the input vector feature x_i, and output values ​​y corresponding to each value x included in the input vector feature x_i. The fusion activation function generates an output vector feature y_i that includes the output values ​​y output relative to each value x included in the input vector feature x_i, and uses the generated output vector feature y_i as the output of the fusion activation function.

[0084] Next, we will explain an example of using a CNN as a neural network. A CNN consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. In a CNN, for example, before the fully connected layers, there are multiple processing blocks configured with pooling layers following two convolutional layers. The number of connections and the connection order of the convolutional and pooling layers should be appropriately set.

[0085] The input layer is fed input data. This input data is typically vector data. During assembly, the input data can be read either as a set of input data in memory (e.g., per channel) or as each feature of input data comprising multiple channels. In the convolutional layers, convolutional processing is performed on the input data from the input layer. In the pooling layers, max pooling is performed on the convolutionally processed data. In the fully connected layers, the data processed by the processing blocks is fully connected to the channels of the fully connected layers. In the output layer, output data is generated as the final output of the CNN.

[0086] Within convolutional layers, convolution, regularization, and activation are performed. However, regularization and activation are not mandatory and can be omitted. If activation is not performed in the convolutional layers, it is performed in other layers. Figure 4 This is a diagram that roughly represents the typical structure of a convolutional layer. Figure 4 In this context, convolution is represented as "Conv." and activation is represented as "Act.". Figure 4 In this example, regularization was omitted.

[0087] In convolutional processing, the input data in the input layer is convolved for each channel. For example, in convolutional processing, one kernel (filter) is used for each channel, and the data after convolution is used to generate a feature map.

[0088] In regularization, the input is a feature map that has undergone convolution, and regularization is applied to this feature map. For example, batch normalization or dropout can be used as regularization methods. General regularization techniques are sufficient.

[0089] In the activation process, the aforementioned fusion activation function is applied to the convolutionally processed or regularly processed data to generate the final output data from the convolutional layer. This output data is then input to the adjacent lower convolutional or pooling layer.

[0090] In the output layer, activation processing is performed on the output from the fully connected layer using an activation function. In this activation processing, the normalized exponential (Softmax) function is applied as an activation function, for example. Alternatively, other activation functions corresponding to the desired output form can be applied. For example, in the case of using a CNN for binary classification, a logistic function is used as the activation function, while in the case of using a CNN for regression problems, a linear mapping is used as the activation function.

[0091] Furthermore, the aforementioned fusion activation function can also be used for activation processing in the output layer, instead of the activation processing used in the convolutional layer. Alternatively, the fusion activation function can be used in activation processing in both the convolutional and output layers.

[0092] Hereinafter, an example of the configuration of the medical data processing device 1 according to this embodiment will be described. In the following description, the medical data processing device 1 is configured to be connected to the medical imaging device 3 and is installed together with the medical imaging device 3 into the medical image diagnostic device 9.

[0093] Figure 5 This diagram illustrates the configuration of the medical image diagnostic device 9 according to this embodiment. Figure 5As shown, the medical image diagnostic device 9 includes a medical data processing unit 1 and a medical imaging unit 3. For example, the medical imaging unit 3 corresponds to a stand, and the medical data processing unit 1 corresponds to a control console connected to the stand. Furthermore, the medical data processing unit 1 can be mounted on the stand of the medical image diagnostic device 9, or it can be implemented by other components that are neither the control console nor the stand of the medical image diagnostic device 9. Examples of such other components include, in the case where the medical image diagnostic device 9 is a magnetic resonance imaging device, a computer or dedicated computing device other than the control console, located in the machine room.

[0094] The medical imaging device 3 performs medical imaging on the subject using an imaging principle corresponding to the type of medical device it is equipped with, and acquires raw data related to the subject. The acquired raw data is transmitted to the medical data processing device 1. For example, the raw data may be k-space data if the medical imaging device 3 is a magnetic resonance imaging (MRI) device, projection data or sine wave data if the medical imaging device 3 is an X-ray computed tomography (CT) device, echo data if the medical imaging device 3 is an ultrasound diagnostic device, coincidence data or sine wave data if the medical imaging device 3 is a PET device, or projection data or sine wave data if the medical imaging device 3 is a SPECT device. Additionally, if the medical imaging device 3 is an X-ray diagnostic device, the raw data is X-ray image data. The medical imaging device 3 is an example of an imaging unit.

[0095] When the medical imaging device 3 is a stand for a magnetic resonance imaging (MRI) device, the stand is subjected to a static magnetic field via a static magnetic field magnet, and a gradient magnetic field is repeatedly applied via a gradient magnetic field coil and an RF pulse is applied via a transmitting coil. Due to the application of the RF pulse, an MR signal is released from the subject. The released MR signal is received via a receiving coil. The received MR signal undergoes signal processing such as A / D conversion by the receiving circuit. The MR signal after A / D conversion is called k-space data. The k-space data is transmitted as raw data to the medical data processing device 1.

[0096] In the case where the medical imaging device 3 is a stand for an X-ray computed tomography (CT) device, the stand rotates the X-ray tube and X-ray detector around the subject, irradiating the subject with X-rays from the X-ray tube. The X-ray detector detects the X-rays that have penetrated the subject. In the X-ray detector, an electrical signal with a peak value corresponding to the dose of the detected X-rays is generated. This electrical signal undergoes signal processing such as A / D conversion by a data acquisition circuit. The electrical signal after A / D conversion is called projection data or sine wave data. The projection data or sine wave data is transmitted as raw data to the medical data processing device 1.

[0097] In the case where the medical imaging device 3 is an ultrasound probe of an ultrasound diagnostic device, the ultrasound probe transmits an ultrasound beam from multiple ultrasound transducers into the body of the patient, and receives the ultrasound waves reflected from the body of the patient via the ultrasound transducers. The ultrasound transducers generate an electrical signal with a peak value corresponding to the sound pressure of the received ultrasound waves. This electrical signal is converted into digital signals by an A / D converter installed on the ultrasound probe, etc. The electrical signal after A / D conversion is called echo data. The echo data is transmitted to the medical data processing device 1 as raw data.

[0098] In the case where the medical imaging device 3 is a PET scanner stand, the stand is simultaneously measured by a simultaneous measurement circuit to generate a pair of 511 keV gamma rays produced by the annihilation of positrons generated from radionuclides accumulated in the patient's body with electrons surrounding the radionuclides. This generates digital data with digital values ​​related to the energy values ​​of the pair of gamma rays (LORs, or Line of Responses) and the detection location. This digital data is called coincidence data or sine curve data. The coincidence data or sine curve data is transmitted as raw data to the medical data processing device 1.

[0099] In the case where the medical imaging device 3 is a C-arm of an X-ray diagnostic device, X-rays are generated from an X-ray tube mounted on the C-arm. X-rays generated from the X-ray tube and transmitted through the subject are received by an X-ray detector, such as an FPD (Flat Panel Display), mounted on or independently of the C-arm. The X-ray detector generates an electrical signal with a peak value corresponding to the dose of the detected X-rays, and performs signal processing such as A / D conversion on this electrical signal. The A / D converted electrical signal is called X-ray image data. The X-ray image data is transmitted as raw data to the medical data processing device 1.

[0100] like Figure 5 As shown, the medical data processing device 1 has a processing circuit 11, a memory 13, an input interface 15, a communication interface 17, and a display 19 as hardware resources.

[0101] The memory 13 is a storage device that stores various types of information, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or integrated circuit. In addition to HDDs or SSDs, the memory 13 can also be a portable storage medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or flash memory. Furthermore, the memory 13 can also be a drive device for reading and writing various types of information to semiconductor memory elements such as flash memory and RAM (Random Access Memory). Moreover, the storage area of ​​the memory 13 can be located either within the medical data processing device 1 or in an external storage device connected via a network.

[0102] The memory 13 stores the program executed by the processing circuit 11, various data used in the processing of the processing circuit 11, etc. The program, for example, is a program pre-installed on the computer from a network or a non-volatile computer-readable storage medium and used to enable the computer to perform the functions of the processing circuit 11. Furthermore, the various data processed in this specification are typically digital data. The memory 13 is an example of a storage unit.

[0103] Memory 13 stores the learned model 90 generated by the model learning device 5. The learned model 90 is a neural network that has been trained on parameters p to be able to take input medical data as input and output output medical data as the target. The learned model 90 is an example of a learned neural network.

[0104] Figure 6 This diagram illustrates an example of the input and output combination of a learned model 90. The learned model 90 accepts input data and outputs output data. For example, as... Figure 6 As shown, the learned model 90 accepts input medical data and outputs medical data. The input and output medical data can be, for example, medical image data related to the subject being processed. Alternatively, instead of medical image data, the raw data of the subject being processed can be used as both input and output medical data. The raw data is the raw data related to the subject being processed. Furthermore, medical images can also be accepted as input medical data, and the recognition results related to the medical images can be output as medical data.

[0105] Furthermore, the raw data involved in this embodiment is not limited to the original raw data acquired by the medical imaging device 3. For example, the raw data involved in this embodiment may also be computational raw data generated by performing forward projection processing on the medical image generated by the restoration function 112 or the inference function 114. In addition, the raw data involved in this embodiment may also be raw data obtained by performing any data processing such as data compression processing, resolution decomposition processing, data interpolation processing, resolution synthesis processing, etc. on the original raw data. In addition, in the case of 3D raw data, the raw data involved in this embodiment may also be mixed data obtained by performing restoration processing on only 1 axis or 2 axes. Similarly, the medical image involved in this embodiment is not limited to the original medical image generated by the restoration function 112 or the inference function 114. For example, the medical image involved in this embodiment may also be a medical image obtained by performing any image processing such as image compression processing, resolution decomposition processing, image interpolation processing, resolution synthesis processing, etc. on the original medical image.

[0106] Input interface 15 accepts various input operations from the operator, converts the accepted input operations into electrical signals, and outputs them to processing circuit 11. For example, input interface 15 accepts input of medical information, various command signals, etc. from the operator. Input interface 15 is implemented using a mouse, keyboard, trackball, switch button, touchscreen integrating a display screen and touchpad, contactless input circuit using optical sensors, and voice input circuit, etc., for various processing by processing circuit 11. Input interface 15 is connected to processing circuit 11, converting input operations accepted by the operator into electrical signals and outputting them to control circuit. Furthermore, in this specification, input interface is not limited to interfaces with physical operating components such as a mouse and keyboard. For example, processing circuits that accept electrical signals corresponding to input operations from external input devices separate from the device and output such electrical signals to processing circuit 11 are also included in the example of input interface. Input interface 15 is an example of an input unit.

[0107] Communication interface 17 is a network interface that transmits and controls the communication between medical data processing device 1 and external devices via a network.

[0108] Display 19 displays various information. For example, display 19 outputs medical information generated by processing circuit 11, a GUI (Graphical User Interface) for handling various operations from the operator, etc. For example, display 19 is a liquid crystal display or a CRT (Cathode Ray Tube) display. Display 19 is an example of a display unit.

[0109] The processing circuit 11 controls the overall operation of the medical data processing device 1. The processing circuit 11 is a processor that executes camera control function 111, recovery function 112, acquisition function 113, reasoning function 114, image processing function 115, and display control function 116 by calling programs stored in the execution memory 13. Furthermore, in Figure 5 In this description, the camera control function 111, restoration function 112, acquisition function 113, reasoning function 114, image processing function 115, and display control function 116 are implemented by a single processing circuit 11. However, multiple independent processors can also be combined to form the processing circuit, with each processor executing a program to implement each function. Furthermore, the camera control function 111, restoration function 112, acquisition function 113, reasoning function 114, image processing function 115, and display control function 116 can each be installed as an individual hardware circuit. The above description of the functions performed by the processing circuit 11 is the same in the following embodiments and variations.

[0110] Furthermore, while the medical data processing device 1 is described as performing multiple functions from a single control console, these functions can also be performed by individual devices. For example, the functions of the processing circuit 11 can be distributed across different devices.

[0111] The term "processor" used in the above description refers to circuits such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), ASIC, programmable logic device (e.g., Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA)). The processor performs its function by reading and executing the program stored in memory 13. Alternatively, instead of storing the program in memory 13, the program can be directly loaded into the processor's circuitry. In this case, the processor performs its function by reading and executing the program loaded into the circuitry. Furthermore, the processors in this embodiment are not limited to being configured as a single circuit; multiple independent circuits can be combined to form one processor and perform its function. Furthermore, it is also possible to... Figure 5 The multiple components are integrated into a single processor to achieve their functions. The description of the "processor" described above is also the same in the following embodiments and variations.

[0112] The processing circuit 11 controls the medical imaging device 3 according to the imaging conditions through the camera control function 111, and performs medical imaging on the subject. The imaging conditions involved in this embodiment include the imaging principle of the medical imaging device 3 and various imaging parameters. The imaging principle corresponds to the type of medical imaging device 3, specifically, it corresponds to magnetic resonance imaging device, X-ray computed tomography device, PET device, SPECT device, and ultrasound diagnostic device. Imaging parameters include, for example, FOV (Field of View), imaging location, slice position, frame (phase of medical image), temporal resolution, matrix size, presence or absence of contrast agent, etc. In addition, in the case of magnetic resonance imaging, imaging parameters also include, for example, the type of imaging sequence, TR (Time to Repeat), TE (Echo Time), FA (Flip Angle), etc., and the type of k-space filling trajectory. In X-ray computed tomography, imaging parameters also include X-ray conditions (tube current, tube voltage, and X-ray exposure duration, etc.), scan type (non-helical scan, helical scan, synchronous scan, etc.), tilt angle, reconstruction function, number of views per rotation of the rotating frame, rotation speed, and detector spatial resolution. In ultrasound diagnostics, imaging parameters also include focal spot position, gain, transmitted intensity, received intensity, PRF, beam scanning method (sector scan, convex scan, linear scan, etc.), and scanning mode (B-mode scan, Doppler scan, color Doppler scan, M-mode scan, A-mode scan, etc.).

[0113] The processing circuit 11 restores medical images by performing restoration processing on the raw data transmitted from the medical imaging device 3 through the restoration function 112. The restoration processing involved in this embodiment includes any one of restoration from raw data to raw data, restoration from raw data to image data, and restoration from image data to image data. Furthermore, restoration processing from raw data defined by a certain coordinate system to 2D or 3D image data defined by another coordinate system is also referred to as reconstruction processing or image reconstruction processing. The restoration processing involved in this embodiment is, for example, denoising restoration or data error feedback restoration. For example, image reconstruction involved in the restoration processing involved in this embodiment can be classified as analytical image reconstruction and successive approximation image reconstruction. For example, analytical image reconstruction involved in MR image reconstruction includes Fourier transform or inverse Fourier transform. Analytical image reconstruction involved in CT image reconstruction includes FBP (filtered back projection), CBP (convolution back projection), or applications thereof. As a successive approximation image reconstruction, there are methods such as EM (expectation maximization), ART (algebraic reconstruction technique), or their applications. The processing circuit 11 used to implement the restoration function 112 is an example of an image generation unit.

[0114] The processing circuit 11 acquires processing target data through the acquisition function 113. In this embodiment, the processing target data is medical data related to the subject being processed. The medical data of the processing target is, for example, medical image data obtained through medical imaging of the subject. The processing circuit 11, which implements the acquisition function 113, is an example of an acquisition unit.

[0115] The processing circuit 11 calculates inference data by applying the learning completion model 90 to the processing object data through the inference function 114. In this embodiment, the processing circuit 11 applies the learning completion model 90 to the input medical data related to the subject, thereby generating output medical data as inference data. The output medical data is, for example, a desired medical image generated by performing image processing on a medical image. The processing circuit 11 used to implement the inference function 114 is an example of an inference unit.

[0116] The processing circuit 11 performs various image processing operations on the medical image generated by the restoration function 112 and the output image generated by the inference function 114 through the image processing function 115. For example, the processing circuit 11 performs 3D image processing such as volume drawing, surface drawing, pixel value projection processing, MPR (Multi-Planer Reconstruction) processing, and CPR (Curved MPR) processing. In addition, the processing circuit 11 can also perform alignment processing as image processing.

[0117] The processing circuit 11 displays various information on the display 19 via the display control function 116. For example, the processing circuit 11 displays a medical image generated by the restoration function 112, an output image generated by the inference function 114, and a medical image after image processing by the image processing function 115.

[0118] Next, the operation of the inference processing performed by the acquisition function 113 and the inference function 114 of the medical data processing device 1 will be explained. The inference processing involves applying the medical input data to the learning completion model 90 and causing the learning completion model 90 to output the desired output medical data. Furthermore, the processing procedures described below are merely examples, and each procedure can be appropriately modified within possible limits. Additionally, for the processing procedures described below, steps can be appropriately omitted, substituted, or added according to the implementation method.

[0119] In the inference process, the processing circuit 11 first obtains the learning completion model 90 from the memory 13 by acquiring the function 113, and the processing circuit 11 obtains the input medical data of the processing object to be applied to the learning completion model 90 from the medical camera device 3.

[0120] Next, the processing circuit 11 applies the learning completion model 90 to the input medical data through the inference function 114. The learning completion model 90 accepts the input medical data and generates output medical data. The processing circuit 11 obtains the output medical data generated by the learning completion model 90 as the inference result.

[0121] Next, the effects of the medical data processing device 1 according to this embodiment will be explained.

[0122] The medical data processing apparatus 1 according to this embodiment acquires processing target data and applies a learned completion model 90 to the processing target data to calculate inference data. The learned completion model 90 has a fusion activation function. The fusion activation function performs calculations based on multiple activation functions and multiple mixing coefficients corresponding to the multiple activation functions for each of the multiple unit network structures that transform input vector elements into output vector elements. The unit network structure is, for example, a node, layer, or channel of a neural network. Furthermore, one fusion activation function can be set for one node, or one fusion activation function can be set for multiple nodes. That is, the unit network structure can also be multiple nodes, multiple layers, multiple channels, etc.

[0123] Specifically, the fusion activation function can apply multiple activation functions to the same input vector feature to compute multiple first output values, apply multiple mixing coefficients to each of the multiple first output values ​​to compute multiple second output values, and compute the output vector feature based on the multiple second output values. For example, when performing the operation represented by equation (1) on the output values ​​z1 and z2 of multiple activation functions A1 and A2, the fusion activation function can perform the operation based on the multiple output values ​​z1 and z2 and multiple coefficients a1, a2, and a3 to obtain the output value y, and compute the output vector feature y_i based on the multiple output values ​​y. Here, coefficients a1, a2, and a3 correspond to mixing coefficients, output values ​​z1 and z2 correspond to first output values, and output value y corresponds to second output values.

[0124] With the above configuration, the medical data processing apparatus 1 according to this embodiment can achieve high inference performance compared to the case where the user arbitrarily selects the activation function to be used, by preparing multiple activation functions as candidates in advance and using a learning completion model 90 having a fusion activation function that integrates the prepared multiple activation functions for inference.

[0125] The medical image diagnostic apparatus 9 according to this embodiment includes the medical data processing apparatus 1 described above, and a medical imaging apparatus 3 for performing medical imaging on a subject. The medical data processing apparatus 1 can acquire medical data acquired by the medical imaging apparatus 3 as processing target data. Then, the medical data processing apparatus 1 can apply a learning completion model 90 to the acquired medical data to calculate inference data.

[0126] With the above configuration, the medical image diagnostic apparatus 9 according to this embodiment can utilize the learning completion model 90, which has high inference performance through the use of a fusion activation function, for desired inference processing of medical data acquired in the medical image diagnostic apparatus 9. For example, when performing image processing on medical images acquired in the medical imaging device 3 using the learning completion model 90, inference performance can be improved compared to the case where the user arbitrarily selects the activation function used in the activation process.

[0127] Next, an example of the configuration of the model learning device 5 involved in this embodiment will be described. Figure 7 This is a diagram showing the configuration of the model learning device 5. (For example...) Figure 7 As shown, the model learning device 5 has a processing circuit 51, a memory 53, an input interface 55, a communication interface 57, and a display 59 as hardware resources.

[0128] The processing circuit 51 controls the overall operation of the model learning device 5. The processing circuit 51 executes the processor that acquires the function 511 and learns the function 512 by calling the program stored in the execution memory 53.

[0129] The processing circuit 51 acquires the learning data used in the learning of the neural network through the acquisition function 511. The learning data includes input learning data and output learning data (hereinafter referred to as correct output data). The processing circuit 51 used to implement the acquisition function 511 is an example of an acquisition unit.

[0130] Processing circuit 51 generates a learned neural network by training the neural network based on the learning data through learning function 512. At this time, processing circuit 51 updates the learning parameters and mixing coefficients of the fusion activation function for each unit network construction to minimize the error-based loss function. This error is the difference between the output data (hereinafter referred to as inferred output data) of the neural network based on the input learning data and the correct output data. Processing circuit 51, used to implement learning function 512, is an example of a learning unit. Learning parameters are the parameters of the synthesis function defined for the neural network. Examples of learning parameters include weighting matrices and biases.

[0131] The memory 53 can be a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or integrated circuit that stores various types of information. In addition to HDDs or SSDs, the memory 53 can also be a portable storage medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or flash memory. Furthermore, the memory 53 can also be a drive device for reading and writing various types of information with semiconductor memory elements such as flash memory and RAM (Random Access Memory). Moreover, the storage area of ​​the memory 53 can be located either within the model learning device 5 or in an external storage device connected via a network.

[0132] The memory 53 stores the program executed by the processing circuit 51, various data used in the processing of the processing circuit 51, etc. The program, for example, is a program pre-installed on the computer from a network or a non-volatile computer-readable storage medium, which enables the computer to perform the functions of the processing circuit 51. Furthermore, the various data processed in this specification are typically digital data. The memory 53 is an example of a storage unit.

[0133] Memory 53 stores, for example, a model learning program 50 for learning a neural network. Additionally, memory 53 temporarily stores learning data for the neural network. Furthermore, memory 53 stores multiple activation functions as candidates for fusion activation functions.

[0134] Input interface 55 accepts various input operations from the operator, converts the accepted input operations into electrical signals, and outputs them to processing circuit 51. For example, input interface 55 accepts input of medical information, various command signals, etc. from the operator. Input interface 55 is implemented using a mouse, keyboard, trackball, switch button, touchscreen integrating a display screen and touchpad, contactless input circuit using optical sensors, and voice input circuit, etc., for various processing by processing circuit 11. Input interface 55 is connected to processing circuit 51 and converts the input operations accepted by the operator into electrical signals and outputs them to control circuit. Furthermore, in this specification, input interface is not limited to interfaces with physical operating components such as a mouse and keyboard. For example, processing circuits that accept electrical signals corresponding to input operations from external input devices separate from the device and output such electrical signals to processing circuit 51 are also included in the example of input interface. Input interface 55 is an example of an input unit.

[0135] Communication interface 57 is a network interface that transmits and controls communication between medical data processing device 1 and external devices via a network.

[0136] Display 59 displays various information. For example, display 59 outputs medical information generated by processing circuit 51, a GUI (Graphical User Interface) for handling various operations from the operator, etc. For example, display 59 is a liquid crystal display or a CRT (Cathode Ray Tube) display. Display 59 is an example of a display unit.

[0137] Next, the model learning process executed by the processing circuit 51 of the model learning device 5 according to the model learning procedure 50 will be described. The model learning process is the process of generating a learned model 90 by training a machine learning model based on the learning data.

[0138] In the model learning process, the processing circuit 51 first acquires learning data, including multiple learning samples, from the learning data storage device 7 via the acquisition function 511. The learning samples include a combination of input learning data and correct output data. Correct output data is the data expected to be output from the neural network when the input learning data is input. Correct output data can also be referred to as teacher data. Input learning data and correct output data can be, for example, medical images generated by photographing a patient. Input learning data and correct output data can also be medical images generated by photographing an arbitrary phantom.

[0139] Next, processing circuit 51 generates inferred output data through learning function 512 via forward propagation of a neural network based on the input learning data. Furthermore, the parameters of the neural network are set to initial values ​​during the first forward propagation. Next, processing circuit 51 calculates the error between the generated inferred output data and the input correct output data through learning function 512. Next, processing circuit 51 calculates the gradient vector through backward propagation of the neural network based on the calculated error through learning function 512. Next, processing circuit 51 updates the parameters of the entire neural network, including the mixing coefficients, based on the calculated gradient vector through learning function 512.

[0140] For example, when the fusion activation function is a function that performs the operation of equation (1) above, which combines activation functions A1 and A2, the learning parameters are updated by training the neural network, and the coefficients a1-a3 in equation (1) are updated as mixing coefficients. Furthermore, when the fusion activation function is a function that performs the operation of equation (2) above, which combines activation functions A1 and A2, the learning parameters are updated by training the neural network, and the coefficients a1-a6 in equation (2) are updated as mixing coefficients.

[0141] The processing circuit 51 determines whether the termination condition is met. The termination condition can be set, for example, if the number of repetitions reaches a predetermined number. Alternatively, the termination condition can be set if the gradient vector is less than a threshold. If the termination condition is not met, the processing circuit 51 repeatedly performs the above processing using the same learning sample or other learning samples. If the termination condition is met, the processing circuit 51 outputs the updated neural network as the learned model 90. The learned model 90 is stored in the memory 13 of the medical data processing device 1.

[0142] This concludes the description of the model learning process performed by the model learning device 5 according to this embodiment. Furthermore, the above-described learning process is an example, and this embodiment is not limited to it.

[0143] As described above, the model learning program 50 of this embodiment causes the model learning device 5 to execute the learning function 512. The learning function 512, for a neural network having an input layer that takes input learning data as input, an output layer that outputs output data corresponding to the input learning data, and at least one intermediate layer disposed between the input and output layers, applies the input learning data to generate inferred output data. Then, the learning function 512 updates the parameters of the neural network, including multiple mixing coefficients, to make the inferred output data approximate the correct output data.

[0144] Furthermore, the model learning device 5 according to this embodiment can acquire learning data including input learning data and output learning data, and train a neural network based on the learning data to generate a completed neural network. The neural network has multiple unit network structures with learning parameters that transform input vector elements into output vector elements. Additionally, the neural network has a fusion activation function that performs operations based on multiple activation functions and multiple mixing coefficients corresponding to each of the multiple unit network structures. The model learning device 5 can update the learning parameters and the multiple mixing coefficients to minimize the error between the output data of the neural network based on the input learning data and the output learning data.

[0145] With the above configuration, according to this embodiment, a function for activation processing can be generated through machine learning. Therefore, optimal parameters can be set as the mixing coefficients of the fusion activation function, enabling an inference device with further improved inference performance.

[0146] (First variation)

[0147] The first variation of the embodiment is described below. This variation is obtained by modifying the configuration of the embodiment as follows. Descriptions of configurations, operations, and effects that are the same as those in the embodiment are omitted.

[0148] In this variation, we illustrate a case where a neural network is used to output multiple output data for a single input data. The neural network is configured, for example, to accept medical images as input data and output multiple medical images after undergoing different image processing steps.

[0149] Figure 8 This is a diagram that roughly represents the case of the operation based on the fusion activation function involved in this variation. Figure 8 In one example shown, the fusion activation function consists of six activation functions A1-A6 and two blending functions M1 and M2. Figure 8 In this context, activation functions A1-A6 are represented as “Act.1”-“Act.6”.

[0150] Activation functions A1-A3 are different types of activation functions. Additionally, activation functions A4-A6 are different types of activation functions. The types of activation functions used as activation functions A1-A3 can be the same as or different from the types of activation functions used as activation functions A4-A6.

[0151] The input vector element x_i is applied to each activation function A1-A6. Each activation function A1-A6 accepts each value x of the input vector element x_i as input and outputs the result of applying the activation function to each value x as output values ​​z1-z6.

[0152] The mixing function M1 combines the three output values ​​z1-z3 obtained from activation functions A1-A3 according to the mixing coefficient to calculate one output value y1. The mixing function M2 combines the three output values ​​z4-z6 obtained from activation functions A4-A6 according to the mixing coefficient to calculate one output value y2.

[0153] The activation functions A1-A6 and the blending functions M1 and M2 described above are sequentially applied to each value x included in the input vector element x_i, and output values ​​y1 and y2 corresponding to each value x included in the input vector element x_i are output. The fusion activation function generates an output vector element y1_i including the output value y1 and an output vector element y2_i including the output value y2, which are used as the output of the fusion activation function.

[0154] Furthermore, as a neural network that outputs multiple output data for a single input data, it is also possible to construct a neural network that outputs multiple output data by applying multiple types of activation functions to the input data, using the output of each activation function as multiple channels as subsequent inputs, and repeatedly performing the aforementioned convolution and activation processes. For example, if the number of channels in the input data to each activation function is 10 and there are 3 activation functions, then the number of channels input to the next convolution process becomes 30. In this case, the fusion operation in the fusion activation function becomes part of the coefficients of the next convolution process. In this case, the same effect as the above-described implementation can be obtained.

[0155] (Second variation)

[0156] The second variation of the embodiment will be described. This variation is obtained by modifying the configuration of the embodiment as follows. Descriptions of configurations, operations, and effects that are the same as those in the embodiment are omitted.

[0157] The fusion activation function involved in this variation is a function that applies multiple blending coefficients to the same input vector feature to calculate multiple first output values, applies multiple activation functions to the multiple first output values ​​to calculate multiple second output values, and calculates the output vector feature based on the multiple second output values.

[0158] Figure 9 This diagram roughly illustrates the operation based on the fusion activation function involved in this variation. Here, as an example, we illustrate the case where the fusion activation function is a function obtained by combining activation functions A1, A2, and A3. The fusion activation function calculates the output value z'1 (=b1·x) by multiplying the input value x input to activation function A1 by a coefficient b1, calculates the output value z'2 (=b2·x) by multiplying the input value x input to activation function A2 by a coefficient b2, and calculates the output value z'3 (=b3·x) by multiplying the input value x input to activation function A3 by a coefficient b3. Output values ​​z'1-z'3 each correspond to the first output value. Coefficients b1-b3 each correspond to the mixing coefficients.

[0159] Next, the fusion activation function applies activation function A1 to the output value z'1, activation function A2 to the output value z'2, and activation function A3 to the output value z'3. Then, the fusion activation function applies a mixture function M to the output values ​​z1 of activation function A1, z2 of activation function A2, and z3 of activation function A3, and calculates the output value y. The output values ​​z1, z2, and z3 correspond to the second output value. The mixture function M can be a first-order function, a second-order function, or a polynomial of degree three or higher, with the output values ​​z1, z2, and z3 as variables.

[0160] (3rd variation)

[0161] The third variation of the embodiment will be described. This variation is obtained by modifying the structure of the embodiment as follows. Descriptions of structures, operations, and effects that are the same as those in the embodiment are omitted.

[0162] The fusion activation function involved in this variation is a function that applies multiple blending coefficients and multiple activation functions to the input vector features to reconstruct them, and then calculates the output vector features.

[0163] Here, as an example, we illustrate the case where the fusion activation function is the function obtained by combining activation functions A1 and A2. The fusion activation function is operated on using a function pre-reconstructed by multiplying activation function A1 by coefficient c1, and a function pre-reconstructed by multiplying activation function A2 by coefficient c2. Here, coefficients c1 and c2 each correspond to the mixing coefficients.

[0164] The fusion activation function applies the reconstructed activation functions A1 and A2 to the input value x, and applies the aforementioned mixture function M to the output values ​​z1 and z2 of the reconstructed activation function A1 and A2, respectively, to calculate the output value y. The mixture function M can be a first-order function, a second-order function, or a polynomial of order three or higher, with the output values ​​z1 and z2 as variables.

[0165] (4th variation)

[0166] The fourth variation of the embodiment is described below. This variation is obtained by modifying the configuration of the embodiment as follows. Descriptions of configurations, operations, and effects that are the same as those in the embodiment are omitted.

[0167] In the neural network involved in this variation, a regularization method is introduced as a countermeasure against overfitting.

[0168] As a regularization method, one example is to set all the mixing coefficients to the same value. For example, when the fusion activation function is a function that combines the output values ​​z1 and z2 of activation functions A1 and A2 using the above equation (1), overlearning can be suppressed by setting the coefficients a1-a3 of all fusion activation functions set according to the construction of each unit network to the same value.

[0169] Alternatively, the fusion activation functions that exist according to each network construction can be pre-grouped, and the mixing coefficients of all fusion activation functions belonging to each group can be set to the same value.

[0170] Alternatively, among the multiple fusion activation functions included in the neural network, the mixing coefficients corresponding to the same activation function can be set to the same value. In this case, the mixing coefficients (e.g., a1 in equation (1)) corresponding to the same type of activation function (e.g., activation function A1) are set to the same value in all fusion activation functions.

[0171] In this embodiment, the neural network is trained as a whole, and fusion activation functions are generated for each individual network unit. Therefore, the generated multiple fusion activation functions may be different functions constructed for each individual network unit, or they may be the same function.

[0172] Other regularization methods include adding a cost function to the loss function, which is based on the error between the inferred output data and the correct input output data. The smaller the difference between multiple mixing coefficients, the smaller the cost function. For example, the sum of squares of the mixing coefficients could be added as the cost function. Thus, by optimizing the parameters and mixing coefficients to minimize the error between the inferred output data and the correct output data, overlearning can be suppressed.

[0173] As another regularization method, one approach applicable to transfer learning using learning data outside the target domain can be cited. In this method, the learning parameters and mixing coefficients of a learned, completed neural network generated by training the neural network on learning data outside the target domain are used as initial values, and the neural network is trained using regular learning data. In the case where the regular learning data is medical image data, for example, image data from a general domain outside the medical field can be used as the learning data outside the target domain to train the neural network, thereby generating a learned, completed model. Then, by using the learning parameters and mixing coefficients of the generated learned, completed model as initial values ​​and training the neural network using medical image data as the learning data, the degradation of general performance caused by overlearning can be suppressed, and inference performance can be improved.

[0174] Other regularization methods include data augmentation, adding L2 regularization terms to the loss function for the learning parameters and / or mixture coefficients, and sparse regularization by adding L2 regularization terms to the loss function for the learning parameters and / or mixture coefficients. Alternatively, methods that apply L2 or L1 regularization to the difference in the learning parameters or mixture coefficients based on the transfer learning described above can also be used.

[0175] Furthermore, the regularization method is not limited to the specified method; various commonly used regularization methods can also be used. Additionally, multiple of the methods mentioned above can be combined.

[0176] (Other variations)

[0177] Furthermore, the neural network described in this embodiment can also be applied to complex networks. Additionally, the neural network described in this embodiment can also be applied to neural networks without bias. For example, in a complex network without bias, such as for denoising complex images like MRI images, the fusion activation function of this embodiment can be used instead of the activation function.

[0178] The neural network described in this embodiment is also useful for inference devices that process complex data. Furthermore, the activation function or neural network used can be differentiated depending on whether the acquired data is complex or not. For example, inference processing can be performed using a neural network employing the fusion activation function described in this embodiment only when the acquired data is MRI image data obtained from a magnetic resonance imaging device or echo data obtained from an ultrasound diagnostic device.

[0179] In this embodiment, a neural network suitable for medical data has been described as an example, but the fusion activation function of this embodiment can also be applied to neural networks that perform image recognition for general images. For example, a DNN that outputs the result of identifying which of the following objects—"cat," "dog," "horse," or "cow"—is displayed in an image can be considered as such a neural network. In training this neural network, for example, an image displaying a certain animal is used as input learning data, and a one-hot vector representing the type of animal ("cat," "dog," "horse," or "cow") is used as the correct output data.

[0180] According to at least one of the embodiments described above, the inference performance of learning-complete neural networks using activation functions can be improved.

[0181] Several embodiments have been described above, but these embodiments are given as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made to the forms of the embodiments described herein without departing from the spirit of the invention. These forms or modifications are included in the scope or spirit of the invention and are included in the scope of the claims and their equivalents.

Claims

1. A reasoning device, wherein, have: Acquisition Department, Acquisition of Medical Data; and The inference unit uses a learned neural network to calculate inference data based on the medical data. The learned neural network has a fusion activation function, which performs operations based on multiple activation functions and multiple mixing coefficients corresponding to the multiple activation functions for each unit network construction that transforms input vector features into output vector features. The fusion activation function applies the multiple mixing coefficients to the same input vector feature to calculate multiple first output values, applies the multiple activation functions to the multiple first output values ​​respectively to calculate multiple second output values, and calculates the output vector feature based on the multiple second output values.

2. A medical image diagnostic device, wherein, have: The reasoning device as claimed in claim 1; and Medical imaging devices are used to perform medical imaging on the subject of an examination. The acquisition unit acquires the medical data obtained by the medical camera device, and uses it as the medical data.

3. A reasoning method, wherein, have: Obtain the process, obtain medical data; and In the inference process, a learned neural network is used to calculate the inference data based on the medical data. The learned neural network has a fusion activation function, which performs operations based on multiple activation functions and multiple mixing coefficients corresponding to the multiple activation functions for each unit network construction that transforms input vector features into output vector features. The fusion activation function applies the multiple mixing coefficients to the same input vector feature to calculate multiple first output values, applies the multiple activation functions to the multiple first output values ​​respectively to calculate multiple second output values, and calculates the output vector feature based on the multiple second output values.

4. A method for generating a learning-complete neural network, wherein, have: The acquisition process involves acquiring learning data, including input and output learning data, as medical data; and The learning process involves training a neural network based on the learning data to generate a completed neural network. The neural network has: A multi-unit network with learning parameters is constructed to transform input vector features into output vector features; as well as The activation functions are fused, and operations are performed on each of the multiple unit network constructions based on multiple activation functions and multiple mixing coefficients corresponding to the multiple activation functions. The fusion activation function applies the multiple mixing coefficients to the same input vector feature to calculate multiple first output values, applies the multiple activation functions to each of the multiple first output values ​​to calculate multiple second output values, and calculates the output vector feature based on the multiple second output values. In the learning process, the learning parameters and the plurality of mixing coefficients are updated to minimize the error between the output data of the neural network based on the input learning data and the output learning data.

5. The learning-complete neural network generation method as described in claim 4, wherein, The learning process includes setting the mixing coefficients corresponding to the same activation function to the same value among the plurality of fusion activation functions included in the neural network.

6. The learning-complete neural network generation method as described in claim 4, wherein, The learning process includes adding a cost function, where the smaller the difference between the multiple mixing coefficients, the smaller the value.

7. The learning-complete neural network generation method as described in any one of claims 4 to 6, wherein, The learning process includes: using the learning parameters and mixing coefficients of the neural network generated by training the neural network with learning data other than the target as initial values, and performing the training of the neural network based on the learning data.

Images