Adaptable neural network
Through two neural network systems, one of which generates task-specific parameters and adjusts the structure of the other neural network, the problem of difficulty in switching between multiple tasks is solved, and high efficiency and high accuracy is achieved. Task adaptive analysis.
Patent Information
- Application Number
- CN201980055866.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-30
- Filing Date
- 2019-08-20
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-08-20
AI Technical Summary
Existing neural networks are suitable for specific tasks, making it difficult to flexibly switch between multiple tasks, and the accuracy of multi-task neural networks is low, resulting in the need to create multiple neural networks to meet different task needs, increasing processing power and storage need.
A system consisting of two neural networks, one of which (the second neural network) is used to generate task-specific parameters, adjust the structure of the other neural network (the first neural network) to adapt it to different tasks, only Two neural networks can achieve highly personalized and task-adaptive analysis.
The high efficiency and dynamic flexibility of the neural network system are realized, the need for multiple task-specific neural networks is avoided, the accuracy of each task is improved, and the complexity of the system is reduced.
Smart Images

Figure CN112639833B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network systems, and in particular to neural network systems capable of performing multiple evaluations of input data. Background Art
[0002] WO 2018 / 081563 discloses methods and systems for determining neural network architectures. One of these methods includes: using a controller neural network to generate a batch of output sequences, each output sequence in the batch defines a corresponding architecture of a sub-neural network, which is configured to perform a specific neural network task; for each output sequence in the batch: training a corresponding instance of the sub-neural network having the architecture defined by the output sequence; evaluating the performance of the trained instance of the sub-neural network on the specific neural network task to determine the performance index of the trained instance of the sub-neural network on the specific neural network task; and using the performance index for the trained instance of the sub-neural network to adjust the current value of the controller parameter of the controller neural network.
[0003] Neural networks (e.g., networks employing deep learning models) are increasingly used for data analysis tasks (e.g., image analysis and speech recognition). Typically, a neural network is applied to input data to generate desired output information or data. A typical neural network operates by applying a numerical model formed by a series of analysis steps to the input data to generate or predict output data indicative of the outcome of the desired task. Each analysis step is often referred to as a "layer" of the neural network.
[0004] For example, a neural network may process an X-ray image of a subject's chest (i.e., input data) to predict or identify whether the subject has a particular disease (the desired task), wherein the neural network provides output data that predicts whether the subject has the particular disease or the probability that the subject has the particular disease.
[0005] Neural networks are typically tuned to perform a specific task, i.e., they are trained using training data to generate a specific output. The process involves collecting training data and applying a numerical model (e.g., a numerical model formed by a plurality of different layers or analysis steps) to the training data to provide a desired output. Parameters of the numerical model are then modified based on a comparison between the provided desired output and known values of the desired output (i.e., values obtained from the training data) to improve the performance of the numerical model. Such a neural network can then be applied to new input data to predict a desired output based on the new input data.
[0006] However, one problem with neural networks is that a (once trained) neural network is adapted to a specific task. This means that performing multiple tasks or input data evaluation requires the creation of multiple neural networks. Disadvantageously, this requires each neural network to have a large amount of processing power and / or storage capacity.
[0007] Of course, multi-task neural networks (i.e., networks that can provide more than one output) are known in the art. Typically, such networks operate by performing a first set of analysis steps or layers on the input data to generate initial processed input data before performing a separate task-specific analysis step or layer on the initial processed input data for each desired task. However, such approaches lack individualized functionality for each task - because the initial layers are shared between tasks, they can be less accurate than task-specific dedicated neural networks. Summary of the invention
[0008] It would therefore be desirable to provide a neural network system that is capable of providing highly customized and task-specific analysis of input data without requiring the use of multiple neural networks.
[0009] The invention is defined by the independent claims. The dependent claims define advantageous embodiments.
[0010] According to an example according to one aspect of the present invention, there is provided an adaptable neural network system for processing first input data. The neural network system comprises: a first neural network formed by a plurality of sequential layers, wherein the first neural network is adapted to generate output data by processing the first input data using the plurality of sequential layers; a second neural network adapted to generate at least one task-specific parameter for the plurality of sequential layers of the first neural network based on second input data indicating a desired task to be performed on the first input data; and a neural network modifier adapted to modify the first neural network based on the at least one task-specific parameter generated by the second neural network, so as to adapt the first neural network to the desired task whenever the second neural network receives new second input data.
[0011] Thus, the second neural network (i.e., the generator) adapts the first neural network (i.e., the actor) to a different task. In particular, the second neural network is adapted to modify or weight the parameters of the different layers of the first neural network in order to tune the first neural network for a specific task. Thus, the second neural network defines the task or prediction performed by the first neural network. In this way, a highly personalized and task-adaptable neural network system can be provided using only two neural networks.
[0012] The goal of the second neural network is to produce in the first neural network a network architecture with parameters (e.g., weight / bias values) that are specialized to provide the best possible performance for a given task (e.g., given the specific conditions and / or distribution of input data). The goal of the first neural network is to perform a specific task (i.e., make predictions) using the weight values generated by the second neural network. To achieve these goals, the second neural network adapts the layers of the first neural network to specialize the first neural network for a specific task.
[0013] Thus, embodiments of the present invention avoid the need to generate multiple task-specific neural networks, while also allowing each task to be highly personalized - providing a neural network system with high performance. The proposed network system architecture also provides greater dynamic flexibility for the processing of input data.
[0014] The second input data may indicate a desired task to be performed by the first neural network, for example, defining a desired prediction of the first neural network. In other words, the second input data indicates a desired prediction task of the first neural network so as to define the type of output data.
[0015] In at least one embodiment, the second neural network is adapted to generate the at least one task-specific parameter further based on third input data providing information about the first input data. In other words, the parameters of the first neural network may depend on information of the first input data, e.g., metadata. The third input data may thereby provide context information associated with the first input data. This advantageously allows the first neural network to be personalized for the specific first input data being studied, thereby improving the accuracy of the neural network system.
[0016] Preferably, said second input data comprises an input query to said adaptable neural network system, said input query indicating a desired task to be performed.
[0017] Thus, the neural network system can function to perform a user-specified task (e.g., as specified in an input query). For example, this can be used to provide clinicians with a useful tool to check their assumptions about a subject / patient.
[0018] The input query may include a hypothesized attribute associated with the first input data, and the first neural network is adapted to determine a value indicating whether the hypothesized attribute is correct. The neural network system may thus act as a hypothesis tester, determining whether the hypothesis is correct by appropriately adapting the first neural network using the hypothesis provided as input to the second neural network. The first neural network may, for example, be adapted to provide a probability indicating a likelihood that the hypothesized attribute is correct. In other embodiments, the first neural network may be adapted to output a value indicating a confirmation or rejection of the hypothesized attribute (e.g., the result of a one-tailed test or a two-tailed test).
[0019] In some examples, the neural network system includes: a text processing unit adapted to process text of the first input data for providing to the second neural network. In particular, the text processing unit can extract features of allowable dimensions (i.e., having a specific size) for processing by the second neural network.
[0020] Preferably, said plurality of sequential layers of said first neural network comprises a predetermined number of adjustable parameters, and said second neural network is adapted to generate task-specific parameters for each of said predetermined number of adjustable parameters.
[0021] This allows the second neural network to be adapted to the architecture and / or structure of the first neural network, for example by setting certain parameters to zero or null values. This effectively allows the overall architecture of the first neural network to be controlled by the second neural network. Thus, this enables and improves the control of the first neural network by the second neural network and results in a first neural network that is more suitable for performing the desired task.
[0022] The second neural network may include a plurality of sequential layers, and the at least one task-specific parameter is output by a sequential last layer of the plurality of sequential layers. Thus, in some embodiments, the two neural networks may be provided as separate "black boxes".
[0023] Of course, in some embodiments, a portion of the at least one task-specific parameter may be output by an earlier layer in the plurality of sequential layers, while a different portion may be output by a sequentially last layer in the plurality of sequential layers. That is, at least some of the at least one task-specific parameter is output by a sequentially last layer in the plurality of sequential layers. It should be understood that the second neural network may be designed to provide only task-specific parameters for the first neural network (i.e., without providing any other outputs).
[0024] Preferably, the first neural network cannot change the parameters of the second neural network (i.e., the second neural network is independent of the first neural network). Of course, it should be understood that the first neural network can define the maximum number of parameters (e.g., weights or biases) that can be received from the second neural network (i.e., help define the initial structure of the second neural network).
[0025] The first input data may include clinical / medical information of the subject, and the second input data may include a hypothesized diagnosis or symptoms of the subject. For example, the first input data may include a medical image of the subject. Optionally, the second neural network is further adapted to generate the at least one task-specific parameter based on metadata (e.g., subject information) associated with the medical image of the subject.
[0026] It has been recognized that embodiments of the present invention are particularly useful in the context of analyzing medical information because such information often includes complex and not fully understood data that can be advantageously and automatically identified by a neural network system.
[0027] According to an example according to one aspect of the present invention, there is also provided a method for processing first input data using an adaptable neural network system including a first neural network formed by a plurality of sequential layers. The method includes: using a second neural network to generate at least one task-specific parameter for the plurality of sequential layers of the first neural network based on second input data indicating a desired task to be performed on the first input data; modifying the first neural network based on the at least one task-specific parameter so that the first neural network is adapted to the desired task each time the second neural network receives new second input data; and generating output data by processing the first input data using the modified first neural network.
[0028] The step of generating at least one task-specific parameter comprises generating the at least one task-specific parameter further based on third input data providing information about the first input data.
[0029] Optionally, the second input data comprises an input query to the adaptable neural network system, the input query indicating a desired task to be performed.
[0030] In at least one embodiment, the plurality of sequential layers of the first neural network comprises a predetermined number of tunable parameters, and the at least one task-specific parameter comprises a task-specific parameter for each of the predetermined number of tunable parameters.
[0031] The second neural network may include a plurality of sequential layers, and the at least one task-specific parameter is output by a sequential last layer of the plurality of sequential layers.
[0032] According to an example according to another aspect of the present invention, there is provided a computer program comprising code means for implementing any of the above methods when the program is run on a computer.
[0033] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] For a better understanding of the invention and to show more clearly how it may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:
[0035] Figure 1 A neural network system according to an embodiment of the present invention is illustrated;
[0036] Figure 2 illustrates a first neural network for a neural network system according to an embodiment;
[0037] Figure 3 illustrates a second neural network for a neural network system according to an embodiment; and
[0038] Figure 4 A method of processing first input data according to an embodiment of the present invention is illustrated. DETAILED DESCRIPTION
[0039] Embodiments of the present invention will be described with reference to the accompanying drawings.
[0040] It should be understood that the detailed description and specific examples indicate exemplary embodiments of the device, system and method, but this is intended for illustrative purposes only and is not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood from the following description, claims and drawings. It should be understood that the drawings are only schematic and are not drawn to scale. It should also be understood that the same reference numerals are used in all drawings to indicate the same or similar parts.
[0041] According to the concept of the present invention, a neural network system formed by two neural networks is proposed. One of the neural networks adjusts the structure of the other neural network based on information about a specific task so that the other neural network is suitable for performing the specific task. Therefore, an adaptable neural network system capable of performing different tasks on input data can be realized.
[0042] Embodiments are based at least in part on the recognition that a general neural network can be adapted to a specific task by adjusting parameters (e.g., weights and / or biases) of various layers of the general neural network, and that these parameters can be generated by a second neural network. Thus, the second neural network can process information about the desired task in order to adapt the general (i.e., first) neural network to perform the desired task.
[0043] The illustrative embodiments may be used as an aid to diagnostic systems, for example, where medical images are processed using a neural network system to predict or identify the presence of different medical conditions or diagnoses.
[0044] In the context of the present invention, a "layer" refers to a processing or analysis step of a neural network or numerical model for processing input data. As is known in the art, the output of any given layer may be referred to as a "feature map." The skilled person will appreciate that the term "task-specific parameters" may refer to the weights and / or biases of a layer of a neural network.
[0045] Figure 1 The architecture of an adaptable neural network system 1 according to an embodiment of the present invention is illustrated. Generally, the neural network system 1 is adapted to process first input data 11 to generate output data 15.
[0046] The neural network system 1 comprises a first neural network 4 which processes first input data 11 to generate output data 15. The first neural network 4 comprises a series of sequential layers 4a-4c, each of which is sequentially applied to the input data 11, the output of each previous layer providing input to the next layer.
[0047] Each layer 4a-4c of the first neural network 4 may have one or more parameters that can be modified to adapt the first neural network 4 to perform a particular task. For example, where each layer represents an algorithm or processing step applied to the output of the previous layer, each processing step may include a different one or more parameters within the processing step.
[0048] The second neural network 5 generates one or more task-specific parameters 7 for the plurality of sequential layers 4a-4c of the first neural network 4. In particular, the second neural network 5 processes the second input data 12 indicating the desired task to be performed on the first input data 11 to generate the task-specific parameters 7.
[0049] The second neural network 5 may, for example, comprise a plurality of sequential layers 5a-5c adapted to generate task-specific parameters 7 for modifying or customizing (the layers 4a-4c of) the first neural network 4 so as to perform a desired task indicated by the second input data 12. Thus, one or more task-specific weights and / or biases 7 can be output by a sequential last layer of the plurality of sequential layers 5a-5c.
[0050] In this manner, the second neural network 5 generates parameters (e.g., weights or values) for adapting the plurality of sequential layers 4a-4c of the first neural network 4 to a specific task. The specific or desired task is identified / defined by the second input data 12, which may include an input query indicating the desired task to be performed on the first input data 11 (e.g., a prediction of the presence / absence of a specific parameter using the first input data 11).
[0051] For example, the values generated by the second neural network 5 may be used to adjust or modify one or more parameters of at least one layer 4a-4c of the first neural network 4. In some examples, the second neural network 5 is adapted to generate a modified value for each parameter of each layer 4a-4c of the first neural network 4 (e.g., the value of the weights or biases of the layers of the first neural network 4).
[0052] therefore, Figure 1 Each arrow of the illustrated parameter 7 in FIG. 4 may represent at least one weight, bias, or other modified value of a corresponding layer of the first neural network 4 .
[0053] A neural network modifier 6 is provided to modify the first neural network 4 using the task-specific parameters generated by the second neural network 5. For example, the neural network modifier may modify one or more parameters of one or more layers 4a-4c of the first neural network 4.
[0054] In this way, the neural network modifier 6 is able to adapt the first neural network 4 to perform a desired task (eg, prediction of the presence / absence of a specific parameter) based on the parameters generated by the second neural network 5 .
[0055] For example, the neural network modifier 6 may appropriately weight (ie, change the value of) or otherwise modify at least one parameter of each layer 4a-4c of the first neural network 4 based on the parameters generated by the second neural network 5.
[0056] Preferably, the first input data 11 comprises clinical / medical information of the subject / patient (e.g., an X-ray image or an ECG trace), and the second input data 12 comprises a query for confirming or rejecting the presence of a specific sign / symptom / diagnosis or a hypothesis of the presence of a specific sign / symptom / diagnosis. Thus, the first input data 11 may comprise clinical / medical information of the subject, and the second input data 12 may comprise a hypothesized diagnosis or signs / symptoms of the subject.
[0057] For example, the first input data 11 may include an X-ray image of a subject's bones, and the second input data 12 may include queries regarding the following examples: whether the subject has a fracture; whether the subject suffers from an osteochondroma; or whether the subject suffers from subacromial impingement. The second neural network 5 adapts the first neural network 4 to solve the task indicated by the second input data 12 (e.g., determining whether the subject has a fracture).
[0058] As another illustrative example, the first input data 11 may include a medical image of a subject (e.g., an X-ray image or a CT scan), and the second input data 12 may include a query as to whether the medical image depicts a tumor. In this example, the second neural network 5 generates parameters for modifying the first neural network 4 so that when the modified first neural network 4 processes the first input data 11, the modified first neural network 4 predicts whether the medical image depicts a tumor. Thus, the output data 15 provides an indication of whether the medical image (i.e., the first input data 11) depicts a tumor.
[0059] As another illustrative example, the first input data 11 may include information of an ECG trace of a subject, and the second input data 12 may include a query (or hypothesis) as to whether the subject suffers from one of the following: tachycardia, heart attack, suffers from hyperkalemia, atrial fibrillation, etc. The second neural network 5 is capable of adapting or modifying the first neural network 4 such that the first neural network 4 processes the ECG trace to establish an answer to the query of the second input data 12 (e.g., “Does the subject suffer from hyperkalemia?”) or to confirm or reject the hypothesis provided by the second input data 12 (e.g., “Does the subject suffer from hyperkalemia”) or to provide a probability that the hypothesis is true.
[0060] Preferably, the output data 15 indicates confirmation or rejection of the hypothesis indicated by the second input data 12 or a probability value indicating the likelihood that the hypothesis is true. In other words, the role of the first neural network 4 can be to confirm or reject a general hypothesis or to provide a probability value for the general hypothesis - wherein the second neural network 5 then adapts the first neural network 4 to the specific hypothesis.
[0061] Thus, the hypothesis provided by the second input data 12 indicates the task to be performed (ie, whether to reject or not reject the hypothesis specified by the second input data 12).
[0062] Examples of hypotheses for an X-ray image of a subject's chest include: pneumonia is present; a tumor is present; calcification is present; fluid is in the lungs, etc.
[0063] Thus, the second input data 12 may define a confirmation / rejection task or a probability determination task to be performed by the first neural network 4. Thus, the second neural network 5 may be adapted to tailor the first neural network 4 to predict the likelihood that a hypothesis (defined by the second input data 12) is correct or to confirm / reject the hypothesis - for example based on a statistical evaluation or whether the likelihood is greater than a predetermined percentage.
[0064] The first neural network 4 may output a binary result, indicating confirmation or rejection of the hypothesis (eg, presence or absence of a tumor), or a percentage likelihood that the hypothesis is correct (eg, an 80% chance that a tumor is present).
[0065] In other examples, the neural network system 1 is adapted to provide diagnostic information as output data, wherein the first input data 11 and the second input data 12 are medical data (eg, medical images or reports).
[0066] For example, the first input data 11 may include a medical image, and the second input data 12 may include a medical report. The medical report may be processed to identify a disease name or symptoms, which are provided as the second input data 12.
[0067] The second neural network 5 can then adapt the first neural network 4 to generate a predicted final diagnosis based on the medical image, wherein the prediction process (i.e., the first neural network) is adapted based on information provided in the medical report. Such a product is particularly useful in training clinicians or other staff (e.g., regarding the relationship between medical images and medical reports).
[0068] Recurrent neural networks (eg, long short-term memory based networks) may be used to convert free text (ie, unstructured text) into structured input queries used as second input data 12. This avoids the need to manually extract tags in medical reports, for example.
[0069] In a further embodiment, the second neural network 5 is adapted to generate the parameters 7 further based on third input data 13 , which third input data provide information about the first input data 11 .
[0070] For example, where the first input data 11 includes a medical image of an object, the third input data 13 may include metadata (of the image) (e.g., image capture method (e.g., X-ray or CT scan), exposure time, location, view, etc.) or object-specific information (e.g., age, gender, diagnosis information, symptom information, etc.).
[0071] Such an embodiment enables the first neural network 4 to be customized or personalized to the first input data 11, thereby improving the prediction accuracy of the neural network system when performing a specific task identified by the second input data 12.
[0072] It has been recognised that the information provided by the third input data 13 can have an impact on the (actual) outcome of the desired task as indicated by the second input data 12. Therefore, the second neural network 5 can use the third input data 13 to appropriately modify the first neural network 4 to reflect this impact of the third input data.
[0073] For example, the presence of a first disease in a subject can make the presence of another disease more / less likely. Thus, if the second input data 12 indicates a desired task of determining whether a first disease is present (e.g. and the first input data 11 provides an X-ray image of the subject) and the third input data indicates that the known presence of the second disease increases the likelihood of the presence of the first disease, the second neural network 5 may modify the first neural network 4 such that the first neural network 4 is more likely to determine the presence of the first disease.
[0074] This improves the accuracy of the prediction or task performed by the neural network system 1. In other words, the proposed neural network system enables taking into account dependencies between third input data and output data.
[0075] The value of any of the first input data 11, the second input data 12 or the (optional) third input data 13 is binary, categorical or continuous.
[0076] Each of the first neural network and the second neural network may include a neural network having a convolutional layer or a fully connected layer. The output of each network (i.e., the parameters 7 and the output data 15) is a continuous function of its input value - this means that the two neural networks can be trained using a standard stochastic gradient descent method - wherein the gradient information is obtained from a standard back-propagation algorithm.
[0077] The first neural network 4 and the second neural network 5 may be formed according to any known network architecture. However, the "DenseNet" architecture is particularly suitable for use as the first neural network 4, as it uses very few filters per layer - this enables the second neural network 5 to be very efficient.
[0078] In particular, each layer of the DenseNet architecture uses the feature maps (i.e., outputs) of all previous layers as input. Therefore, because features are reused, DenseNet requires far fewer parameters to achieve equivalent or better performance than competing network architectures. In other words, the second neural network 5 requires fewer parameters to be generated than other neural networks known in the art, and thus provides a more efficient neural network system.
[0079] A complete description of a "DenseNet" architecture suitable for use as a neural network in some embodiments of the present invention is set forth in "Densely connected convolutional networks" by Huang, G., Liu, Z., Weinberger, KQ, and van der Maaten, L. (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, No. 2, July 2017, p. 3).
[0080] The design of the second neural network 5 may depend on the design of the first neural network 4. In particular, the second neural network 5 may be designed such that the number of parameters 7 provided by the second neural network 5 matches the maximum possible number of parameters of the first neural network 4.
[0081] This can be performed, for example, by summing all parameters in all layers 4a-4c of the first neural network 4. The number of parameters N in a single convolutional layer L It can be determined, for example, using the following formula:
[0082] N L =N i *N o *k x *k y +n b (1)
[0083] Among them, N i Equal to the number of input channels of the layer, N o Equal to the number of output channels of the layer, k x is the kernel size in the x direction, k y is the kernel size in the y direction, and n b is the number of biases or bias units in this layer, which is equal to the total number of output channels N o The total number of parameters in each layer is N L The sum can be calculated (ie, the total number of parameters in the first neural network 4 can be generated).
[0084] Those skilled in the art will appreciate that different types of layers may use different formulas (i.e., different formulas than the one presented in formula (1)) to calculate the number of parameters for modification by the second neural network. For example, long short-term memory, fully connected, and / or separable convolutional layers will be constructed differently as convolutional layers, and therefore may require different methods to calculate the number of parameters N contained therein. L .
[0085] It has been recognised that, for the second neural network 5, a few layers (i.e. <10 consecutive layers) of convolutional and / or fully connected layers are typically sufficient to generate suitable parameters for the first neural network 4, under the constraints that the second input data 12 matches the expected input dimensions of the application and the number of parameters 7 output by the second neural network matches the number of parameters present in the first neural network 4.
[0086] Of course, the value of one or more of the parameters generated by the second neural network 5 can be 0. In this way, the second neural network 5 is able to control the topology of the first neural network 4 (i.e., by excluding some weights, biases, variables or parameters from the layers or processing steps 4a-4c of the first neural network 4). This improves the level of control of the second neural network 5 over the first neural network 4, thereby improving the adaptability of the neural network system 1.
[0087] In other words, the second neural network 5 may be adapted to control the topology or architecture of the first neural network 4 by enabling the second neural network 5 to set one or more parameters (and therefore variables) of the first neural network 4 to zero (0) or a null value.
[0088] Thus, a neural network system 1 is presented that is capable of self-adapting to a desired task. In particular, the neural network system 1 enables interactive hypothesis generation and testing by enabling a clinician or other user to input (via second input data 12 or a query) a hypothesis about first input data 11 that can be tested.
[0089] To provide background examples, Figure 2 and Figure 3 A first network and a corresponding second network according to an embodiment of the present invention are respectively illustrated.
[0090] In each figure, the different layers of the network are illustrated, along with the input shape "in" and output shape "out" of each respective layer. Those skilled in the art will well appreciate that "input shape" refers to the data structure or data format (i.e., vector / matrix / tensor structure / size) received by a given layer of a neural network, and "output shape" refers to the data structure or data output (i.e., vector / matrix / tensor structure / size) provided by a given layer of a neural network (i.e., to be received by subsequent layers).
[0091] First reference Figure 2 , the first network comprises a series of layers adapted to process first input data to generate output data. Here, the first input data comprises a grayscale (single channel) 256×256 image, and the output data is a single value representing the probability that a particular hypothesis is true. Thus, the data structure or format associated with the input data is (256, 256, 1), and the data structure or format of the output data is (1).
[0092] Each layer of the first network is adapted to perform a specific task on the output of the previous layer.
[0093] The input layer Image_In receives the pixel values of the 256×256 image (ie, the first input data). Since it is an input layer, the output of the input layer has a shape of (256, 256, 1). In some examples, the input layer may be a normalization layer.
[0094] The two-dimensional convolutional layers Conv2D_1, Conv2D_2, Conv2D_3 perform convolution operations on the output of the previous layer. The convolution process will be well known to those skilled in the art. In the illustrated scenario, each convolutional layer performs the convolution process using a 2D filter kernel of size 3×3, stride 1, and padding with zeros (i.e., zeros are added to the edges of the input matrix so that the output of the convolution is the same size as the input, a concept known in the art). The number of zero rows / columns to be added to the edges of the input depends on the size of the filter, and for a 3×3 filter, the padding size is 1.
[0095] Pooling layers Pool_1, Pool_2, and Pool_3 perform pooling operations on the output of the previous layer. Here, the pooling layer performs a maximum pooling process on the output of the previous layer. In the illustrated scenario, the pooling layer performs a 2×2 maximum pooling with a stride of 2.
[0096] The different pooling layers AveragePool perform an average pooling process on the output of the previous layer.
[0097] The output layer Pred_Out is a dense or fully connected layer that performs a linear operation on the output of the previous layer. A single output value (i.e., output data) representing the probability of a particular hypothesis being true can then be generated through a sigmoid activation function. The number of weights in a dense or fully connected layer is equal to the number of input channels N i Multiply by the number of output channels N o The number of deviations is equal to the number of output channels N o .
[0098] The first network includes a series of sequences of the above layers, which are sequentially applied to the input data to generate the output layer. In the illustrated example, three convolutional pooling layer pairs are used to process the output of the input layer before the output layer generates the output data.
[0099] Table 1 illustrates the data structure of weights and biases for each layer (layer name). Table 1 also illustrates the corresponding number of parameters associated with each layer (eg, parameters calculated using formula (1)).
[0100] Therefore, for a convolutional layer with a weight structure of (3, 3, 1, 16) and a bias structure of (16), the total number of parameters is 160.
[0101] In this example, the weight structure represents (k x , k y , N i , N o ). As mentioned above, N i Equal to the number of input channels of the layer, N o Equal to the number of output channels of the layer, k x is the kernel size in the x direction, k y is the kernel size in the y direction. The deviation structure represents (n b ), where n b is the number of biases, which is related to the number of output channels N o same.
[0102] To make it clearer, Table 1 also illustrates the shapes or data structures received as input (input shape) and provided as output (output shape) for each layer. Those skilled in the art will well understand that the input shape indicates the data structure received by the neural network layer, and the output shape indicates the data structure output by the neural network layer.
[0103] Due to the nature of the pooling operation, the pooling layer does not require weights and / or biases (i.e., parameters). Similarly, the input layer does not require weights and / or biases and is therefore not associated with any parameters.
[0104]
[0105] Table 1
[0106] The second neural network is intended to generate task-specific weights and / or biases (i.e., parameters) for the relevant layers of the first neural network. Thus, the second neural network is adapted to adapt the first neural network to perform a specific task or predict a specific outcome.
[0107] Therefore, in the illustrated scenario (as shown in Table 1), the second neural network must generate eight different parameter outputs with different data structures to provide a total of 23361 parameters. Therefore, weights and biases should be generated for each of the convolutional layers Conv2D_1, Conv2D_2, Conv2D_3 and the output layer Pred_Out.
[0108] Figure 3 The diagram is suitable for Figure 2 The first neural network generates task-specific weights and / or biases for the second neural network.
[0109] Figure 3 The second neural network includes two input layers Query_In and Meta_In.
[0110] The query input layer Query_In receives second input data indicating a desired task to be performed on the first input data. For example, the second input data may include an input query to the adaptable neural network system indicating a desired task to be performed. The size of the second input data may be preset (e.g., having a size of 14). In a specific example, a recurrent neural network (e.g., a long short-term memory-based network) converts free text (i.e., unstructured text) into a structured input query that serves as the second input data of the query input layer Query_In.
[0111] The metadata input layer Meta_In receives third input data providing information about the first input data. In other words, the metadata input layer receives metadata providing information about the first input data.
[0112] Dense layers Dense_3, Dense_4, Dense_5, Dense_6, Dense_7, Dense_8, Dense_9, Dense_10, Dense_11, Dense_12 all perform linear operations on the output of the previous layer to generate output. The operation of dense layers will be well known to technicians.
[0113] The concatenation layer Conc_1 concatenates the outputs of multiple layers (ie, two or more data structures) to form a single output (ie, a single data structure). The operation of the concatenation layer is also well known to the skilled person.
[0114] The reshape layers Reshape1_C1W, Reshape2_C1B, Reshape_C2W, Reshape4_C2B, Reshape5_C3W, Reshape6_C3B, Reshape7_OutW, Reshape8_OutB reshape or reshape the output of the previous layer to form a reshaped output that has the same values in different data structures.
[0115] In the illustrated example, the second network takes two input types: a second input data having a 14-dimensional vector, and a third input data having a 9-dimensional vector. Initially, each input is processed by a corresponding subnetwork (i.e., the first subnetwork: Dense_1, Dense_2; and the second subnetwork: Dense_3, Dense_4). Deeper in the neural network, their outputs are cascaded together with the cascade layer Conc_1. In some examples, the cascading can instead occur directly after the input layer for each input type (e.g., concatenating vectors at the input layer so that the input vector is 23). The cascaded output vectors of the subnetworks of the second network are then used as input for calculating weights and biases for the first network.
[0116] In some examples, each dense layer of the first sub-network Dense_1, Dense_2 and the second sub-network Dense_3 and Dense_4 is followed by a non-linear activation layer or processed using a non-linear activation function such as a ReLU layer. However, if the dense layer has non-linearity like a ReLU layer, each sub-network can only be formed by two dense layers.
[0117] The output of the cascade layer Conc_1 is used as input to each of a plurality of different further sub-networks, each of which generates weight(s) or bias(es) for a layer of the first neural network.
[0118] Each additional subnetwork here includes a dense layer (e.g., Dense_5) and a reshape layer (e.g., Reshape1_C1W).
[0119] The first set of sub-networks generates weights for each corresponding layer (requiring weights) of the first neural network, and the second set of sub-networks generates biases for each corresponding layer (requiring weights) of the first neural network.
[0120] Here, the reshape layer is used as the output layer of the second neural network. In particular, the reshape layer reshapes the data structure into a format for providing weights or biases for the layers of the first neural network. The output of each reshape layer is a data structure that provides weights and / or biases for each layer of the first neural network that requires weights and / or biases.
[0121] return Figure 2 , the first network has a total of eight layers, of which only four have any associated parameters, namely: Conv2D_1, Conv2D_2, Conv2D_3, and Pred_Out. In order to provide weights and biases for each of these layers, Figure 3 The second neural network provides four outputs (to provide weights and biases for each of the four layers). Each output has a data structure suitable for providing a corresponding weight or bias for a particular layer of the first neural network.
[0122] exist Figure 3 In , the suffix W indicates weight, and the suffix B indicates bias. The suffix C1 refers to the convolutional layer Conv2D_1, the suffix C2 refers to the convolutional layer Conv2D_2, the suffix C3 refers to the convolutional layer Conv2D_3, and the suffix Out refers to Figure 1 The output layer Pred_Out of the first neural network.
[0123] Reshape1_C1W provides the weights (indicated by the suffix W) for the convolutional layer Conv2D_1 (i.e., "C1") of the first neural network. Reshape2_C1B provides the biases (indicated by the suffix B) for the convolutional layer Conv2D_1 (i.e., "C1") of the first neural network.
[0124] Reshape3_C2W provides weights (indicated by the suffix W) for the convolutional layer Conv2D_2 (ie, "C2") of the first neural network. Reshape4_C2B provides biases (indicated by the suffix B) for the convolutional layer Conv2D_2 (ie, "C2") of the first neural network.
[0125] Reshape5_C3W provides weights (indicated by the suffix W) for the convolutional layer Conv2D_3 (i.e., "C3") of the first neural network. Reshape6_C3B provides biases (indicated by the suffix B) for the convolutional layer Conv2D_3 (i.e., "C3") of the first neural network.
[0126] Reshape7_OutW provides weights (indicated by the suffix W) for the output layer Pred_Out (ie, “Out”) of the first neural network. Reshape8_OutB provides biases (indicated by the suffix B) for the output layer Pred_Out (ie, “Out”) of the first neural network.
[0127] The weights and biases (ie, parameters) generated by the second neural network are provided as weights and biases of the relevant layers of the first neural network. The first network 4 is able to make predictions for the given first input data using the new weights and biases.
[0128] In this way, the second input data (representing the desired task) and the third input data (representing metadata of the first input data) can influence the processing of the first input data. This allows the first neural network to be adapted to perform a task or predict an output based at least on the desired task (indicated by the second input data).
[0129] In particular, the second neural network is adapted to receive second input data indicative of a desired task to be performed by the first neural network and to generate weights and biases for the first neural network.
[0130] The second neural network may be trained according to any known neural network training mechanism, such as using a gradient descent based training system or a genetic algorithm to train the second neural network.
[0131] Figure 4 A method 40 according to an embodiment of the invention is illustrated. The method processes first input data 11 using an adaptable neural network system comprising a first neural network 4, such as those previously described.
[0132] The method comprises a step 41 of generating, using the second neural network 5, at least one task-specific parameter 7 for the first neural network 4. In particular, step 41 generates at least one task-specific parameter 7 based on second input data 12, the second input data 12 indicating a desired task (e.g. a prediction task) to be performed on the first input data.
[0133] The method 40 further comprises a step 42 of modifying the first neural network 4 based on the at least one task-specific parameter 7. Hence, step 42 adapts the first neural network 4 to the desired task such that the modified first neural network 4' is tailored for the desired task (indicated by the second input data 12).
[0134] The method 40 then performs step 43 of generating output data 15 by processing the first input data 11 using the modified first neural network 4'. Thus, the personalized or adapted neural network 4' generates output data 15.
[0135] The method enables a neural network system to be adapted to perform a specific task by modifying the first neural network 4 based on second input data 12 indicative of the specific task. The second input data 12 is processed by the second neural network 5 to generate parameters 7 or modification values for modifying the first neural network to perform the specific task.
[0136] In some embodiments, step 41 comprises generating at least one task-specific parameter 7 further based on third input data 13 providing information about the first input data 11. Thus, the second input data 12 and the third input data 13 may be used simultaneously to generate task-specific parameter(s) for modifying the first neural network 4.
[0137] Some embodiments of the present invention may be systems, methods and / or computer program products. A computer program product may include (one or more) computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention. Therefore, a computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device (e.g., a controller, a processor or a processing system) to run a method according to the present invention.
[0138] Some embodiments of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products. It should be understood that each box in the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.
[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a fragment or a portion of an instruction, which includes one or more executable instructions for implementing (one or more) specified logical functions.
[0140] In some alternative implementations, the functions indicated in the blocks may not occur in the order indicated in the figure. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagram and / or flow chart illustration, and combinations of blocks in the block diagram and / or flow chart illustration, can be implemented by a dedicated hardware-based system that performs the specified functions or actions or executes a combination of dedicated hardware and computer instructions.
[0141] In summary, the various embodiments provide an adaptable neural network system 1 formed of two neural networks 4, 5. Whenever one neural network 5 receives new second input data 12 indicating a desired task, one of the neural networks 5 adjusts the structure of the other neural network 4 based on information about the specific task, so that the other neural network 4 is suitable for performing the specific task. Therefore, an adaptable neural network system 1 capable of performing different tasks on the input data 11 can be realized.
[0142] Those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed invention by studying the drawings, the disclosure and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "one" or "an" do not exclude multiple. A single processor or other unit can implement the functions of several items recorded in the claims. Although certain measures are recorded in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used to advantage. The computer program may be stored / distributed on a suitable medium, for example, an optical storage medium or solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, for example, via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be interpreted as limiting the scope.
Claims
1. An adaptable neural network system (1) for processing first input data (11) comprising medical information, the neural network system (1) include: a first neural network (4) formed by a plurality of sequential layers (4a, 4b, 4c), wherein the first neural network is adapted to generate output data (15) by processing first input data (11) using the plurality of sequential layers; a second neural network (5) adapted to generate at least one task-specific parameter (7) for the plurality of sequential layers (4a, 4b, 4c) of the first neural network (4) based on second input data (12) indicative of a desired task to be performed by the first neural network on the first input data (11); and A neural network modifier (6) adapted to modify the first neural network (4) based on the at least one task-specific parameter (7) generated by the second neural network (5) so as to adapt the first neural network (4) to the desired task each time the second neural network (5) receives new second input data (12).
2. The adaptable neural network system (1) according to claim 1, in, The second neural network (5) is adapted to generate the at least one task-specific parameter (7) further based on third input data (13) providing information about the first input data (11).
3. The adaptable neural network system (1) according to claim 1 or 2, in, The second input data (12) comprises an input query to the adaptable neural network system (1), the input query indicating a desired task to be performed.
4. The adaptable neural network system (1) according to claim 3, in, The input query comprises a hypothesized attribute associated with the first input data (11), and the first neural network (4) is adapted to determine a value (15) indicating whether the hypothesized attribute is correct.
5. The adaptable neural network system (1) according to any one of claims 1 to 4, in, The plurality of sequential layers (4a, 4b, 4c) of the first neural network (4) comprises a predetermined number of adjustable parameters, and the second neural network (5) is adapted to generate task-specific parameters (7) for each of the predetermined number of adjustable parameters.
6. The adaptable neural network system (1) according to any one of claims 1 to 5, in, The second neural network (5) comprises a plurality of sequential layers (5a, 5b, 5c), and the at least one task-specific parameter (7) is output by a sequential last layer (5c) of the plurality of sequential layers.
7. The adaptable neural network system (1) according to any one of claims 1 to 6, in, The first input data (11) comprises clinical information of a subject, and the second input data (12) comprises a hypothesized diagnosis or symptoms of the subject.
8. The adaptable neural network system (1) according to claim 7, in, The first input data (11) comprises a medical image of the subject.
9. The adaptable neural network system (1) according to claim 8, in, The second neural network (5) is further adapted to generate the at least one task-specific parameter (7) based on metadata (13) associated with the medical image of the subject.
10. A method (40) for processing first input data (11) using an adaptable neural network system (1) comprising a first neural network (4) formed of a plurality of sequential layers, the first input data comprising medical information, the method include: generating (41) at least one task-specific parameter (7) for the plurality of sequential layers of the first neural network (4) based on second input data (12) indicative of a desired task to be performed by the first neural network on the first input data (11), using a second neural network (5); modifying (42) the first neural network (4) based on the at least one task-specific parameter (7) so as to adapt the first neural network to the desired task each time the second neural network (5) receives new second input data (12); and Output data (15) is generated (43) by processing the first input data (11) using the modified first neural network (4').
11. The method (40) according to claim 10, in, The step (41) of generating at least one task-specific parameter (7) comprises generating the at least one task-specific parameter (7) further based on third input data (13) providing information about the first input data.
12. The method (40) according to claim 10 or 11, in, The second input data (12) comprises an input query to the adaptable neural network system, the input query indicating a desired task to be performed.
13. The method (40) according to any one of claims 10 to 12, in, The plurality of sequential layers of the first neural network (4) comprises a predetermined number of adjustable parameters, and the at least one task-specific parameter (7) comprises a task-specific parameter for each of the predetermined number of adjustable parameters.
14. The method (40) according to any one of claims 10 to 13, in, The second neural network (5) comprises a plurality of sequential layers, and the at least one task-specific parameter (7) is output by a sequential last layer of the plurality of sequential layers.
15. Computer program product having a computer program comprising code means for implementing the method according to any one of claims 10 to 14 when the program is run on a computer.
Citation Information
Patent Citations
Neural architecture search
CN108021983A
Performing kernel striding in hardware
CN108073983A
Method, computer program product and device for training a neural network
CN108229539A
Resizing neural networks
CN108369664A
Evolutionary neural network and method of generating an evolutionary neural network
EP1901212A2
