Systems and methods for training neural ode networks

Abstract

The invention relates to a computer-implemented method for training a neural network comprising a neural ordinary differential equation (ODE) block. The neural ODE block can be trained using a first ODE solver. The neural ODE block can be trained using a second ODE solver and verified that the neural ODE block describes an ODE that is an ODE flow. During a forward pass of a training iteration, a first performance value is obtained by applying the first ODE solver to the neural ODE block and a second performance value is obtained by applying the second ODE solver to the neural ODE block. An accuracy parameter of the first ODE solver is adjusted based on a difference between the first performance value and the second performance value.

Description

Technical Field

[0001] This invention relates to a computer implementation method for training neural ordinary differential equation (ODE) networks, and a corresponding system for training neural networks. The invention further relates to a computer-readable medium including instructions for performing the above-described method. Background Technology

[0002] The use of neural networks has become a focal point in countless applications, ranging from healthcare and drug discovery to robotics, manufacturing, and autonomous vehicles. In many applications, neural networks are trained to provide outputs such as predictions or decisions. For example, a neural network can be trained as a classifier, such as for classifying images into one or more categories (naturally, semantic segmentation and detection tasks can be considered special cases of this classification task). Using neural networks as classifiers is particularly attractive, for instance, because it is possible to train the neural network by feeding it training data (e.g., a set of images and their corresponding categories). Based on the training data (e.g., known information), the neural network can determine and refine its weights and other network parameters to achieve the correct classification given an input. These weights and parameters are highly dependent on the neural network architecture employed and have a considerable impact on the performance and reliability of the neural network.

[0003] Therefore, the choice of neural network architecture can be an important consideration. Among the many options, residual neural networks (often called ResNet) have become an important subclass of models because they mitigate the gradient problem that arises when training deep neural networks by adding skip connections between consecutive layers.

[0004] In their paper “Neural Ordinary Differential Equations” (available at https: / / arxiv.org / abs / 1806.07366 and incorporated herein by reference), Chen, Rubanova, Bettencourt, and Duvenaud proposed neural ordinary differential equation (ODE) models (commonly referred to as neural ODE networks) as analogues to continuous deep ResNets. Neural ODE models represent a new family of deep neural network models in which the derivatives of the hidden states are parameterized via the neural network.

[0005] Neural ODEs offer a wealth of theoretical knowledge on ODEs that can be applied to deep learning research without sacrificing the pure predictive performance of ResNets. For example, neural ODEs have been found to be more robust than convolutional neural networks for certain perturbations. Furthermore, inspired by the theoretical properties of solution curves, regularizers have been proposed that further improve the robustness of neural ODE models. Robust neural ODE models can translate into more accurate, faster, and more efficient image classification and offer numerous benefits in a wide range of applications, such as safer and more fuel-efficient autonomous driving, improved clinical diagnosis, and more accurate optical examinations. However, if neural ODEs are chosen due to their theoretical advantages, it is important that the neural network exhibits the same qualitative behavior as analytical ODE flows. Therefore, a training method and system need to be developed that ensures the resulting trained neural network corresponds to analytical ODE flows (such as time-continuous dynamical systems), allowing techniques and knowledge from the ODE domain to be applied to the neural network. Summary of the Invention

[0006] It is well known that finer discretization of the ODE is chosen to represent an analytic ODE flow. However, as the discretization becomes finer, more computational resources and longer training times are required to train the neural ODE network, leading to higher latency. In practice, longer training times and more computational resources may not be available. In many applications, neural network latency can be a significant constraint. For example, the latency of an image classifier used for autonomous driving may be limited, but the image classifier should still be robust. That is, using overly fine discretization can increase the time required for computation and processing, thus increasing latency. In autonomous driving, such as obstacle tracking, it is important to make predictions quickly, i.e., with low latency. As another example, many systems use continuous learning. For example, when deploying devices equipped with neural networks that model analytic ODE flows, it is particularly advantageous to automatically determine whether the neural network still corresponds to an analytic ODE flow (such as a time-continuous dynamical system) in order to maintain the robustness of the neural network. Therefore, it is desirable to obtain a training method that can ensure that the trained neural ODE blocks of the neural network describe, or at least sufficiently describe, analytic ODE flows (such as time-continuous dynamical systems), while reducing training time and computational resource requirements.

[0007] According to a first aspect of the invention, a computer implementation method and a corresponding training system for training a neural network including neural ODE blocks are described. According to another aspect of the invention, a computer-readable medium is described.

[0008] Various aspects involve training a neural network comprising neural ODE blocks. The neural ODE blocks of the neural network can be configured as vector fields describing ODEs. When the neural ODE blocks are trained to describe ODE flows (such as time-continuous dynamical systems), techniques from the field of ODE research can be used to improve prediction, classification, and similar tasks. The neural network comprises neural ODE blocks. A neural ODE block can be part of or the entire neural network, designed to approximate an analytical ODE flow. One method for training a neural network includes accessing model data defining the neural network and accessing a training dataset comprising multiple batches of training data. The method further includes training the neural network on the training dataset and using a first ODE solver. A second ODE solver is also accessed. The training includes initializing accuracy parameters of the first ODE solver and adjusting the accuracy parameters during training by obtaining a first output and a second output of the neural network during a forward pass of an iteration of the training. The first output of the neural network is obtained through the following steps: providing the output of an upstream layer of the neural ODE block to a first ODE solver; applying the first ODE solver to the neural ODE block to obtain the output of the first ODE solver; and feeding the output of the first ODE solver into a downstream layer of the neural ODE block. The second output of the neural network is obtained through the following steps: providing the output of an upstream layer of the neural ODE block to a second ODE solver; applying the second ODE solver to the neural ODE block to obtain the output of the second ODE solver; and feeding the output of the second ODE solver into a downstream layer of the neural ODE block. The method further includes: using the first output of the neural network to evaluate a performance function to obtain a first performance value; and using the second output of the neural network to evaluate the performance function to obtain a second performance value. The method further includes: adjusting the accuracy parameter of the first ODE solver based on the difference between the first performance value and the second performance value.

[0009] To train a neural network including the neural ODE block, the numerical solution can be optimized for evaluating the neural ODE block at discrete time points. However, if the numerical solver is discretized too coarsely, the neural ODE block becomes dependent on the numerical solver and no longer behaves as an ODE flow (such as a time-continuous dynamical system)—it becomes too discrete. Therefore, the second ODE solver verifies whether the first ODE solver is configured to ensure that the neural ODE block is independent of the numerical method used by the first ODE solver by checking whether solutions to the neural ODE block using different numerical methods or different solvers will produce the same or sufficiently similar solutions. If the solutions obtained by using the two solvers are sufficiently similar, the neural ODE block is sufficiently discretized to behave as an ODE flow. In this case, it is possible to adjust the accuracy parameter of the first ODE solver towards coarser discretization, which can reduce training time and computational resource requirements. If the solutions obtained by using the two solvers are significantly different, the neural ODE block has become dependent on the numerical solver. In this case, a more refined discretization is used to adapt to the accuracy parameters of the first ODE solver.

[0010] Optionally, the neural network further includes an upstream block and a downstream block, the upstream block being upstream of the neural ODE block and including at least one layer, and the downstream block being downstream of the neural ODE block and including at least one layer. The layer output of the upstream neural ODE block may be the output of the upstream block. Feeding the output of the first ODE solver downstream in the neural network may include feeding the output of the first ODE solver to the downstream block. Feeding the output of the second ODE solver downstream in the neural network may include feeding the output of the second ODE solver to the downstream block.

[0011] Optionally, the difference between a first performance value and a second performance value is determined, and the magnitude of the difference is compared to a threshold. Adjusting the accuracy parameter may include adjusting the accuracy parameter based on the comparison between the magnitude of the difference and the threshold.

[0012] Optionally, the performance function is a loss function, where a first performance value is a first loss value and a second performance value is a second loss value. Alternatively, the performance function is an accuracy function, where a first performance value is a first accuracy value and a second performance value is a second accuracy value. The performance function quantifies the performance characteristics of the neural network with respect to a batch of training data. For example, the performance function could be a loss function, which provides an indication of the loss in the neural network during training. The loss function can be used to indicate how well the neural network performs during training. In another example, the performance function could be an accuracy function, which measures the accuracy of the neural network during training. For example, if the neural network is trained as a classifier, the accuracy could be based on the proportion of correctly predicted data.

[0013] Optionally, the accuracy parameter is a tolerance, and adjusting the accuracy parameter based on a comparison of the magnitude of the difference with a threshold includes: increasing the tolerance of the first ODE solver if the magnitude of the difference is greater than the threshold, and decreasing the tolerance of the first ODE solver if the magnitude of the difference is less than the threshold.

[0014] Optionally, the accuracy parameter is the step size, and adjusting the accuracy parameter based on a comparison of the magnitude of the difference with a threshold may include: decreasing the step size of the first ODE solver if the magnitude of the difference is greater than the threshold, and increasing the step size of the first ODE solver if the magnitude of the difference is less than the threshold. Optionally, increasing the step size may include multiplying the step size by a first predetermined value, the first predetermined value being greater than 1, and / or decreasing the step size may include multiplying the step size by a second predetermined value, the second predetermined value being between 0 and 1.

[0015] A difference exceeding a threshold indicates a performance degradation and a dependence on a numerical solver used to solve the ODE approximated by the neural ODE block. For example, if the first ODE solver is an adaptive step-size solver, reducing the tolerance of the first ODE solver provides finer discretization, which trains the neural ODE block to behave as an ODE flow. Similarly, if the first ODE solver is a fixed step-size solver, reducing the step size provides finer discretization and trains the neural ODE block to behave as an ODE flow. For example, if the difference is below a threshold, increasing the tolerance or similarly increasing the step size can reduce training time and the computational power and resources required to train the neural network, while still ensuring that the neural ODE block can be described by an ODE flow.

[0016] Optionally, both the first and second ODE solvers are fixed-step ODE solvers, and the second ODE solver has a higher convergence order than the first ODE solver. Optionally, the second ODE solver is selected to have the same or smaller expected discretization error as the first ODE solver. A higher convergence order for the second ODE solver generally results in a smaller discretization error. Therefore, comparing the performance values ​​obtained using the first ODE solver with those obtained using the second ODE solver can be used to detect performance degradation due to the impposed discretization by the first ODE solver and to avoid performance degradation due to coarse discretization from the second ODE solver.

[0017] Optionally, the neural network is an image classifier configured to accept an image as input and classify the image into at least one of a plurality of categories.

[0018] Optionally, the accuracy parameter is periodically adjusted during the training process, such as after a predefined number of iterations. Periodically adjusting the accuracy parameter has the effect of fine-tuning the accuracy parameter, which allows the method to train the neural network in such a way that the neural ODE block can be described by an ODE flow, while reducing training time and the required computational cost.

[0019] Optionally, the accuracy parameter is adjusted non-periodically during the training period. In some cases, adjusting the accuracy parameter at irregular intervals may be beneficial. For example, near the beginning of training the neural network, the accuracy parameter can be adjusted more frequently to ensure that the neural network is trained using a first ODE solver that is discretized sufficiently finely.

[0020] Optionally, the accuracy parameters are repeatedly adjusted until training of the neural network is complete. Intermittently adjusting the accuracy parameters as described above throughout the training process ensures that the accuracy parameters of the first ODE solver are applied appropriately for the duration of the training, which improves the training of the neural network.

[0021] Optionally, the first ODE solver and / or the second ODE solver are based on at least one of Runge-Kutta methods (such as the Euler method, the midpoint method, the Heun method, classical higher-order Runge-Kutta methods, the Bogacki-Shampine method, and the Ralston method) and multi-step methods (such as the Adams-Bashforth method, the backward differential formula BDF method, or the Adams-Moulton method). The second ODE solver may be based on a method used as an embedded higher-order solver. For example, if the first ODE solver is based on the Euler method, the second ODE solver may be based on the midpoint method. As another example, if the first ODE solver is based on the second-order Bogacki-Shampine method, the second ODE solver may be based on the third-order Ralston method. In yet another example, the first ODE solver can be based on the Adams-Bashforth method, and the second ODE solver can be based on the Adams-Moulton method.

[0022] Optionally, the method further includes: outputting model data representing the trained neural network. Outputting model data representing the trained neural network enables the acquisition and use of the neural network trained via the method to perform classification, decision, and / or accuracy tasks.

[0023] Optionally, the method further includes: receiving an image as input to the neural network, and classifying the image into at least one of a plurality of categories. Image classification can be used in a wide variety of applications, such as in autonomous vehicles, healthcare applications, and robotics, such as in automated optical inspection.

[0024] Those skilled in the art will appreciate that two or more of the above embodiments, implementations and / or optional aspects of the present invention can be combined in any manner deemed useful.

[0025] Computer implementations of the modifications and variations corresponding to the system described herein, or modifications and variations of any computer-readable medium, can be performed by those skilled in the art based on this description, and vice versa. Attached Figure Description

[0026] These and other aspects of the invention will become apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. In the drawings:

[0027] Figure 1 A system for training a neural network including a block of neural ordinary differential equations (neural ODEs) is shown according to an embodiment of the present invention;

[0028] Figure 2 A system for using a trained neural network including a neural ODE block is shown according to an embodiment of the present invention;

[0029] Figure 3a The figure shows a graph of the numerically computed trajectory of the initial value problem for a neural ODE model trained with a small step size;

[0030] Figure 3b The figure shows a graph of the numerically computed trajectory of the initial value problem for a neural ODE model trained with a large step size;

[0031] Figure 4a A detailed example of training a neural network including neural ODE blocks is shown;

[0032] Figure 4b A detailed example of training a neural network including neural ODE blocks according to an embodiment of the present invention is shown;

[0033] Figure 4c A detailed example of training a neural network including neural ODE blocks according to an embodiment of the present invention is shown;

[0034] Figure 5 A computer implementation method for training a neural network including neural ODE blocks according to an embodiment of the present invention is shown; and

[0035] Figure 6 A computer-readable medium containing data is shown.

[0036] It should be noted that items with the same reference numerals in different figures have the same structural features and the same function, or the same signal. Since the function and / or structure of such items have already been explained, it is unnecessary to repeat this explanation in the detailed description.

[0037] Reference Symbol List

[0038] The following list of reference symbols is provided to facilitate the interpretation of the accompanying drawings, and this list should not be construed as restrictive.

[0039] 021 Data storage device

[0040] 022 External data storage device

[0041] 030 Training Data

[0042] 040 defines the model data for neural networks.

[0043] 072 Sensor

[0044] 100 Training System

[0045] 120 Data Interface

[0046] 121 Data Communication

[0047] 122 Access to data storage devices

[0048] 140 Processor Subsystem

[0049] 200 system

[0050] 220 Data Interface

[0051] 221 Data Communication

[0052] 222 Access to external data storage devices

[0053] 223 Data Communication

[0054] 224 sensor data

[0055] 240 Processor Subsystem

[0056] 260 Sensor Interface

[0057] 410 A batch of training data

[0058] 420 Upstream block of the neural network

[0059] 430 First ODE Solver

[0060] 435 Second ODE Solver

[0061] 440 Neural ODE Blocks of Neural Networks

[0062] 450 Downstream blocks of the neural network

[0063] 460 The first output of the downstream block

[0064] 465 The second output of the downstream block

[0065] 470 Performance Functions

[0066] 475 The second output of the performance function

[0067] 480 Comparison Functions

[0068] 600 Computer-readable media

[0069] 610 data. Detailed Implementation

[0070] The following describes a method and system for training neural networks that include blocks of neural ordinary differential equations (ODEs). The neural ODE block can be part of or the entire neural network, configured to approximate the ODE using a neural ODE architecture. See references for details. Figure 1 and Figure 2 The training system and the resulting trained system are described, with reference to... Figure 4a , 4b The 4Cs describe the computer implementation method. (Reference) Figure 5 A computer-implemented training method is described in detail. Examples throughout the following description will be provided in the context of image classification, although it is to be understood that the invention is not limited thereto—the invention can be similarly applied to any regression task or any application in which a feedforward network can be used. While the examples provided throughout this description relate to neural networks with a single neural ODE block, it is to be understood that neural networks are not limited thereto. In some embodiments, the neural network may include multiple neural ODE blocks. In such embodiments, each neural ODE block can be trained in a similar manner.

[0071] Since numerical solvers play a crucial role in approximating solutions to ODEs, it is important to understand how the choice of numerical methods affects the training of neural ODE models. The inventors have discovered that if a trained neural ODE model corresponds to an ODE that is sufficiently approximated by the applied discretization, any discretization with similar or smaller discretization errors should produce the same predictions. Because the solution to an ODE is independent of the type of numerical solver used to solve it, the neural ODE model should similarly be independent of the type of numerical ODE solver. Therefore, a second ODE solver can be used to ensure that the neural ODE model is trained to behave as an ODE flow during the training process, thereby enabling the resulting trained network to possess effective ODE flow semantics. This behavior allows the trained neural network to benefit from techniques and advancements in the field of ODEs.

[0072] Figure 1A training system 100 for training a machine-learnable neural network according to an embodiment of the present invention is illustrated. The neural network may include neural ODE blocks. In some embodiments, the neural network may further include upstream blocks and / or downstream blocks. A neural ODE block refers to a block of a neural network parameterized to model an ODE flow. An upstream block refers to one or more layers in the neural network upstream of the neural ODE block, and a downstream block refers to one or more layers in the neural network downstream of the neural ODE block. In other words, a neural ODE block may be between an upstream block and a downstream block. For example, an input may be fed to an upstream block, the output of the upstream block may be fed to the neural ODE block, and the output of the neural ODE block may be fed to a downstream block. The output of the downstream block may be the output of the neural network. For example, if the neural network is trained to perform an image classification task, the output of the downstream block may be the resulting classification. In some embodiments, the neural network may not include either or both of the downstream and upstream blocks. In this case, the input of the neural network may be directly input to the neural ODE block, and the output of the neural ODE block may be the output of the neural network.

[0073] The training system 100 may include a data interface 120 and a processor subsystem 140, which may be configured to communicate internally via data communication 121. The data interface 120 may enable access to model data 040 that defines the model. The model data may include at least a set of parameters (such as the weights of a neural network). The data interface 120 may further be used to enable access to a training data set 030. In some embodiments, the model data 040 may be used in training datasets such as those generated by... Figure 2 The trained system 200 is used when making decisions (such as image classification).

[0074] The processor subsystem 140 can be configured to access data 030 and 040 during operation of the training system 100 and using the data interface 120. For example, Figure 1 As shown, data interface 120 can provide access 122 to external data storage device 021, which may include the data 030, 040. Alternatively, data 030, 040 can be accessed from an internal data storage device that is part of training system 100. Alternatively, data 030, 040 can be received from another entity via a network. Generally, data interface 120 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 021 can take any known and suitable form.

[0075] Processor subsystem 140 can obtain training data set 030. The training dataset may include multiple training data elements. Training data 030 may include one or more batches of training data, each batch of which may include multiple training data elements. In some embodiments, each training data element may include an input (such as an input image) and a corresponding ground truth (such as a ground truth classification). In some embodiments, the input may be obtained from a sensor communicatively coupled to the neural network. In this figure, the training dataset is accessed via data interface 120, but this is not mandatory; for example, the training data may be measured during training.

[0076] The processor subsystem 140 can train a neural network based on training dataset 030. The neural network can be trained using a first ODE solver and a second ODE solver, the second ODE solver employing a different numerical ODE solving method than the first ODE solver. Training the neural network may include determining the weights of the neural network by optimizing a performance function according to any known optimization technique, wherein the first ODE solver is applied to the neural ODE block. In the forward pass during a typical training iteration, performance values ​​(i.e., values ​​that can be used to indicate performance) can be obtained through the following steps:

[0077] - Training elements (e.g., input images or input instances) are input or fed into the neural network. More specifically, training elements can be input into the input layer of an upstream block of the neural network;

[0078] - Feed the output of the upstream block to the first ODE solver;

[0079] - Apply the first ODE solver to the neural ODE block;

[0080] - Feed the output of the first ODE solver to the downstream block; more specifically, feed the output of the first ODE solver to the layer downstream of the neural ODE block;

[0081] - Feed the output of downstream blocks (e.g., all layers downstream of the neural ODE block in a neural network) to the performance function;

[0082] - Input the ground truth values ​​of the training elements into the performance function;

[0083] - Performance values ​​are calculated using a performance function based on a comparison between the output of the downstream block and the ground truth.

[0084] In embodiments where the neural network does not include upstream and downstream blocks, or has a neural ODE block and one of the upstream and downstream blocks, the method used to obtain performance can be adjusted. For example, instead of inputting training elements into the neural network, the method can include feeding training elements directly to the neural ODE block via an input layer. Similarly, the output of the first ODE solver can be used as the output of the downstream block.

[0085] In many cases, a loss value can be calculated. In some embodiments, the performance function can be a loss function that determines the loss value based on the output of the downstream block and the ground truth. In some embodiments, a loss function is used in addition to the performance function. For example, the performance function can be an accuracy function. In this case, the loss function can be additionally used to train the weights and parameters of the upstream and / or downstream blocks. Based on the loss value, the parameters and / or weights of the neural network can be adapted.

[0086] However, during the training of the neural ODE block, the neural network that might describe the vector field of the ODE is not directly trained. Instead, the numerical solution is optimized by evaluating the neural network at discrete time points. For this purpose, a second ODE solver can be used to ensure that the neural network is trained to adequately correspond to the ODE flow. The second ODE solver is occasionally applied to the neural ODE block during training, either periodically or non-periodically (e.g., according to a schedule), and the two performance values ​​can be obtained and compared. Since the solution of the neural ODE block should be independent of the type of solver used if it can be described by an analytical ODE flow, the output of the performance function obtained when using the first ODE solver and the output of the performance function obtained when using the second ODE solver should not differ by a significant margin. If this is not the case, it is concluded that the neural ODE is no longer independent of the type of solver used and therefore cannot be described by the ODE flow. In this case, techniques and methods derived from the ODE domain may be unsuitable.

[0087] The degree to which a first performance value differs from a second performance value can be expressed using the concept of difference. That is, the difference can be determined by any means or function of comparison that indicates the level of difference or similarity between the two values. This difference can also be referred to as the result of a comparison. The difference between the output of the performance function obtained when using a first ODE solver and the output of the performance function obtained when using a second ODE solver can be quantified in various ways, such as by subtracting one output from another, by determining the distance between the outputs (e.g., Euclidean distance), or by determining the ratio between the outputs by dividing one output by another, etc. In some embodiments, a magnitude of the difference, such as a subtraction operation, can be used. In other embodiments, it may be sufficient to ensure that the second performance value indicates performance equal to or better than that indicated by the first performance value. The difference can indicate whether there is a performance degradation due to the use of the first ODE solver, which in itself indicates that the neural ODE block has not been trained to adequately approximate the ODE flow. For example, after determining the magnitude of the difference, it can be compared, for example, to a threshold to determine whether to adjust the accuracy parameter.

[0088] If the magnitude of the difference exceeds the threshold, the accuracy parameter of the first ODE solver is adjusted.

[0089] In some embodiments, such as when the first ODE solver is a fixed-step solver, the accuracy parameter is the step size. If the difference exceeds a threshold, the step size can be increased. In some embodiments, if the difference is less than a threshold, or if the difference is less than or equal to a threshold, the step size can be decreased.

[0090] In some embodiments, such as when the first ODE solver is an adaptive step-size solver, the accuracy parameter is tolerance. If the difference exceeds a threshold, the tolerance can be decreased. In some embodiments, if the difference is less than or equal to a threshold, the tolerance can be increased.

[0091] Reference Figure 4b Let me describe the process in detail.

[0092] As an optional component, the training system 100 may include a sensor interface (not shown) for obtaining sensor data from one or more sensors. Based on this sensor data, the measurement results can be used as training data elements for training the neural network. For example, the measurement results may include or be based on readings from one or more sensors. Also regarding... Figure 2 To describe the sensor interface.

[0093] The various details and aspects of the operation of the training system 100, including its optional aspects, will be further explained with reference to Figure 4.

[0094] Generally, the training system 100 may be embodied as a single device or apparatus, such as a workstation, e.g., a laptop- or desktop-based workstation, or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory (e.g., RAM) or non-volatile memory (e.g., flash memory). Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, e.g., as field-programmable gate arrays (FPGAs) and / or graphics processing units (GPUs). Generally, each functional unit of the training system 100 may be implemented as a circuit. It should be noted that the training system 100 may also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers employing cloud computing.

[0095] Figure 2 A system 200 for using a trained neural network according to an embodiment of the present invention is illustrated. The trained neural network may include neural ODE blocks. The trained neural network may further include upstream blocks and / or downstream blocks. An upstream block can be used to refer to a layer in the neural network located upstream of a neural ODE block, while a downstream block can be used to refer to a layer in the neural network located downstream of a neural ODE block. System 200 can be communicatively coupled to a computer-controlled entity operating in an environment. For example, the computer-controlled entity may be an autonomous vehicle, robot, or other device. The environment of the computer-controlled entity may refer to the physical environment of the computer-controlled entity, such as its surroundings.

[0096] System 200 may include a data interface 220 and a processor subsystem 240, which may be configured to communicate internally via data communication 221. Data interface 220 may be used to access model data 040 that defines a neural network. The model data may include at least a set of parameters (e.g., weights) of the neural network. It may be trained according to the training methods described herein, for example, by… Figure 1 The training system 100 is used to train the neural network. The system 200 can also be configured to train the neural network in addition to applying it; for example, the system 200 can be used in conjunction with... Figure 1 The training system has 100 combinations.

[0097] The processor subsystem 240 can be configured to access data 040 during operation of system 200 and using data interface 220. For example, as Figure 2As shown, data interface 220 can provide access 222 to external data storage device 022, which may include the data 040. Alternatively, data 040 can be accessed from an internal data storage device that is part of system 200. Alternatively, data 040 can be received from another entity via a network. Generally, data interface 220 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 022 can take any known and suitable form.

[0098] The processor subsystem 240 can be configured to: during operation of the system 200 and using the data interface 220 to acquire trained models and data, such as sensor data, and use neural ODE blocks of neural networks to perform decision-based tasks, such as classification tasks.

[0099] In some instances, processor subsystem 240 can be configured to perform classification tasks (such as image classification), wherein an input instance (such as an image) is acquired, for example, via sensor interface 260, and the input instance is classified according to an ODE modeled by a neural ODE block. Based on the classification obtained for the input instance, system 200 can accordingly control communicatively coupled entities.

[0100] As an optional component, system 200 may include a sensor interface 260 for acquiring sensor data 224 indicative of the state of a computer-controlled entity from one or more sensors. As a non-limiting example, a single sensor 072 is shown in the figure. Sensor interface 260 may internally communicate with processor subsystem 240 via data communication 223. Generally, sensor data 224 may include measurements of one or more physical quantities communicatively coupled to the computer-controlled entity of system 200 and / or the environment of the computer-controlled entity (e.g., the controlled device and / or the environment in which the device interacts).

[0101] Sensor interface 260 can be configured for various types of sensor signals, such as measurements of various physical quantities like temperature, pressure, light intensity, motion, and acceleration, and various signals that can be used to determine these physical quantities, such as video signals, radar / liDAR signals, ultrasonic signals, etc. One or more sensors 072 can be located within a computer-controlled entity, but remote sensing is also possible. Sensor interface 260 can also access sensor data from other locations, such as data storage devices or network locations. Sensor interface 260 can have any suitable form, including but not limited to low-level communication interfaces such as I2C or SPI-based data communication, data storage interfaces such as memory interfaces or persistent storage interfaces, or personal network, local area network, or wide area network interfaces such as Bluetooth, Zigbee, or Wi-Fi interfaces, or Ethernet or fiber optic interfaces. Sensor 072 can be included in system 200.

[0102] As an optional component, system 200 may include a display output interface or any other type of output interface (not shown) for outputting one or more determined classifications or decisions to a presentation device (such as a display). For example, the display output interface may generate display data for the display, thereby enabling the display to present one or more classifications or decisions in a sensory-perceptible manner (e.g., as a visualization on a screen).

[0103] The determined classifications, predictions, and / or decisions can be used to train and / or control a computer-controlled system. For example, the computer-controlled system can be system 200 itself, or system 200 can communicate with an external computer-controlled system to train and / or control the external computer-controlled system, for example, by providing the determined classifications or decisions; or by determining and providing interactive strategies or control commands based on the classifications or decisions.

[0104] For example, system 200 may be a computer-controlled system for interacting with an environment, such as a robotic system. System 200 may include an actuator interface (not shown). The actuator interface can be used to provide actuator data to the actuator, thereby enabling the actuator to perform actions in the environment. For example, the actuator may be an electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. Processor subsystem 240 may be configured to determine actuator data based on a determined classification or judgment, for example, to move a robotic arm to a desired position. The actuator may be part of system 200.

[0105] System 200 can also communicate with an external computer-controlled system for training and / or controlling it. For example, system 200 may be a subsystem of a vehicle, and the neural network may correspond to a model of the vehicle's mechanical components. In this case, system 200 may, for example, provide the determined classification or judgment to another subsystem of the vehicle for controlling the vehicle based on the determined classification or judgment. Alternatively, system 200 itself may determine control signals for controlling the vehicle based on the judgment or classification. Many variations will be conceived by those skilled in the art.

[0106] Generally, system 200 can be embodied as a single device or apparatus, such as a workstation, e.g., a laptop- or desktop-based workstation, or a server. For example, system 200 can be an embedded computer, such as one built into an autonomous vehicle or robotic agent, or part of a control system of a computer-controlled entity. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory (e.g., RAM) or non-volatile memory (e.g., flash memory). Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, e.g., as field-programmable gate arrays (FPGAs) and / or graphics processing units (GPUs). Generally, each functional unit of the system may be implemented as a circuit. It should be noted that system 200 can also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers employing cloud computing.

[0107] Using neural ODEs or neural ODE blocks within neural networks allows a multitude of tools and techniques developed in the field of ODE research to be applied to neural networks. However, in order to fully enjoy the theoretical advantages of neural ODEs, it is crucial that the neural network approximates the real analytical ODE solution sufficiently, which is often difficult to achieve.

[0108] To evaluate whether the approximation is close enough to the analytical ODE solution, a performance function such as a loss function can be used. For ease of illustration, this method will be described with reference to the Empirical Risk Minimization (ERM) problem as defined below. Any loss function (derived from...) can be used. l (Representations such as L2 loss, cross-entropy error, etc.)

[0109]

[0110] in:

[0111] It is the training data set.

[0112] It is a non-negative loss function, and

[0113] f It has weight w A neural ODE model (e.g., a neural network including neural ODE blocks) is shown below:

[0114]

[0115] in It is a neural network, and u and d These represent the upstream and downstream layers, respectively. In other words, f u This represents the layer upstream of the neural ODE block in the neural network, and f d This represents a layer in a neural network that is downstream of the neural ODE block.

[0116] Analytical flow defined as a dynamical system:

[0117]

[0118] Due to the vector field of the dynamic system f v It is defined by a neural network or a neural network block, therefore the evaluation It is intractable and uses the numerical scheme Ψ. t To calculate Ψ can be classified as a type of fixed step size. h = TK -1 The method could be an adaptive step-size solver. This applies to the per-initialization problem (IVP) of the neural ODE. z n (0)= f u ( x n The trajectory calculated using a specific solver Ψ is defined as follows: , k =1... K ,in K It is uniquely defined for fixed-step solvers.

[0119] Although the above description relates to loss functions, the invention is not limited thereto. In some embodiments, the performance function may be an accuracy function, and the performance value may be an accuracy value. That is, the loss function used to train a neural network (e.g., upstream blocks and / or downstream blocks) may be accompanied by different performance functions, such as accuracy functions. For example, this may be applicable in classification neural networks.

[0120] In other words, the neural network can be trained iteratively by optimizing the loss function (e.g., using the empirical risk minimization problem described above). At various points during training, such as periodically after a defined number of iterations, the parameters of the first ODE solver can be adjusted by comparing the performance of the neural network using a first ODE solver with that using a second ODE solver. For example, training the neural network (e.g., updating the parameters and weights in the upstream and downstream blocks) can be performed for up to 49 iterations, and at the 50th iteration, the accuracy parameter of the first ODE solver can be adjusted according to the method described herein. The training of the neural network (e.g., updating the parameters and weights in the upstream and downstream blocks) can then be repeated for another given number of iterations, and in the next iteration, the accuracy parameter of the first ODE solver can be re-evaluated by comparing performance values ​​and adjusted accordingly. This process can be repeated, for example, until training is complete.

[0121] It's also possible to measure the performance of a neural network by calculating an accuracy value using an accuracy function. This might be applicable, for example, in classification neural networks. An accuracy value can be defined as the number of correct predictions out of a total number of predictions. In machine learning, the output of a neural network used for a classification task is typically a vector containing values ​​for each of several classes. For example, if a classification neural network is trained to classify an input into one (or more) of five classes, this vector might have the format (0.1, 0.2, 0.5, 0.1, 0.1). This vector can be transformed such that the largest entry is converted to 1, and the other entries are converted to 0. In the example above, the provided vector would be converted to (0, 0, 1, 0, 0). This vector can then be compared to a second vector that encodes the true class (e.g., the ground truth). If the two vectors match, the prediction is correct. This process can be repeated for each training element in a batch of training data. Therefore, accuracy can be calculated as the ratio of correct predictions to overall predictions. It's important to understand that the accuracy function and the resulting accuracy value indicate the accuracy of the neural network. However, the accuracy parameters of the first ODE solver are parameters of the first ODE solver and should not be confused with the accuracy function of the neural network and the obtained accuracy values.

[0122] In other embodiments, the performance value may be the output of the corresponding ODE solver. In other words, comparing the performance of a neural network using a first ODE solver with the performance of a neural network using a second ODE solver may include directly comparing the outputs of the first and second ODE solvers.

[0123] During the training of a neural ODE model (e.g., a neural network comprising blocks of neural ODEs), the neural network describing the vector field of the ODE is not directly trained. Instead, the numerical solution is optimized for evaluating the neural network at discrete time points.

[0124] Since numerical solvers play a crucial role in approximating solutions to ODEs, the impact of the choice of numerical method on the training of neural ODE models should be considered. In particular, the discretization step of the numerical solver may affect the resulting ODE flow. Figure 3a and 3b The graphic illustration illustrates this effect.

[0125] Figure 3a The figure shows a graph of the numerically computed trajectory of the initial value problem for a neural ODE model trained with a small step size. Figure 3a It is shown that when a neural network is trained using a fixed-step ODE solver with small steps, the numerically computed trajectories of individual IVPs do not intersect, which is an indication that the learned flow approximates the true solution of the ODE.

[0126] Figure 3b The graph shows the numerically computed trajectory of the initial value problem for a neural ODE model trained with a large step size. Figure 3a on the contrary, Figure 3b This demonstrates that if a neural network is trained using a fixed-step ODE solver with a large step size, the trajectories of the IVP do indeed intersect. This behavior indicates that numerical solutions for solvers with large step sizes do not always correspond to the true solutions of the ODE. The discretization error of such solvers is so large that the resulting numerical solutions no longer maintain the properties of the ODE solutions.

[0127] although Figure 3a and 3b The description involves using a fixed-step ODE solver, but it's important to understand that this is similar to using an adaptive-step ODE solver. However, in the case of an adaptive-step ODE solver, the tolerance can be varied, rather than the step size. For example, when an adaptive-step ODE solver used to train a neural network uses a sufficiently low tolerance, the IVP trajectories will not intersect; while when the tolerance of an adaptive-step ODE solver used to train a neural network is higher, the IVP trajectories will intersect.

[0128] Tolerance and step size are both examples of accuracy parameters.

[0129] Figure 3a and 3b The diagram illustrates that when the ODE solver has a sufficiently small step size, or similarly a sufficiently low tolerance, the neural network exhibits the same behavior as the true analytical solution of the ODE. However, when the ODE solver's step size or tolerance is too high, the neural network's behavior no longer corresponds to the behavior of the true analytical solution of the ODE. In other words, with a large step size or high tolerance of the ODE solver, the solution becomes dependent on the choice of ODE solver. Therefore, if a neural network is trained using an ODE solver with a sufficiently small step size or sufficiently low tolerance, testing the neural network with another ODE solver with an even smaller step size or lower tolerance should achieve the same or similar accuracy. However, if the ODE solver used to train the neural network has an excessively large step size or tolerance, a performance comparison with another ODE solver with a smaller step size or lower tolerance will indicate a difference. That is, a sharp drop in test performance may be observed.

[0130] By testing the neural network using a second ODE solver different from the one used to train it, and comparing the performance of the neural network obtained from each ODE solver, it becomes possible to determine the step size—the step size at which the ODE interpretation becomes effective (e.g., exhibiting the behavior of the ODE flow, regardless of the choice of numerical solver). While using very small step sizes and very low tolerances makes it more likely to train the neural network to properly approximate the ODE, unnecessarily small step sizes or low tolerances can require additional computation, and therefore additional training and time, and may also require additional computational power and resources. Therefore, it is desirable to identify a step size or tolerance that is as large as possible, but still trains the neural network to approximate the ODE as an ODE flow. The following Figure 4a and 4b The description relates to computer implementation methods for training neural networks based on these principles.

[0131] Figure 4a A detailed example of training a neural network comprising neural ODE blocks is shown. The neural network may further include upstream and / or downstream blocks. More specifically, Figure 4a The forward pass of the training iteration is illustrated. Elements or batches or training data TD 410 are fed into the upstream layer UL 420 of the neural network, which is, for example, a layer upstream of the neural ODE block N-ODE 440 in the neural network. The output of the upstream layer UL 420 is then fed into the first ODE solver ODE-S1430. In some embodiments, as will be... Figure 4cAs indicated, there may be no upstream block between the input of the training data and the neural ODE block. In this embodiment, the training data TD 410 can be directly input into the first ODE solver ODE-S1 430, for example, via an input layer (which is an example of an upstream layer). The first ODE solver ODE-S1 430 can be initialized with an accuracy parameter. For example, if the first ODE solver is a fixed-step ODE solver, the accuracy parameter can be the step size, or, for example, if the first ODE solver is an adaptive-step ODE solver, the accuracy parameter can be the tolerance. The first ODE solver ODE-S1 430 is then applied to the neural ODE block N-ODE 440. The output of the application of the first ODE solver ODE-S1 430 to the neural ODE block N-ODE 440 is then fed into the downstream layer DL450 of the neural network. The downstream layer DL 450 of the neural network includes a layer downstream of the neural ODE block N-ODE 440. The output of the downstream layer DL 450 is referred to as the first output OUT-1 460 and is fed into the performance function PF 470. In some embodiments, as will be... Figure 4c As indicated, there may be no downstream block between the neural ODE block and the output of the neural network. That is, the output of the neural ODE block can be, for example, the output OUT-1460 of the neural network via the output layer. The output layer is an example of a downstream layer. Typically, the performance function PF470 is optimized over many iterations of training by adjusting the parameters and weights of the neural network—including the parameters and weights of each of the upstream layer UL 420, the downstream layer 450, and the neural ODE block N-ODE440. For example, the performance function PF470 can be one of the following: a loss function, an accuracy function, the direct output of the neural ODE block, or the direct output of the neural network.

[0132] Despite Figure 4a The ODE block is represented as an independent block, but in some embodiments, the neural ODE block N-ODE 440 may include a first ODE solver ODE-S1 430.

[0133] To illustrate the use of the principles described above to train a neural network, a diagram is shown. Figure 4b , Figure 4b and Figure 4a The difference lies in the elements marked with dashed lines.

[0134] Intermittently during training, the forward pass of the training iterations is adapted to also use a second ODE solver. In some embodiments, the second ODE solver may be included in the neural ODE block or may be external to it. Figure 4b The diagram illustrates the adapted forward pass of the training iteration.

[0135] For reference Figure 4a As described, a batch of training data TD 410 is input into the upstream layer UL 420. The upstream layer UL 420 can also be referred to as the upstream block. The output of the upstream block UL 420 is then fed into the first ODE solver ODE-S1 430 and the second ODE solver ODE-S2 435. The output of the upstream block UL 420 can be fed into or provided to the two solvers ODE-S1 and ODE-S2 simultaneously or sequentially in any order. The output of the upstream block is an example of the layer output upstream of the neural ODE block.

[0136] In addition to the reference Figure 4a In addition to the forward pass described and represented by solid arrows, additional aspects represented by dashed lines and arrows are also described. The combination of the forward pass represented by solid lines and arrows and the additional aspects represented by dashed lines and arrows can be referred to as the adapted forward pass. Each of the first ODE solver ODE-S1 430 and the second ODE solver ODE-S2 435 is independently applied to the neural ODE block N-ODE 440.

[0137] Each of the first ODE solver ODE-S1 430 and the second ODE solver ODE-S2 435 can independently pass their respective outputs to the downstream layer DL 450. The downstream layer DL 450 can be collectively referred to as the downstream block.

[0138] The output of the second ODE solver ODE-S2 435 is processed and disposed of by the downstream block DL 450 independently of the output of the first ODE solver ODE-S1 430. In some embodiments, the output of the second ODE solver ODE-S2 435 is sequentially fed into the downstream block DL 450 before or after the output of the first ODE solver ODE-S1 430, or the outputs of both the first ODE solver ODE-S1 430 and the second ODE solver ODE-S2 435 can be passed to the downstream block DL 450 simultaneously, although they are processed completely independently. In some embodiments, the downstream block can be replicated such that the output of the second ODE solver ODE-S2 435 is fed into the replicated downstream block DL 450 in a manner completely independent of the processing of the output of the first ODE solver ODE-S1 430.

[0139] For reference Figure 4aAs described, the first output from downstream block DL 450 is denoted as OUT-1 460. First output OUT-1 460 is the result of the output from the first ODE solver ODE-S1 430 being fed to downstream block DL 450. The second output from downstream block DL 450 is denoted as OUT-2 465. Second output OUT-2 465 is the result of the output from the second ODE solver ODE-S2 435 being fed to downstream block DL 450. First output OUT-1 460 and second output OUT-2 465 are obtained independently via downstream block DL 450.

[0140] Each of the first output OUT-1 460 and the second output OUT-2 465 can be input into a performance function. The first output OUT-1 460 can be fed into performance function PF 470. The second output OUT-2 465 can be fed into performance function PF475. Performance functions PF 470 and PF 475 can be the same performance function, and... Figure 4b The performance functions are represented as different functions in the diagram to emphasize that, although these performance functions are preferably identical in form, they are independently applied to each of outputs OUT-1 460 and OUT-2 465 and produce different performance values. In other words, a first performance value is obtained from performance function PF470, which is applied to the first output OUT-1 460 of downstream block DL 450. The first performance value corresponds to the use of the first ODE solver ODE-S1 430. The second performance value corresponds to the use of the second ODE solver ODE-S2 435. In other words, the first performance value represents the performance obtained when the first ODE solver ODE-S1 430 is applied to neural ODE block N-ODE440. The second performance value represents the performance obtained when the second ODE solver ODE-S2 435 is applied to neural ODE block N-ODE 440. Performance functions PF470, 475 can be loss functions, accuracy functions, etc., as described above.

[0141] In some embodiments, the first output OUT-1 460 and the second output OUT-2 465 can be considered as a first performance value and a second performance value, respectively. In this case, performance functions PF 470, 475 can correspond to capturing the first output OUT-1 460 and the second output OUT-2 465. For example, if the first output OUT-1 460 and the second output OUT-2 465 are significantly different (e.g., the difference exceeds a threshold), a performance degradation can be detected. For example, if the distance between the first output OUT-1 460 and the second output OUT-2 465 is large (e.g., above a threshold), the accuracy parameters of the first ODE solver can be adjusted, for example, by reducing the step size or reducing the tolerance.

[0142] Then, in block 480, the first performance value is compared with the second performance value. The comparison of the first and second performance values ​​may include determining the difference between them. In some embodiments, the accuracy parameter of the first ODE solver ODE-S1 430 may be adjusted based on whether the magnitude of the difference between the first and second performance values ​​exceeds a threshold. Determining the difference between the first and second performance values ​​may include: subtracting the first performance value from the second performance value (or vice versa); dividing the first performance value by the second performance value (or vice versa); determining the distance between the first and second performance values ​​(e.g., Euclidean distance), etc. For example, if the first and second performance values ​​are similar, their quotient should be close to 1.

[0143] For example, if the magnitude of the difference between the first performance value and the second performance value exceeds a threshold, a performance degradation is observed. To address this performance degradation and adapt the first ODE solver ODE-S1 430 to improve its discretization, the step size or the tolerance can be reduced. In embodiments where the first ODE solver ODE-S1 430 is a fixed-step solver, the step size of the first ODE solver can be reduced. Reducing the step size results in finer discretization, which may be necessary to obtain an approximation of the ODE flow. For example, the step size of the first ODE solver can be reduced by a predetermined percentage. The current step size can be multiplied by a number between 0 and 1 to obtain a new step size. In embodiments where the first ODE solver ODE-S1 430 is an adaptive-step solver, the tolerance of the first ODE solver can be reduced. The tolerance of the ODE solver refers to the permissible estimation error used to adapt the step size in an adaptive-step ODE solver. As the tolerance increases, the magnitude of the permissible error increases, thus allowing the use of larger step sizes. Conversely, as the tolerance decreases, the permissible error decreases, and a smaller step size is used. A smaller step size means using more steps (e.g., a higher number of steps), which reduces the resulting error.

[0144] The adaptive step-size solver can be parameterized using an error weighting function or a tolerance. For example, the tolerance can be adjusted by simultaneously adjusting both the relative and absolute tolerances. In some embodiments, adjusting the tolerance of the adaptive step-size solver may include adjusting the error weighting function.

[0145] In some embodiments, when the first ODE solver and the second ODE solver are adaptive step-size solvers, the tolerance of the second ODE solver can be set based on the tolerance of the first ODE solver. For example, the tolerance of the second ODE solver can be a small fraction of the tolerance of the first ODE solver. More specifically, the relative tolerance of the second ODE solver can be set to a small fraction of the relative tolerance of the first ODE solver, and the absolute tolerance of the second ODE solver can be set to a small fraction of the absolute tolerance of the first ODE solver.

[0146] If the magnitude of the difference between the first performance value and the second performance value falls below a threshold, or in some embodiments, when the magnitude of the difference equals the threshold, the accuracy parameter can be reduced. A magnitude of difference falling within the threshold indicates that the neural ODE block N-ODE 440 is approximating the analytical ODE flow and is independent of the numerical solver used to solve it. In this case, it is possible to increase the accuracy parameter to reduce the time and / or computational power and resources required for training. For example, if the first ODE solver ODE-S1 430 is a fixed-step ODE solver, the step size can be increased. For example, the current step size of the first ODE solver ODE-S1 430 can be multiplied by a number greater than 1 to obtain a new step size. In some embodiments, if the first ODE solver ODE-S1 430 is an adaptive-step solver, the tolerance of the first ODE solver ODE-S1 430 can be increased.

[0147] It uses both a first ODE solver and a second ODE solver, and as follows: Figure 4bThe adapted forward pass shown can be repeated at regular or irregular intervals during training. For example, the adapted forward pass can be applied after a predefined number of iterations of training (e.g., every 50 iterations). The adapted forward pass can be repeated periodically or non-periodically until training ends or is completed. The adapted forward pass can be repeated according to a schedule, and the intervals between repetitions of the adapted forward pass can be regular or irregular. In some embodiments, the use of the adapted forward pass can be triggered by an event, such as when the loss value changes by more than a threshold within a predetermined number of iterations. In some embodiments, the frequency of repeated adapted forward passes can be based on the magnitude of the difference between a first performance value and a second performance value, as well as the value of a threshold. For example, the frequency of the adapted forward pass can also be reduced if the magnitude of the difference between the first performance function and the second performance function is decreasing relative to a threshold.

[0148] The first ODE solver ODE-S1 430 and the second ODE solver ODE-S2 435 can use different numerical solver methods. The second ODE solver ODE-S2 435 can have an expected discretization error equal to or smaller than that of the first ODE solver ODE-S1 430. The second ODE solver ODE-S2 435 can have a higher convergence order than the first ODE solver ODE-S1 430. In other words, the second ODE solver ODE-S2 435 can be selected to have an expected discretization error equal to or smaller than that of the first ODE solver ODE-S1 430. As will be understood, the expected discretization error of the ODE solver may differ from the actual (e.g., the implemented) discretization error. However, this difference does not affect the effectiveness of the method described herein. That is, the method remains effective even if the actual discretization error of the second ODE solver exceeds that of the first ODE solver.

[0149] For example, the first ODE solver, ODE-S1 430, can be based on at least one of the Runge-Kutta method and multi-step methods. Examples of Runge-Kutta methods include the Euler method, the midpoint method, the Heun method, classical higher-order Runge-Kutta methods (e.g., the classical fourth-order Runge-Kutta method), and the Ralston method. Examples of multi-step methods include Adams-Bashforth, Adams-Moulton, and the backward differential formula (BDF). These methods can be used at various orders. For example, Adams-Bashforth methods of orders 1 to 12 can be used. In some embodiments (such as embodiments where the first and second ODE solvers are adaptive step-size solvers), the second ODE solver is based on an embedded higher-order method. For example, if the first ODE solver is based on the Euler method, the second ODE solver can be based on the midpoint method. As another example, if the first ODE solver is based on the second-order Bogacki-Shampine method, then the second ODE solver can be based on the third-order Ralston method. As another example, if the first ODE solver is based on the Adams-Bashforth method, then the second ODE solver can be based on the Adams-Moulton method. Further examples of pairings between the first and second ODE solvers include Dormand-Prince, Bogacki-Shampine, Heun-Euler, and Tsitouras 5 / 4. It should be understood that this list is merely exemplary and not exhaustive, and the methods described herein are not limited to this.

[0150] This process can also be described in algorithmic form, as shown in the pseudocode below:

[0151] initialize starting accuracy_param;

[0152] while Training do

[0153] if Iteration % 50 == 0 then

[0154] test_performance = calculate_performance_second_ODE_solver;

[0155] if difference(test_performance, train_performance)>0.1then

[0156] new_accuracy_param = 0.5*accuracy_param;

[0157] else

[0158] new_accuracy_param = 1.1*accuracy_param;

[0159] end

[0160] end

[0161] end

[0162] Wherein, `train_performance` represents the performance of the first ODE solver, and `test_performance` represents the performance of the second ODE solver. Performance can be a loss function, accuracy, etc. In some embodiments, the test performance and training performance (e.g., the second performance value and the first performance value, respectively) can be multidimensional. In this case, determining the difference between the first performance value and the second performance value can include determining the Euclidean distance, etc. The values ​​used as the threshold (0.1), the frequency of the adapted forward pass (every 50 iterations), and the values ​​used for the adaptation accuracy parameters (0.5 and 1.1) are purely illustrative, and it is to be understood that the methods described herein are not limited thereto.

[0163] Throughout the training of a neural network, the parameters and weights of the network (e.g., upstream blocks, neural ODE blocks, and downstream blocks) can be updated over multiple iterations, for example, by optimizing the loss function. At different points during training, adapted forward passes are used to adjust the accuracy parameters of the first ODE solver. For example, training a neural network may include: 50 iterations during which the parameters and weights are updated by optimizing the loss function (this may be referred to as "normal forward pass"); one iteration during which the accuracy parameters of the first ODE solver are adjusted (e.g., by the adapted forward pass described above); followed by 50 iterations of normal forward pass, one iteration of adapted forward pass, and so on, until training ends. Although this example involves the periodic use of adapted forward passes, it is to be understood that this is not limiting. In some embodiments, adapted forward passes are applied occasionally or non-periodically at irregular intervals.

[0164] Although this description has so far mentioned examples where neural networks include upstream and downstream blocks, it is important to understand that these are optional. Figure 4cThe figure illustrates this embodiment. In some embodiments, the input may be fed to the first ODE solver ODE-S1 430 only via the input layer. In other words, in some embodiments, there may be no layer between the input layer and the neural ODE block N-ODE 440. Additionally or alternatively, the performance value may be calculated using the output OUT-1 460 of the ODE solver ODE-S1 430. In other words, in some embodiments, the output of the ODE solver may be the output of a neural network. There may be no layer between the neural ODE block N-ODE 440 and the output layer.

[0165] in other words, Figure 4c The illustration shows an embodiment in which the neural network consists of neural ODE blocks without additional upstream or downstream layers. For simplicity, the input and output layers of the neural network are not shown, as in the simplest case, these layers may simply pass one or more training elements or one or more outputs without providing any modifications to them. This is purely exemplary. In some embodiments, the neural network may include upstream blocks as previously described, or downstream blocks as previously described, or both.

[0166] exist Figure 4c In the illustrated embodiment, the adapted forward pass is similarly adjusted. That is, the training data TD410 can be input into the second ODE solver ODE-S2435. The output OUT-2465 of the second ODE solver ODE-S2435 can be input into the performance function PF475.

[0167] Despite Figure 4c Only one neural ODE block N-ODE 440 is depicted in this paper, but it should be understood that the methods described herein can be applied to neural networks that include multiple neural ODE blocks. In some such embodiments, each neural ODE block can be invoked by a corresponding ODE solver, or the same ODE solver can be used to invoke multiple neural ODE blocks.

[0168] Figure 5 A computer implementation method 500 for training a neural network including neural ODE blocks is illustrated according to an embodiment of the present invention. The neural network includes neural ODE blocks. In some embodiments, the neural network may further include upstream blocks and / or downstream blocks. An upstream block refers to a layer in the neural network upstream of a neural ODE block, and a downstream block refers to a layer in the neural network downstream of a neural ODE block. Method 500 may correspond to Figure 1 The operation of the training system 100. However, this is not a limitation, as another system, apparatus, or device may also be used to perform method 500.

[0169] Method 500 may include, in an operation entitled “Accessing Neural Network and Training Data”, access 510 defining the model data of the neural network, and access 510 including a training dataset comprising multiple batches of training data. Each element of the training dataset may include an input instance and a corresponding ground truth value.

[0170] Method 500 may include, in an operation entitled "Training a Neural Network", training a 520-fold neural network on a training dataset and using a first ODE solver. In some embodiments, the neural network may include the first ODE solver. In other embodiments, the first ODE solver may be external to the neural network.

[0171] During training 520, the method may further include accessing a second ODE solver. The second ODE solver is preferably different from the first ODE solver. In some embodiments, the second ODE solver is based on a different numerical solution method than the first ODE solver. Training the neural network 520 may include: initializing the accuracy parameters of the first ODE solver and adjusting the accuracy parameters during training.

[0172] During the forward pass of training the neural network 520, method 500 may include: obtaining 530 the first output of the neural network in an operation entitled "obtaining the first output of the neural network".

[0173] Obtaining the first output of the 530 neural network may include, in an operation entitled “Providing a First ODE Solver”, providing the layer output of an upstream layer of the neural ODE block to the first ODE solver. In some embodiments, the layer output of an upstream layer of the neural ODE block may be an input to the neural network, such as training data. In some embodiments, the layer output of an upstream layer of the neural ODE block may be the output of an upstream block. That is, after a batch of training data has been fed into the upstream layer (e.g., the upstream block) of the neural network, the output of the upstream block can be obtained and fed into the first ODE solver.

[0174] Obtaining the first output of the 530 neural network may include: in an operation entitled "Applying the First ODE Solver", applying the first ODE solver to the neural ODE block to obtain the output of the first ODE solver. In other words, the first ODE solver invokes the neural ODE block.

[0175] Obtaining the first output of the 530 neural network may include: in an operation entitled "Feeding the first ODE output downstream", feeding the output of the first ODE solver to a layer downstream of the neural ODE block. In some embodiments, feeding the output of the first ODE solver may include: feeding the output of the first ODE solver to a downstream block. In some embodiments, feeding the output of the first ODE solver may include: outputting the output of the first ODE solver via an output layer as the output of the neural network.

[0176] During the forward pass of training the neural network 520, method 500 may include obtaining the second output of the neural network 540 in an operation entitled "obtaining the second output of the neural network". The operations of obtaining the first output of the neural network 530 and obtaining the second output of the neural network 540 may be performed simultaneously, substantially simultaneously, or sequentially.

[0177] Obtaining the second output of the 540 neural network may include, in an operation entitled “Providing a Second ODE Solver,” providing the layer output of the upstream layer of the neural ODE block to the second ODE solver. In some embodiments, the layer output of the upstream layer of the neural ODE block may be the input to the neural network, such as training data. In some embodiments, the layer output of the upstream layer of the neural ODE block may be the output of the upstream block. That is, after the batch of training data has been fed into the upstream layer (e.g., the upstream block) of the neural network, the output of the upstream block can be obtained and fed into the second ODE solver.

[0178] Obtaining the second output of the 540 neural network can include: in the operation titled "Applying the Second ODE Solver", applying the second ODE solver to the 542 neural ODE block to obtain the output of the second ODE solver. In other words, the second ODE solver invokes the neural ODE block.

[0179] Obtaining the second output of the 540 neural network may include, in an operation entitled "Feeding the Second ODE Output Downstream", feeding the output of the second ODE solver 543 into a layer downstream of the neural ODE block. In some embodiments, feeding the output of the second ODE solver 543 into a layer downstream of the neural ODE block may include feeding the output of the second ODE solver into a downstream block. In some embodiments, feeding the output of the second ODE solver 543 may include outputting the output of the second ODE solver as the output of the neural network. In other words, in some embodiments, the layer downstream of the neural ODE block may be an output layer.

[0180] After obtaining the first output and the second output of the neural network, method 500 includes: in an operation entitled "Evaluating a performance function using the first output", using the first output of the neural network to evaluate a performance function 550 to obtain a first performance value. The performance function can quantify the performance characteristics of the neural network with respect to a batch of training data. That is, the first performance value corresponds to the performance value obtained when a first ODE solver is applied to a neural ODE block. The performance function can be a loss function, an accuracy function, etc. The first performance value can be a loss value, an accuracy value, etc. In some embodiments, the performance value can be the output of the corresponding ODE solver or the output of the neural network. The performance function can be a loss function, which can take the form of average cross-entropy loss with or without a regularization term, or any other loss function. In some embodiments, the performance function can be a loss function.

[0181] Sequentially, simultaneously, or substantially simultaneously, method 500 includes: in an operation entitled "Evaluating a performance function using a second output," using the second output of the neural network to evaluate the performance function 560 to obtain a second performance value. That is, the second performance value corresponds to the performance value obtained when a second ODE solver is applied to the neural ODE block. The performance function used in operation 550 may be the same performance function used in operation 560. The first and second performance values ​​are determined independently of each other.

[0182] After evaluating the performance function and obtaining a first performance value and a second performance value, the method may further include: in an operation entitled “determining the difference”, determining the difference between the first performance value and the second performance value.

[0183] The method may further include, in an operation entitled “comparing the difference to a threshold”, comparing the magnitude of the difference between the first performance value and the second performance value to a threshold 580.

[0184] The method further includes, in an operation entitled “Adjusting Accuracy Parameters,” adjusting the accuracy parameters of a first ODE solver based on the difference between a first performance value and a second performance value. In some embodiments, adjusting the accuracy parameters of the first ODE solver may be based on a comparison of the magnitude of the difference with a threshold. In some embodiments, adjusting the accuracy parameters of the first ODE solver may be based on the ratio between the first performance value and the second performance value, or any other comparison metric. In some embodiments, for example, if the first ODE solver is a fixed-step solver, adjusting the accuracy parameters of the first ODE solver may include adjusting the step size of the first ODE solver. For example, if the magnitude of the difference is greater than a threshold, adjusting the accuracy parameters may include decreasing the step size of the first ODE solver. If the magnitude of the difference is less than a threshold, the step size of the first ODE solver may be increased. In some embodiments, decreasing the step size may include multiplying the current step size by a predetermined value, which is a number between 0 and 1. In some embodiments, increasing the step size may include multiplying the current step size by another predetermined value, which is a number greater than 1.

[0185] In some embodiments, such as if the first ODE solver is an adaptive step-size solver, the accuracy parameter may be a tolerance. For example, if the magnitude of the difference between the first performance value and the second performance value is greater than a threshold, adjusting the accuracy parameter may include decreasing the tolerance of the first ODE solver. If the magnitude of the difference between the first performance value and the second performance value is less than a threshold, adjusting the accuracy parameter may include increasing the tolerance of the first ODE solver. Increasing the tolerance of the first ODE solver may include multiplying the current tolerance by a predetermined value greater than 1. Decreasing the tolerance of the first ODE solver may include multiplying the current tolerance by a predetermined value between 0 and 1.

[0186] Operations 530 to 590 may be performed during the forward pass of a training iteration. These operations 530 to 590 may be repeated intermittently during the training of the neural network, such as periodically or non-periodically. In some embodiments, operations 530 to 590 may be repeated periodically, such as after a predetermined number of iterations (e.g., every 50 iterations), or non-periodically, until the training of the neural network is completed.

[0187] In some embodiments, method 500 further includes: outputting model data representing the trained neural network. In some embodiments, the method further includes: receiving an image as input to the neural network and classifying the image into at least one of a plurality of categories. For example, the image may be an image representation of the input, such as sensor data, audio, text, video, etc. Although the example of image classification has been used, it is to be understood that the method used to train the neural network can be applied to any feedforward neural network.

[0188] What will be understood is, generally speaking Figure 5 The operations of method 500 can be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, where applicable, the order being subject to a specific order necessary, for example, by input / output relationships. For example, operations 530 and 540 can be performed in any order or simultaneously, and operations 550 and 560 can be performed in any order or simultaneously. As another example, operation 550 can be performed after operation 530, and operation 560 can be performed after operation 540.

[0189] This method(s) can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. Similarly, Figure 6 As illustrated, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 600, for example, in the form of a machine-readable physical tag sequence 610 and / or as a sequence of elements with different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a temporary or non-temporary manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 6 The image shows CD 600.

[0190] It should be noted that the above embodiments are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments.

[0191] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or steps other than those recited in the claims. The article "a (or an)" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one," when preceding a list or group of elements, indicate the selection of all elements or any subset of elements from that list or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components may be embodied by the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A computer implementation method (500) for training a neural network, said neural network comprising a neural ordinary differential equation (ODE) block (440), said method comprising: - Access (510) defines the model data for the neural network; - Access (510) to a training dataset that includes multiple batches of training data (410), where each batch of training data includes multiple input images; as well as - The neural network is trained (520) on the training dataset (410) and using the first ordinary differential equation ODE solver (430); The method further includes accessing a second ODE solver, wherein the first ODE solver and the second ODE solver are fixed-step ODE solvers, and the second ODE solver has a higher convergence order than the first ODE solver. The training includes: initializing the accuracy parameters of the first ODE solver (430), and adjusting the accuracy parameters during the training by performing the following steps during the forward pass of the training iterations: - Obtain the first output (460) of the neural network (530) by the following steps: - Provide (531) the layer output upstream of the neural ODE block to the first ODE solver (430). - Apply (532) the first ODE solver (430) to the neural ODE block (440) to obtain the output of the first ODE solver (430); and - The output of the first ODE solver (430) is fed (533) into the layer downstream of the neural ODE block; - Obtain the second output (465) of the neural network described in (540) through the following steps: - Provide (541) the layer output upstream of the neural ODE block to the second ODE solver (435). - Apply (542) the second ODE solver (435) to the neural ODE block (440) to obtain the output of the second ODE solver (435); and - The output of the second ODE solver (435) is fed (543) into the layer downstream of the neural ODE block; - The first output (460) of the neural network is used to evaluate (550) the performance function (470) to obtain a first performance value, the performance function (470) quantifying the performance characteristics of the neural network with respect to a batch of training data; - Use the second output (465) of the neural network to evaluate the performance function (475) (560) to obtain a second performance value; - Adjust the (590) accuracy parameter based on the difference between the first and second performance values. - The first and second outputs of the neural network are classifications to at least one of a plurality of categories.

2. The computer implementation method (500) according to claim 1, comprising: - Determine the difference between the first performance value and the second performance value (570); - Compare the magnitude of the difference to a threshold (580); The adjustment of the accuracy parameter (590) includes: adjusting the accuracy parameter based on a comparison of the magnitude of the difference with a threshold.

3. The computer implementation method (500) according to claim 2, wherein the accuracy parameter is a tolerance, and wherein adjusting the accuracy parameter based on a comparison of the magnitude of the difference with a threshold comprises: If the magnitude of the difference is greater than a threshold, the tolerance of the first ODE solver (430) is reduced, and if the magnitude of the difference is less than a threshold, the tolerance of the first ODE solver (430) is increased.

4. The computer implementation method (500) according to claim 2, wherein the accuracy parameter is a step size, and wherein adjusting the accuracy parameter based on a comparison of the magnitude of the difference with a threshold comprises: If the magnitude of the difference is greater than the threshold, the step size of the first ODE solver (430) is reduced, and if the magnitude of the difference is less than the threshold, the step size of the first ODE solver (430) is increased.

5. The computer implementation method (500) according to any one of claims 1 to 4, wherein the second ODE solver (435) is selected to have the same or smaller expected discretization error as the first ODE solver (430).

6. The computer-implemented method (500) according to any one of claims 1 to 4, wherein the neural network is an image classifier.

7. The computer-implemented method (500) according to any one of claims 1 to 4, wherein the accuracy parameter is periodically adjusted during the training, such as after a predefined number of iterations.

8. The computer-implemented method (500) according to any one of claims 1 to 4, wherein the accuracy parameter is adjusted non-periodically during the training.

9. The computer implementation method (500) according to any one of claims 1 to 4 further includes repeatedly adjusting the accuracy parameter until the training of the neural network is completed.

10. The computer-implemented method (500) according to any one of claims 1 to 4, wherein the first ODE solver (430) and / or the second ODE solver (435) are based on at least one of the Runge-Kutta method and a multi-step method; and / or The second ODE solver (435) is based on a method that is used as a higher-order embedded solver than the first ODE solver (430).

11. The computer implementation method (500) according to any one of claims 1 to 4, further comprising outputting model data representing the trained neural network.

12. The computer-implemented method (500) according to any one of claims 1 to 4, further comprising receiving an image as input to the neural network and classifying the image into at least one of a plurality of categories.

13. A computer-readable medium (600) comprising temporary or non-temporary data (610) representing instructions, which, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any one of claims 1-12.

14. A system (100) for training a neural network, the neural network comprising a neural ordinary differential equation ODE block (440), the system (100) comprising: - Data interface (120), which is configured as follows: - Access the model data (040) that defines the neural network; - Access the training dataset (030) which includes multiple batches of training data, where each batch of training data includes multiple input images; - Processor subsystem (140), which is configured to: - The neural network is trained on the training dataset and using the first ODE solver (430). The data interface (120) is further configured to access a second ODE solver (435), wherein the first ODE solver and the second ODE solver are fixed-step ODE solvers, and the second ODE solver has a higher convergence order than the first ODE solver. The processor subsystem (140) is configured to train the neural network by initializing the accuracy parameters of the first ODE solver (430) and adjusting the accuracy parameters during the training period, wherein adjusting the accuracy parameters during the training period is performed by performing the following steps during the forward pass of the training iterations: - Obtain the first output (460) of the neural network through the following steps: - Provide the layer output upstream of the neural ODE block to the first ODE solver (430). - Apply the first ODE solver (430) to the neural ODE block (440) to obtain the output of the first ODE solver (430); and - The output of the first ODE solver (430) is fed into the layer downstream of the neural ODE block; - Obtain the second output (465) of the neural network through the following steps: - Provide the layer output upstream of the neural ODE block to the second ODE solver (435). - Apply the second ODE solver (435) to the neural ODE block (440) to obtain the output of the second ODE solver (435); and - The output of the second ODE solver (435) is fed into the layer downstream of the neural ODE block; - The first output (460) of the neural network is used to evaluate the performance function (470) to obtain a first performance value, the performance function (470) quantifying the performance characteristics of the neural network with respect to a batch of training data; - Use the second output (465) of the neural network to evaluate the performance function (475) to obtain a second performance value; - Adjust the accuracy parameters based on the difference between the first performance value and the second performance value.

Images