Neural network with layers for solving semi-definite programming

Abstract

A system (100) for applying a neural network to an input instance is disclosed. The neural network comprises an optimization layer for determining values of one or more output neurons from values of one or more input neurons by joint optimization parameterized by one or more parameters. An input instance is obtained. Values of the one or more input neurons to the optimization layer are obtained and an input vector of the one or more input neurons is determined therefrom. An output vector of the one or more output neurons is computed from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semi-definite program defined by the one or more parameters. Values of the one or more output neurons are determined from the respective computed output vector.

Description

TECHNICAL FIELD

[0001] The present invention relates to a system for applying a neural network to an input instance (e.g. an input image), and a corresponding computer-implemented method. The present invention further relates to a system for training a neural network, and a corresponding computer-implemented method. The present invention further relates to a computer-readable medium comprising instructions to perform the method or data representing parameters of a neural network. BACKGROUND

[0002] Autonomous devices have to make decisions based on the observed environment. For example, an autonomous driving system for a vehicle can use data from cameras and / or various other sensors to analyze traffic situations and use this information for controlling the vehicle. Similarly, a safety system for a non-autonomous car can analyze traffic situations to assist the driver, e.g. to brake if a dangerous situation occurs. In this example and in other settings, making a decision includes not only recognizing objects, but also making inferences, e.g. about which recognized objects or combinations of objects are dangerous. Making automated inferences also occurs in other control systems, medical image analysis, etc.

[0003] In these and other kinds of systems, neural networks, and in particular deep neural networks, can be used to make such inferences about the environment. A neural network typically comprises one or more layers, in each of which the values of one or more output neurons are computed based on the values of one or more input neurons. For example, a neural network can have at least 10, 100 or 1000 neurons, at least 2, 5 or 10 layers, etc. For example, the layers can differ in which input neurons are connected to which output neurons, and / or which functions are used to compute the values of the output neurons. Typically, one or more parameters of the layers are learnable, e.g. such parameters can include weights that determine how much the value of an output neuron is influenced by the values of the respective input neurons it is connected to. Various types of layers are known per se. For example, a convolutional neural network typically comprises an input layer, an output layer and one or more layers in between - typically called hidden layers. Such hidden layers can include convolutional layers, ReLU layers, pooling layers, fully connected layers, normalization layers, etc. For example, a combination of such layers can be used to perform various kinds of image classification tasks.

[0004] In the article "OptNet: Differentiable Optimization as a Layer in Neural Networks" by Brandon Amos and J. Zico Kolter (incorporated by reference and available at https: / / arxiv.org / abs / 1703.00443In the paper (available at http: / / arxiv.org / abs / 1506.06249), a deep neural network is proposed that is designed to capture constraints and dependencies between the hidden states of the neural network. To this end, a quadratic program is integrated as a layer in the network. This layer is inserted into the neural network to capture constraints and complex dependencies between the hidden states of the neural network. Both the output of this layer and the data defining the quadratic program problem can depend on the values of the previous layers. The output values of this layer are obtained by solving the quadratic program using a primal-dual interior point method. SUMMARY

[0005] According to a first aspect of the present invention, a system for applying a neural network to an input instance is proposed as defined by claim 1. According to a further aspect of the present invention, a system for training a neural network is proposed as defined by claim 10. According to a further aspect, a computer implemented method of applying and training a neural network is proposed as defined by claim 12 and claim 13, respectively. According to a further aspect of the present invention, a computer readable medium as defined by claim 14 and a computer readable medium as defined by claim 15 are proposed.

[0006] In various embodiments, the neural network comprises at least one optimization layer for determining values of one or more output neurons from values of one or more input neurons by joint optimization. Generally, in joint optimization, the values of the one or more output neurons are not computed independently from each other. For example, joint optimization can comprise minimizing a loss function based on the values of the one or more output neurons. Optimization can be joint in the sense that the output values that minimize the loss function depend on each other, e.g., for a given output neuron, its optimal value combined with a set of values for the other output values can be different from its optimal value combined with another set of values. In other words, the value of an output neuron that minimizes the loss function can depend on the values of the output neurons if the values of the other output neurons are fixed. Thus, the values of the output neurons can be computed in joint optimization. Joint optimization can be parameterized, e.g., the optimization problem to be solved can depend on one or more parameters.

[0007] Interestingly, computing the output vector can comprise solving a semidefinite program defined by the one or more parameters. Semidefinite programs are interesting because they allow expressing a relatively wide class of optimization problems that can be solved relatively efficiently. For example, semidefinite programs can comprise relaxations of MAXSAT problems. MAXSAT problems and related problems such as MAXCUT are of particular interest because they allow capturing a wide range of discrete logical relationships between the input and output values of the optimization layer, e.g., they allow efficiently capturing hard and / or global relationships between the input and output values of the optimization layer.

[0008] Interestingly, in various embodiments, such a relationship is defined by jointly optimized parameters, enabling the neural network to effectively learn the relationship between the input and output neurons of the optimization layer in an end-to-end manner. For example, the neural network can be trained in such a way that it learns, in the same process, how to determine the inputs to the optimization layer (e.g., as output of a convolutional layer, a ReLU layer, etc.) and / or what relationship to maintain between these inputs and the output of the optimization layer as defined by the set of parameters, and / or how to use the output of the optimization layer in further layers of the neural network (e.g., fully connected layers, further optimization layers, etc.).

[0009] However, as recognized by the inventors, MAXSAT problems and other kinds of discrete optimization are typically not suitable for being directly included in a neural network. For example, for training a neural network, it is typically desirable that the forward pass of the neural network (e.g., computing the output values from the input values) is amenable to backpropagation. For example, it is desirable if the input / output relationship of the optimization layer can be described as a function, preferably a differentiable and / or continuous function. Moreover, it is well-known that MAXSAT and related problems are difficult to solve exactly. In particular, MAXSAT is known to be NP-hard. This also makes it difficult to efficiently include such problems in a neural network.

[0010] Therefore, the inventors conceived to perform the optimization on vectors associated with the input and output values rather than on the values themselves. Such a vector optimization can be better amenable to being applied as a neural network layer. For example, by appropriately transforming the input values into vectors, performing the vector optimization, and transforming the output vectors back into values, a neural network layer with a continuous and / or differentiable parameterization can be obtained. Moreover, the vector optimization can be relatively efficiently solvable, e.g., a relaxation of a MAXSAT, which is an NP-hard optimization problem, can only provide an approximate solution to the problem, but provides such a solution much more efficiently than solving the original problem. Specific examples of vector optimizations obtained as relaxations of MAXSAT are provided below.

[0011] By determining an input vector for one or more input neurons (e.g., an input vector for each input neuron), the optimization layer can be applied to the values of the input neurons. The input vectors each comprise a plurality of values. In particular, for a MAXSAT relaxation, the vectors can comprise at least elements, where n is the number of values of the optimization layer, e.g., input variables / values, output variables / values, and also optionally auxiliary variables, as discussed in more detail later. For example, the input vector for an input neuron (x) can be determined from the scalar based on the values of the input neuron (x) as a predetermined vector (x) ). ​a scalar multiple of the predetermined vector, e.g. The predetermined vector may be referred to as a true direction.

[0012] Given an input vector, the output vector of the one or more output neurons can be determined by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters. The output vector typically has the same dimensionality as the input vector. For example, the optimization can be a MAXSAT relaxation or a MAXCUT relaxation, in other words, the output vector can correspond to an approximate solution to the respective optimization problem. However, other SDPs can also be used, e.g., SDPs obtained as vector relaxations of other optimization problems. Various optimization examples are provided below.

[0013] The value of the one or more output neurons can be determined from the output vector of the respective computation. For example, if the semidefinite program is a relaxation of a given optimization problem, e.g., a vectorized problem corresponding to a discrete optimization of values, the value can be determined as an approximate solution to the problem to which it is a relaxation. For example, the value of the output neuron ) can be determined based on a directional difference between the output vector of the output neuron ) and a predetermined vector ), e.g. In some examples, a random rounding is applied, e.g., by generating one or more random hyperplanes from the unit sphere and checking whether the output vector and the predetermined vector are on the same side of the one or more hyperplanes, to determine the value of the one or more output neurons.

[0014] Multiple optimization layers can be applied in a neural network, e.g., at the same layer or at multiple layers. For example, a sequence of multiple optimization layers can be applied subsequently. In various embodiments, the parameters for the joint optimization of multiple optimization layers are correlated, e.g., the multiple optimization layers are parameterized by the same or at least overlapping sets of parameters. For example, this can conceptually enable the multiple optimization layers to learn the same kind of relationships between their respective inputs. In such cases, fewer parameters can be required to avoid overfitting, etc.

[0015] By means of the various measures discussed herein, a neural network can be provided that improves on conventional deep learning and / or existing logical learning methods in several respects. The neural network can be enabled to reliably learn various tasks without any prior knowledge, e.g., improved image classification. By wrapping the powerful and versatile primitive of semidefinite programming, and in particular MAXSAT relaxations, our solver can enable implicit logical reasoning to take place where it is needed within a larger framework, without needing to know the exact structure of the domain.

[0016] For example, in various embodiments, a differentiable smooth MAXSAT solver is provided that can be embedded within more complex deep architectures, allowing efficient end-to-end learning of complex logical relationships from examples without any hard-coding of these relationships.

[0017] As discovered by the inventors, using the techniques presented herein, various kinds of problems can be solved that are essentially impossible to reliably learn without prior knowledge for traditional deep learning / logical learning methods. For example, in a fully end-to-end setting, a classical convolutional network can be integrated with a differentiable MAXSAT solver to learn constraint satisfaction problems that are challenging for modern neural network architectures.

[0018] The techniques described herein can be applied to various kinds of data, such as sensor data, audio data, video data, radar data, LiDAR data, ultrasound data, or various individual sensor readings or their histories. For example, in embodiments, a neural network is used for image captioning or image classification. In these and other applications, the network can include one or more convolutional layers, followed by one or more optimization layers, and one or more output layers, such as fully connected layers or sentence generation layers, e.g., layers of an LTSM network, recurrent neural network, etc., as known in the art.

[0019] As another example, a neural network according to embodiments can be used to recognize actions in a video, e.g., a sequence of images. For example, in these and other embodiments, the neural network can include one or more convolutional layers, followed by one or more optimization layers, followed by at least one fully connected layer.

[0020] In various embodiments, a neural network as described herein can be capable of performing classification, regression, etc., based on relatively complex relationships, e.g., hard constraints, in the data. Traditional neural networks can not be able to efficiently capture such constraints. For example, many layers can be needed to express the relationship, and thus, more parameters and / or training data can be needed, and / or the neural network can be more prone to overfitting. Thus, a neural network according to embodiments can need fewer trainable parameters to express logical relationships in the data, or can better express such logical relationships despite having the same or fewer trainable parameters. At the same time, it can not be necessary to pre-constrain the kinds of relationships that the optimization layers can learn, e.g., it can not be necessary to explicitly define between which values the optimization layers learn relationships.

[0021] As a particular example, consider a system that analyzes traffic conditions, e.g., as part of an autonomous vehicle, as part of a safety system for a non-autonomous vehicle, etc. Such a system can classify traffic conditions as dangerous or non-dangerous, for example. A root cause for a traffic condition being dangerous can be cars approaching each other. However, in many training instances, cars approaching each other can be because it is raining, because it is rush hour on the road. Especially if the training data set is small and / or if the two such factors frequently occur together, a conventional neural network can fail to learn which factor is the root cause and which factor is merely a correlation. In fact, a conventional neural network can use many layers and / or neurons to learn such a logical constraint, or the neural network can fail to effectively capture such a logical constraint at all. As a result, there can be no reward for a conventional neural network to learn such an inference. Instead of such a neural network, making an inference, e.g., about a traffic condition being dangerous, both when its root cause is satisfied and when a correlated factor is present, can be more efficient. The result can be a network that generalizes better beyond the training data.

[0022] However, using an optimization layer as described herein can allow more efficient learning of such logical relationships between data, e.g., one or a few clauses can be sufficient to learn that approaching cars are a relevant factor and that rain is merely a correlation. For example, a small number of parameters of the optimization layer can allow efficient capture of such logical relationships. As a result, a network can be obtained that generalizes better beyond the training data, and / or that requires fewer training instances and / or parameters to obtain a given performance.

[0023] Interestingly, various neural networks as defined herein can not require an existing known set of relationships or rules. For example, there can be no need for seeding with prior structure about which variables are allowed to interact with each other. In fact, in various embodiments, for a broad class of problems, the entire structure and its parameters can be learned end-to-end.

[0024] Optionally, the input instance comprises an image, and the neural network comprises at least one convolutional layer followed by one or more optimization layers. This way, a particularly good image classifier can be obtained.

[0025] Optionally, the semi-definite programming comprises a relaxation of a MAXSAT problem, a MAX2SAT problem, or a MAXCUT problem about values of the at least one or more input neurons and the one or more output neurons. MAXCUT, MAX2SAT, and MAXSAT allow particularly efficient representation of logical constraints, and solving of vectorized relaxations of them can be relatively efficient.

[0026] Optionally, the output vector is computed by optimizing one or more auxiliary vectors jointly with the output vector. The auxiliary vectors are typically not used elsewhere in the neural network. By introducing auxiliary vectors, the size of the semidefinite program can be reduced. For example, such auxiliary vectors can effectively play a role similar to register memory, e.g. in the clauses defining the MAXSAT problem, thereby allowing for better inference.

[0027] Optionally, the output vector can be computed by minimizing a linear combination of inner products of the input vector, the output vector, and optionally the auxiliary vectors . For example, the matrix V may comprise each of the input vector, the output vector, and if used, each of the auxiliary vectors. The matrix V may additionally comprise "true directions", e.g. fixed vectors initialized randomly, e.g. as further detailed elsewhere. Typically, the vectors are constrained, e.g. each of them can be constrained to have unit norm. In this way, a semidefinite program can be obtained that can be solved efficiently.

[0028] Optionally, the coefficients of the linear combination of inner products of two vectors are given by the inner product of two weight vectors associated with the respective vectors , which are included in the jointly optimized parameters. For example, the weight vectors can define a matrix S, where each row represents a clause, and each column represents the coefficients of the clause for the respective input, output and auxiliary variables. The optimization problem can be formulated as minimizing V over the output and auxiliary vectors of the matrix , e.g. the problem is constrained to V the vectors having unit norm. The values of the weight vectors can be seen as the weights of the optimization layer. For example, a MAXSAT relaxation can be obtained this way.

[0029] Optionally, the two weight vectors have less elements than the total number of input vectors, output vectors and optional auxiliary vectors, e.g. at most half. In this way, the knowledge learned by the neural network, e.g. in the form of clauses, can be compressed efficiently. In fact, having less elements can help to avoid overfitting, and thus allow the network to capture / focus on the most important logical relationships of the data.

[0030] ​Optionally, the output vector is computed by repeatedly updating the output vector or the auxiliary vector based on the current values of the output vector and the auxiliary vector and the input vector. In other words, the output vector is computed by so-called coordinate descent. Coordinate descent not only has good convergence properties, it is also particularly well suited for learning by backpropagation due to its ordered nature and can thus be particularly well integrated in a neural network learning framework. Coordinate descent also has the advantage that it can be efficiently parallelized and / or implemented on GPUs.

[0031] Optionally, the jointly optimized parameters comprise weight vectors associated with respective input vectors, output vectors and optionally auxiliary vectors , respective entries of the weight vectors define a linear combination of the vectors representing the clauses to be jointly optimized The output vector can be computed by:

[0032] - initializing a state matrix, e.g., where and , columns of the state matrix indicate values of clauses with respect to input vectors, output vectors and optionally auxiliary vectors; and

[0033] - repeatedly updating the output vector or the auxiliary vector based on the current values of the output vector and the auxiliary vector and the input vector, which comprises updating the state matrix based on the updated output vector or the auxiliary vector.

[0034] By keeping track of the state matrix, e.g., and using it in the coordinate descent, a relatively efficient update can be obtained, e.g., an update of can be provided, where n is the number of variables, k is the size of the vectors, and m is the number of clauses (e.g., of a MAXSAT problem) as discussed above.

[0035] Optionally, multiple updates of the output vector and / or the auxiliary vector of the coordinate descent can be performed in parallel. For example, by using coordinate descent, the training and / or application of neural networks can be particularly well suited for GPU implementations. During the inner loop of the coordinate descent in applying a neural network, all computation of gradient terms can be parallelized by parallelizing the computation of the state matrix and / or the rank 1 update to the state matrix and similarly for training. Thus, using (e.g., by a matrix multiplication ​Low-rank semidefinite programming (as defined) can be particularly beneficial, e.g. traditional full-rank coordinate gradient descent can not generally allow such parallelization. This can lead to significant performance improvements.

[0036] Training a neural network can be performed efficiently by backpropagation through the computations that determine the output values from the input values. Interestingly, with the measures discussed herein, such updates can be performed efficiently, e.g. the forward pass can be continuous and / or differentiable such that the backward pass can be derived analytically and / or computed efficiently. In particular when the joint optimization comprises repeatedly updating the output vector or the auxiliary vector based on the current values of the output vector and the auxiliary vector and the input vector, during training, the loss gradient with respect to the input vector and / or one or more weights can be determined by backpropagation through said repeated updates, thereby allowing to obtain optimized parameters in a way that integrates well with neural network training techniques for training other layers of the neural network, e.g. the entire neural network can be trained in an end-to-end fashion.

[0037] It will be appreciated to the skilled person that two or more of the above-mentioned embodiments, implementations and / or optional aspects of the application can be combined in any way deemed useful.

[0038] Any modifications and variations of the computer-implemented method and / or any computer-readable medium, which correspond to the described modifications and variations of the corresponding system, can be performed by the skilled person on the basis of the present description. BRIEF DESCRIPTION OF DRAWINGS

[0039] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter, by way of example, and with reference to the accompanying drawings, in which

[0040] Figure 1 A system for training a neural network is shown;

[0041] Figure 2 A system for applying a neural network is shown;

[0042] Figure 3 A system for applying a neural network is shown;

[0043] Figure 4 An optimization layer is shown;

[0044] Figure 5 A detailed example is shown of how to train / use an optimization layer;

[0045] Figure 6 A method of applying a neural network is shown;

[0046] Figure 7A method of training a neural network is shown.

[0047] Figure 8 A computer-readable medium including data is shown.

[0048] It should be noted that the various figures are purely schematic and not drawn to scale. In the various figures, elements corresponding to elements already described can have the same reference numerals. DETAILED DESCRIPTION

[0049] Figure 1 A system 100 for training a neural network is shown. The system 100 can include a data interface 120 and a processor subsystem 140, which can internally communicate via data communication 124. The data interface 120 can be used to access data 050 representing parameters of a neural network. The neural network can include at least one optimization layer. The optimization layer can be used to determine values of one or more output neurons from values of one or more input neurons by joint optimization. The data 050 can include one or more parameters of the joint optimization. The data interface 120 can also be used to access a training dataset 030 including one or more training instances.

[0050] The processor subsystem 140 can be configured to access the data 030, 050 during operation of the system 100 and using the data interface 120. For example, as shown in Figure 1 the data interface 120 can provide access 122 to an external data storage 020, which can include the data 030, 050. Alternatively, the data 030, 050 can be accessed from an internal data storage that is part of the system 100. Alternatively, the data 030, 050 can be received from another entity via a network. In general, the data interface 120 can take various forms, such as a network interface to a local or wide area network (e.g., the Internet), a storage interface to an internal or external data storage, etc. The data storage 020 can take any known and suitable form.

[0051] The processor subsystem 140 can be configured to train the neural network during system operation. Training the neural network can include determining a neural network loss gradient with respect to one or more input neurons and / or one or more parameters of the joint optimization from a loss gradient with respect to one or more output neurons of the optimization layer. The gradient can be determined by:

[0052] - determining a loss gradient with respect to an output vector of the one or more output neurons from the gradient with respect to the output neurons; and

[0053] - determining a loss gradient with respect to the input vector of one or more input neurons from the gradient with respect to the output vector and determining therefrom a neural network loss gradient with respect to the input neurons, and / or determining a loss gradient with respect to one or more parameters from the gradient with respect to the output vector.

[0054] The loss can be a loss of applying the neural network to one or more training instances of the training data set. When applying the neural network to a training instance, the output vector of one or more output neurons can be computed from the input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semi-definite program defined by the one or more parameters. Backward pass can be performed accordingly.

[0055] As an optional component, the system 100 can comprise an image input interface (not shown) or any other type of input interface for obtaining sensor data from a sensor such as a camera (not shown). The processor subsystem 140 can be configured to obtain one or more training instances of the training data set 030 as sensor data obtained from the sensor.

[0056] Various details and aspects of the operation of the system 100, including optional aspects thereof, will be further elucidated with reference to the following description and accompanying drawings, wherein Figure 4 and Figure 5 are further illustrated.

[0057] Generally, the system 100 can be embodied as or in a single device or apparatus such as a workstation (e.g. laptop or desktop based) or a server. The device or apparatus can comprise one or more microprocessors executing appropriate software. For example, the processor subsystem can be embodied by a single central processing unit (CPU), but also by a combination or system of such CPUs and / or other types of processing units. The software can have been downloaded and / or stored in a corresponding memory, e.g. a volatile memory such as a RAM or a non-volatile memory such as a flash memory. Alternatively, functional units of the system, e.g. the data interface and the processor subsystem, can be implemented in the device or apparatus in the form of programmable logic, e.g. as a field-programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system can be implemented in the form of circuitry. Note that the system 100 can also be implemented in a distributed manner, e.g. involving different devices or apparatuses such as a distributed server, e.g. in the form of cloud computing.

[0058] Figure 2A system 200 for applying a neural network to an input instance is shown. The system 200 can include a data interface 220 and a processor subsystem 240, which can internally communicate via data communication 224. The data interface 220 can be used to access data 050 representing a neural network. The neural network can include at least one optimization layer. The optimization layer can be used to determine values of one or more output neurons from values of one or more input neurons by joint optimization. The data 050 can include one or more parameters of the joint optimization.

[0059] The processor subsystem 240 can be configured to access the data 050 during operation of the system 200 and using the data interface 220. The data 050 can be obtained by training a neural network according to the methods described herein (e.g., by the system 100). For example, as shown in FIG. 1, the data interface 220 can provide access 222 to an external data storage 022, which can include the data 050. Alternatively, the data 050 can be accessed from an internal data storage that is part of the system 200. Alternatively, the data 050 can be received from a further entity via a network. In general, the data interface 220 can take various forms, such as a network interface to a local or wide area network (e.g., the Internet), a storage interface to an internal or external data storage, etc. The data storage 022 can take any known and suitable form. Figure 2

[0060] The processor subsystem 240 can be configured to obtain an input instance during operation of the system. The processor subsystem 240 can be further configured to apply the neural network 050 to the input instance during operation of the system. Applying the neural network can include obtaining values of one or more input neurons to the optimization layer and determining an input vector of the one or more input neurons therefrom. Applying the neural network can further include computing an output vector of one or more output neurons from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semi-definite program defined by the one or more parameters. Applying the neural network can also include determining values of the one or more output neurons from the respective computed output vector.

[0061] As an optional component, the system 200 can include an image input interface 260 or any other type of input interface for obtaining sensor data from a sensor, such as a camera 280. The sensor data can be included in the instance to be processed by the neural network. For example, the camera can be configured to capture image data 262, and the processor subsystem 240 is configured to obtain the instance from the image data 262 obtained via the input interface 260.

[0062] ​As an optional component, the system 200 can comprise an actuator interface (not shown) for providing actuator data to an actuator, causing the actuator to implement an action in an environment of the system 200. For example, the processor subsystem 240 can be configured to determine actuator data based at least in part on an output of the neural network for an input instance, and provide the actuator data to the actuator via the actuator interface.

[0063] Various details and aspects of the operation of the system 200, including optional aspects thereof, will be further elucidated with reference to the following examples. Figure 4 and Figure 5 are further elucidated.

[0064] Generally, the system 200 can be embodied as or in a single device or apparatus, such as a workstation (e.g. laptop or desktop based) or a server. The device or apparatus can comprise one or more microprocessors executing appropriate software. For example, the processor subsystem can be embodied by a single central processing unit (CPU), but also by a combination or system of such CPUs and / or other types of processing units. The software can have been downloaded and / or stored in corresponding memory, e.g. a volatile memory such as a RAM or a non-volatile memory such as a flash memory. Alternatively, functional units of the system, e.g. the data interface and the processor subsystem, can be implemented in the device or apparatus in the form of programmable logic, e.g. as a field programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system can be implemented in the form of circuitry. Note that the system 200 can also be implemented in a distributed manner, e.g. involving different devices or apparatuses, such as a distributed server, e.g. in the form of cloud computing.

[0065] Figure 3 An example of the above is shown, in which a car control system 300 for controlling a vehicle 62 is shown, which comprises a system for applying a neural network to input instances according to an embodiment, e.g. the system 200 (not shown separately) of Figure 2 . In this example, the neural network can be an image classifier. The vehicle 62 can be an autonomous or semi-autonomous vehicle, but this is not essential, e.g. the system 300 can also be a driver assistance system for a non-autonomous vehicle 62.

[0066] For example, the vehicle 62 can incorporate the neural network application system to control the vehicle based on images obtained from a camera 22. For example, the car control system 300 can comprise a camera interface (not shown separately) for obtaining images of the vehicle environment 50 from the camera 22.

[0067] The system can be configured to use a neural network to classify images obtained from the camera 22 to detect a condition of interest in the vehicle environment 50, such as an obstacle with which the vehicle is at risk of colliding. The control system 300 can further comprise an actuator interface (not shown separately) for providing actuator data to actuators to cause the actuators to implement actions to control the vehicle 62. The automotive control system 300 can be configured to determine actuator data to control the vehicle 62 based at least in part on the detection; and provide the actuator data to the actuators via the actuator interface. For example, the actuators can be caused to control steering and / or braking of the vehicle. For example, the control system can control the electric motor 42 to perform (regenerative) braking upon detecting a condition of interest.

[0068] Figure 4 A detailed but non-limiting example of an optimization layer, for example for use by the system 100 or 200, is shown. The optimization layer OL 451 shown here is a MAXSAT layer, in which a semidefinite program is solved in the joint optimization Opt 440, which is a relaxation of the MAXSAT problem. The figure shows a forward pass of the MAXSAT layer, for example from input to output. The layer can take as input discrete or probabilistic assignments of known MAXSAT variables. The layer can output guesses for assignments of unknown variables via an SDP relaxation of the MAXSAT problem with certain weights S Such a layer can be referred to as a "SATNet layer".

[0069] Shown is an input value IV 400 to the optimization layer OL. For the input value can be denoted where denotes a set of inputs to the optimization layer. The input value can be discrete (for example ), or probabilistic (for example ).

[0070] As part of the optimization layer OL, in operation Rx 410 the input value IV is relaxed to obtain an input vector IVX 420, for example an input vector for each input value A suitable relaxation is discussed below. Preferably, the relaxation is performed by a continuous and / or differentiable function.

[0071] Based on the input vector IVX, an output vector OVX 470 is computed by jointly optimizing Opt 440 at least the output vector OVX with respect to the input vector IVX, for example by optimizing a loss function with respect to at least the output vector OVX while keeping the input vector IVX fixed. For the output vector OVX can be denoted where A set of outputs of the optimization layer. In this case, the optimization Opt is an SDP relaxation of MAXSAT; the parameters PAR 460 of the SDP relaxation can include weights that define the SDP relaxation S A detailed example of such a relaxation is discussed elsewhere.

[0072] In this case, by performing a rounding Rd 480, the output values OV 490 of the optimization layer OL can be determined from the optimized output vector OVX. In this example, the output values may be discrete (e.g., ), or probabilistic (e.g. The rounding Rd can be performed by a random rounding or other type of rounding explained elsewhere.

[0073] As a detailed example of an optimization layer OL, the following description of a SATNet layer is given with reference to the formulas and algorithms referenced in Figure 5 discussed in more detail:

[0074]

[0075] .

[0076] Figure 5 A detailed, but non-limiting example of how a neural network can be applied and / or trained according to embodiments is shown. In this example, a neural network NN 550 is shown.

[0077] For example, when applying the neural network NN, an input instance II 530 can be obtained. For example, the input instance II can be an image or other type of sensor data. The input instance II can be input by a user from a sensor or the like. In various embodiments, the neural network NN can be configured to determine a classification CL 539 of the input instance, although this is not required. The classification CL can be used, for example, to determine a control signal for an actuator to implement an action in an environment from which the input instance II was sensed. The arrows shown in the figure can indicate various data dependencies in a forward pass of applying the neural network.

[0078] For example, when training a neural network NN, backpropagation through the neural network can be performed, e.g., by iteratively determining loss gradients of the neural network and adapting the neural network parameters to reduce the loss. For example, based on one or more input instances II with corresponding labels (e.g., in a mini-batch), a loss can be determined that indicates a difference between a classification of the neural network NN for the respective input instances II and the provided labels of the respective instances. For example, to train the neural network NN, an optimization framework known per se in the art can be used, such as the Adam optimizer disclosed in Diederik P. Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization,” (incorporated herein by reference and available at https: / / arxiv.org / abs / 1412.6980 For example, the loss to be minimized can be a cross-entropy loss, a negative log-likelihood, a mean squared error, etc.

[0079] As shown in the figure, the neural network NN can comprise at least one optimization layer. The neural network NN can also comprise multiple optimization layers, e.g., where one or more outputs of a first optimization layer are input to a second optimization layer, etc.

[0080] For example, to learn to apply a sequence of operations to an input sequence of length L , the neural network NN can comprise a sequence of L -1 optimization layers (e.g., SATNet layers). For example, the sequence of operations can be a sequence of XOR (exclusive OR) operations that are applied to the input sequence successively to compute its parity. For example, in case the same operation is applied multiple times, some or all of the optimization layers can have associated weights. For example, in case of a binary function such as XOR, a first optimization layer can receive the first two values as input, and a layer d may receive the received values d and the output of layer d- 1. If each layer learns to compute the respective operation, the combined system will correctly compute the overall operation. Note that by using the techniques presented herein, such neural networks with long sequences of optimization problems can be trained with any intermediate supervision.

[0081] As another example of a neural network NN structure, the neural network can comprise one or more convolutional neural network layers followed by one or more optimization layers (e.g., SATNet layers). In this way, the neural network can be capable of making inferences from low-level sensor input features (e.g., low-level image features). The optimization layers can be followed by, e.g., additional layers to perform classification, regression, encoding into latent features, etc.

[0082] As the skilled person will appreciate, the above specific neural network examples are merely for illustration of several specific uses of the optimization layer as presented herein. Various alternatives or combinations can be envisaged, wherein the optimization layer as presented herein can be beneficially combined with other types of neural network layers (e.g. layers known from LSTM architectures) and / or in various settings (e.g. in generative adversarial models (GANs), variational autoencoders (VAEs), etc.).

[0083] Now proceeding to the optimization layer. The optimization layer can be used to determine values of one or more output neurons , until In this example, two output neurons are shown, but in general there can be more, e.g. at least 20 or at least 100 output neurons. However, a single output neuron is also possible. The values of the output neurons can be determined from values of one or more input neurons Again, there can be one or more, e.g. at least 20 or at least 100 input values. The number of input values and the number of output values can be different. The output values may be determined from the input values by jointly optimizing at least an output vector corresponding to the output values with respect to an input vector corresponding to the input values, e.g. by determining the output vector that minimizes a loss function that depends on the output vector and the input vector, while keeping the input values constant.

[0084] In various embodiments, the joint optimization can be a relaxation of an optimization problem with respect to the input values and the output values. In particular, in some embodiments, the joint optimization is a MAXSAT relaxation. Other optimization problems such as MAX2SAT or MAXCUT can be similarly relaxed.

[0085] MAXSAT problems are well-known optimization analogs of the satisfiability (SAT) problem, where the goal is to maximize the number of clauses that are satisfied. Consider a MAXSAT instance with n variables and m clauses. Let denote a binary assignment of the problem variables, where is the truth value of variable and define for , where denotes the sign of in clause . The MAXSAT problem can be formulated as:

[0086] (1)

[0087] To form the semidefinite relaxation of (1), the discrete variables can be relaxed into associated continuous variables with respect to some "true direction" . The coefficient vector can be defined as associated with . The SDP relaxation of MAXSAT can be formalized as:

[0088] (2)

[0089] where and .

[0090] Although in general, for various choices of the matrix M leading to a semidefinite program, various linear combinations of the inner products of the vectors in V (·) can be minimized , the choice is particularly advantageous for its correspondence to the MAXSAT problem. In other words, the coefficients of the linear combination of the inner products of two vectors can be given by the inner product of two weight vectors associated with the respective vectors . The weight vectors can be included in the parameters of the joint optimization. The weight vectors can have fewer elements n +1 than the total number of input vectors, output vectors, and optional auxiliary vectors k , in other words, S can be less than V and in this sense, S can provide a low-rank parameterization of the optimization layer.

[0091] In various embodiments, the semidefinite program (2) can be solved by applying coordinate descent. In particular, the objective term depending on can be given by where is the i column vector of S. Restricting the constraints to minimizes this quantity over yields the coordinate descent update:

[0092] where (3)

[0093] These updates can converge to a global optimal fixed point of the SDP.

[0094] In various embodiments, a differentiable, smooth solver is used to solve semidefinite problems, which can be integrated into modern deep network architectures such as neural networks (NNs). Coordinate descent can be used to solve semidefinite programming problems, such as SDP-MAXSAT relations. In particular, deep network layers for satisfyability solutions can be obtained by using SDP relaxation and associated coordinate descent updates for MAXSAT problems, such as... Figure 4 The SATNet layer.

[0095] Now we continue to how a neural network (NN) can be applied to input instance II during the forward pass. Definition Input variables for the optimization layer The index (e.g., a variable with a known assignment), and let With output variables that have unknown assignment values The indices correspond to each other. For example, by evaluating the layers of a neural network NN before the optimization layer, the input value can be determined from input instance II.

[0096] The optimization layer can accept probabilistic or binary inputs. , As input IVi, and can output the assignment of unknown variables. As the output OVi, this assignment is similarly probabilistic or binary. or We respectively ordered and Refers to all input and output assignments. Output It can be generated via SDP. For example, the parameters of the joint optimization can include the low-rank coefficient matrix of SDP.

[0097] Layer initialization: Various aspects of the optimization layer can be configured, such as multiple parameters for joint optimization. For example, in the MAXSAT example, the maximum number of clauses that the layer can represent can be predefined. m .

[0098] Alternatively, the representational power of a layer can be enhanced by using one or more auxiliary variables that are not connected to its inputs or outputs. One or more auxiliary vectors can be jointly optimized with the output vector. To compute the output vector. Adding additional variables to the optimization problem can reduce the number of clauses required to parameterize it. Like input and output variables, auxiliary variables can be represented as auxiliary vectors and thus jointly optimized along with the output vector. The number of auxiliary variables is generally different from the number of input or output vectors. There can be one or more auxiliary vectors, for example, at least 10 or at least 100.

[0099] The input vector, the output vector and / or the auxiliary vector can have a dimension where n may denote the number of actual problem variables (inputs and / or outputs) other than auxiliary variables, e.g. for MAXSAT, this can guarantee that the SDP can be solved optimally.

[0100] In operation Rx 510, the layer input may be relaxed, e.g. the input vector may be determined for the respective input value. Preferably, the input vector is determined for each input value in a continuous and / or differential manner to facilitate backpropagation. As a specific example, for each layer input may be relaxed to the associated via

[0101] (4)

[0102] where the "true direction" may be initialized (e.g. randomly generated) and stored within the layer. Other functions are possible. In fact, in operation Rx, the optimization layer can relax its input to a continuous vector for use in the SDP formulation.

[0103] In operation Cdesc 540, at least the output vector can be computed from the determined input vector by jointly optimizing with respect to the input vector a semi-definite program defined by one or more parameters. The output vector of the one or more output neurons . If there are auxiliary vectors , these can also be optimized. The number of output vectors typically corresponds to the number of output values .

[0104] For example, the one or more parameters can comprise a weight vector S for a MAXSAT relaxation. In this case, for each input vector, there can be one weight vector ; for each auxiliary vector, there can be one weight vector , and / or for each output vector, there can be one weight vector . Typically, the weight vectors each have the same length, thereby together forming a matrix S as described above. More generally, the parameters can comprise or define coefficients , , of a linear combination of inner products of vectors M .

[0105] Interestingly, the output and / or auxiliary vectors are typically determined through optimization, while the input vector remains fixed, as indicated by the arrows in the figure.

[0106] Given input relaxation (We collectively refer to) ,For example V The index is (a subset of the columns), the optimization layer can use coordinate descent, for example by using update (3) to compute continuous output. The value (which we collectively refer to) In other words, the output vector can be calculated by repeatedly updating the output vector OVXi or the auxiliary vector AVXi based on the current values ​​of the output vector OVXi and the auxiliary vector AVXi, as well as the input vector IVXi. Interestingly, these updates can be computed only for output and auxiliary variables; for example, computation can be avoided for variables whose assignments are passed as input to the layer.

[0107] The following provides a detailed example coordinate descent algorithm for the forward pass of MAXSAT relaxation:

[0108] .

[0109] As illustrated in the algorithm above, coordinate descent can be efficiently achieved by maintaining a state matrix, which is calculated... Required items As discussed, the weight vector The corresponding entries—in other words, the matrix S Rows - can define vectors A linear combination of these terms, representing the clauses to be jointly optimized. Thus, the columns of the state matrix... Such clauses can indicate the values ​​of the input vector, output vector, and optional auxiliary vectors. As illustrated, the state matrix can be updated via a rank-one update during each internal iteration. Therefore, it can be implemented as follows: O ( nmk The runtime of each iteration of ) is . It has been observed that in practice, only a small number of iterations are often required for convergence.

[0110] In operation Rd 580, the discrete or probability values ​​of one or more output neurons From the corresponding calculated output vector Determined. Given an output vector, such as the relaxation output from coordinate descent. , the optimization layer can convert these outputs into discrete or probabilistic variable assignments, e.g., via thresholding or stochastic rounding Interestingly, the operation Rdmay be continuous and / or differentiable, or at least allow training as a continuous and / or differentiable function.

[0111] For example, stochastic rounding can be used, where a random hyperplane can be generated, e.g., from a unit sphere r and the assignment can be made:

[0112] (5)

[0113] where is a Boolean output for By using this method or a similar method, only when is similar to (e.g., when is on the same side of the random hyperplane r as the true vector , can be "true".

[0114] During training, instead of or in addition to stochastic rounding, it can be noted that and are on the same side of any given r with probability:

[0115] (6)

[0116] Thus, can be set equal to this probability.

[0117] When applying the neural network NN, for example, a probabilistic output can be output in the same way, or a discrete assignment can be output via thresholding or by explicitly applying stochastic rounding. In the case of stochastic rounding, the rounding can be performed multiple times, and the Boolean solution that maximizes the optimization problem at hand (e.g., the MAXSAT objective (1)) can be picked. For example, can be set.

[0118] Continuing now to training the neural network NN in a backward pass. When training the neural network NN, a backward pass can be performed to determine the neural network loss gradients with respect to one or more input neurons and / or one or more parameters of a joint optimization from loss gradients with respect to one or more output neurons of the optimization layer. Given the gradient of the network loss with respect to the layer output , the loss gradients with respect to the respective layer inputs and weights and The loss can be cross-entropy loss, negative log-likelihood, mean squared error, etc.

[0119] Interestingly, in some cases, the computation can be based on an analytical expression to directly compute the gradient of the expectation, enabling an efficient coordinate descent algorithm. This is exemplified for the SDP relaxation of the MAXSAT problem below. However, explicit analytical expressions are not required, e.g., numerical approximations can be used.

[0120] From the probabilistic outputs to their continuous relaxations. Given , one can use the expression for obtained by pushing the gradient through the probabilistic assignment mechanism, i.e., for each ,

[0121] (7)

[0122] where we obtain by differentiating through equation (6).

[0123] Backpropagation through SDP. Given the result , we next seek to compute and by pushing the gradient through the SDP solver. The solution to the SDP problem can be differentiated with respect to the parameters and inputs of the problem by means of matrix-vector multiplication. To compute the solution to this linear system, one can use a coordinate descent method that closely mirrors the computation in the forward pass and has similar fast convergence properties. As an example, assume that we are given the loss gradient with respect to all continuous SDP output vectors . It is possible to apply analytical expressions for the loss gradient with respect to some weight matrix parameter to compute an explicit form for

[0124] .

[0125] We start by deriving the intermediate term ; since our coordinate descent update (3) can converge to a fixed point, we can find this gradient by rearranging the final update as:

[0126] (8)

[0127] and differentiate through this equation via the implicit function theorem as:

[0128] , for all (9)

[0129] where and . (Here, refers to S the subset of columns of indexed by . Note that for all , the system in (9) can be linear.

[0130] Define to capture the right side of equation (9). Solving for this linear system and applying the chain rule, we see that:

[0131] (10)

[0132] where and are indexed by and , and where . For a fixed , this derivation can be analogous for all ; in fact, typically only the term has an explicit dependence on j . Thus, for a fixed , we can easily compute the loss gradient with respect to all by precomputing the matrix-vector products and . This matrix-vector solution can be solved via block coordinate descent, e.g., as in Algorithm 3 shown below.

[0133] The analytic form of the gradient can be given by:

[0134] (11)

[0135] (12)

[0136] where is the column of i , and denotes the subset of columns of S indexed by . Equation (12) can be the analytic form of the weight gradient used by our layer.

[0137] From the relaxed input to the original input. The gradient (with respect to the input relaxation) can be used to derive the gradient (about actual inputs). For each It can be the case that:

[0138] (13)

[0139] where capture any direct dependencies on (as opposed to dependencies through ). Equation (13) can represent the analytical form of the input gradient used by our layer.

[0140] Backpropagation terms are computed via coordinate descent. Linear systems can be solved via coordinate descent. Interestingly, the technique illustrated in Algorithm 3 below can enable the computation of the desired gradient without the need to explicitly maintain the intermediate Jacobian matrix. By mirroring the forward pass, rank-one updates can be used to maintain and modify the terms needed to compute . In fact, as the inventors have realized, by computing the Jacobian vector product needed for backpropagation using, for example, a coordinate descent method, one can avoid computing the Jacobian matrix itself. The Jacobian matrix can be large, for example, the V matrix used in the SDP relaxation of a MAXSAT problem solution can be , and thus the full Jacobian matrix for this term can be a matrix. Thus, storage and / or computation can be significantly reduced.

[0141] An example procedure for computing the gradient of an SDP relaxation of a MAXSAT problem is detailed below:

[0142] .

[0143] Figure 6 A block diagram of a computer-implemented method 600 of applying a neural network to an input instance is shown.

[0144] The method 600 can comprise, in an operation titled "Accessing data representing a neural network", accessing 610 data representing a neural network. The neural network can comprise at least one optimization layer. The optimization layer can be configured to determine values of one or more output neurons from values of one or more input neurons by joint optimization. The data can comprise one or more parameters of the joint optimization. The method can further comprise, in an operation titled "Obtaining an input instance", obtaining 620 an input instance. The method can further comprise, in an operation titled "Applying the neural network", applying 630 the neural network to the input instance. Applying 630 the neural network can comprise, in an operation titled "Obtaining input values", obtaining 631 values to one or more input neurons of the optimization layer. Applying 630 can further comprise, in an operation titled "Determining an input vector", determining 632 an input vector for the one or more input neurons from the values of the one or more input neurons. Applying 630 can further comprise, in an operation titled "Solving a semidefinite program", computing 633 an output vector for the one or more output neurons from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters. Applying 630 can further comprise, in an operation titled "Determining output values", determining 634 values for the one or more output neurons from the respective computed output vector.

[0145] The method 600 can correspond to the operation of the system 200 of Figure 2 However, this is not a limitation, as the method 600 can also be performed using another system, apparatus, or device.

[0146] Figure 7 A block diagram of a computer-implemented method 700 of training a neural network is shown.

[0147] The method 700 can comprise, in an operation titled "Accessing data representing a neural network, training data", accessing 710 data representing parameters of a neural network and accessing a training data set comprising one or more input instances. The neural network can comprise at least one optimization layer. The optimization layer can be configured to determine values of one or more output neurons from values of one or more input neurons by joint optimization. The data can comprise one or more parameters of the joint optimization.

[0148] The method 700 can further comprise, in an operation titled "Training the neural network", training 720 the neural network. Training 720 can comprise, in an operation titled "Determining a loss gradient", determining 730 a neural network loss gradient with respect to one or more input neurons and / or one or more parameters of the joint optimization from a loss gradient with respect to one or more output neurons of the optimization layer.

[0149] Determining 730 the gradient can include, in an operation titled "Determining a gradient with respect to the output vector," determining 731 a loss gradient with respect to the output vector of the one or more output neurons from the gradient with respect to the output neuron.

[0150] Determining 730 can further include, in an operation titled "Determining a gradient with respect to the input vector," determining 732 a loss gradient with respect to the input vector of the one or more input neurons from the gradient with respect to the output vector. Determining 730 can further include, in an operation titled "Determining a gradient with respect to the input neuron," determining 733 a neural network loss gradient with respect to the input neuron from the loss gradient with respect to the input vector.

[0151] Instead of, or in addition to, operations 732 and 733, determining 730 can include, in an operation titled "Determining a gradient with respect to the parameter," determining 734 a loss gradient with respect to the one or more parameters from the gradient with respect to the output vector.

[0152] The output vector of the one or more output neurons can be computed from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semi-definite program defined by the one or more parameters.

[0153] Method 700 can correspond to the operations of system 100 of Figure 1 However, this is not a limitation, as method 700 can also be performed using another system, apparatus, or device.

[0154] It should be appreciated that, in general, the operations of methods 600 and 700 can be performed in any suitable order (e.g., consecutively, simultaneously, or a combination thereof), subject to a particular order being necessary (e.g., by input / output relationships), if applicable.

[0155] The method(s) can be implemented on a computer as a computer- implemented method, special purpose hardware, or a combination of both. Also as Figure 8 As also illustrated in FIG. 8, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 800, e.g., in the form of a series 810 of machine-readable physical marks and / or as a series of elements having different electrical (e.g., magnetic or optical) properties or values. The executable code can be stored in a transitory or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 8 An optical disc 800 is shown. Alternatively, the computer-readable medium 800 can include transitory or non-transitory data 810 representing parameters of a joint optimization as described elsewhere in this specification.

[0156] While the neural networks as presented herein are applicable to a wide range of problems, several specific examples are provided below.

[0157] Action recognition from video: Suppose we are given a video sequence as input and the goal is to recognize some action that occurred in the scene. For example, in autonomous driving systems, the system makes decisions by utilizing visual data. In many such cases, the rules for making decisions cannot be explicitly or easily defined. However, by applying the techniques described herein, a neural network can be trained and / or used that can perform learned logical reasoning based on a set of visual information.

[0158] Image captioning: We can consider the problem of describing a scene in a digital image by a sentence. By utilizing a training set of pairs of images along with their short descriptions, the presented network can learn general rules that give a description and can better generalize to unseen images. For example, the neural network can include a LTSM or recurrent neural network after the optimization layer.

[0159] Recognition of handwritten text: Suppose we are given the text contained in a document, which is for example scanned or given as a digital image, and the goal is to recognize the text. Even for the case where the text has been printed, some noise or missing data should be taken into account. When we consider the diversity of handwritten text recognition, the problem becomes even more challenging and therefore the complexity of the problem grows dramatically. In this task, we should take into account even more imperfect observations, which are difficult to handle without a large amount of training data. In this setting, the presented neural network can learn how to recognize common mistakes made by humans. Therefore, a smaller number of training examples can be needed to train the network. Furthermore, as the network learns a more general model, the recognition task can be solved more accurately because the model can explore finer relationships within the data, so a better distinction can be made between some uncertain conditions.

[0160] Natural language recognition: Similar to text recognition, the techniques disclosed herein can also be applied to natural language recognition. Since we cannot expect perfect data (e.g. noiseless data), missing information can be modeled. This can be achieved by the neural networks as presented herein, learning more complex relationships in the training data using the optimization layer.

[0161] Examples, embodiments or optional features, whether or not they are indicated as being non-limiting, are not to be taken as limiting the claimed invention.

[0162] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The verb "comprise" and its derivates do not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The expression "at least one of A and B" when preceding the list of elements A and B, should be understood as used in the sense of "at least one of the elements A or at least one of the elements B". For example, the expression "at least one of A and B" should be understood as including solely A; solely B; A and B both; or both A and B. The application can be implemented by means of both hardware and software, and any combination thereof. In a device claim enumerating several means, those means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A system (200) for applying a neural network to an input instance, wherein the input instance includes an image, the system comprising: - Data interface (220) for accessing data (050) representing a neural network, the neural network including at least one optimization layer, wherein the optimization layer is used to determine the values ​​of one or more output neurons from the values ​​of one or more input neurons through joint optimization, the data including one or more parameters of the joint optimization; - The processor subsystem (240) is configured as follows: - Obtain an input instance; - Applying the neural network to the input instance to classify the image includes: - Obtain the values ​​of the one or more input neurons in the optimization layer, and determine the input vector of the one or more input neurons from them; - The output vector of the one or more output neurons is computed from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters; - Determine the values ​​of the one or more output neurons from the corresponding calculated output vector.

2. The system (200) of claim 1, wherein the neural network includes at least one convolutional layer, followed by an optimization layer.

3. The system (200) according to claim 1 or 2, wherein the semidefinite programming includes relaxation of the MAXSAT problem or MAX2SAT problem or MAXCUT problem with respect to the values ​​of at least one or more input neurons and one or more output neurons.

4. The system (200) according to claim 1 or 2, wherein, The processor subsystem (240) is configured to compute the output vector by jointly optimizing one or more auxiliary vectors with the output vector.

5. The system (200) according to claim 1 or 2, wherein the processor subsystem (240) is configured to minimize the inner product of the input vector, the output vector, and an optional auxiliary vector ( Linear combination of ) The output vector is calculated using .

6. The system (200) according to claim 5, wherein the coefficients of the linear combination of the inner product of two vectors are derived from the inner product of the two weight vectors associated with the respective vectors ( The two weight vectors are given and included in the parameters of the joint optimization.

7. The system (200) according to claim 6, wherein the two weight vectors ( , It has fewer elements than the total number of input vectors, output vectors, and optional auxiliary vectors.

8. The system (200) according to claim 1 or 2, wherein, The processor subsystem is configured to compute the output vector by repeatedly updating the output vector or auxiliary vector based on the current values ​​of the output vector and auxiliary vector, as well as the input vector.

9. The system (200) of claim 8, wherein the parameters for joint optimization include weight vectors associated with corresponding input vectors, output vectors, and optional auxiliary vectors, the corresponding entries of the weight vectors defining linear combinations of the vectors representing the clauses to be jointly optimized, and the processor subsystem (240) is configured to compute the output vector in the following manner: - Initialize the state matrix, whose columns indicate clause values ​​with respect to the input vector, output vector, and optional auxiliary vector; and - Repeatedly update the output vector or auxiliary vector based on the current values ​​of the output vector and auxiliary vector, as well as the input vector, including updating the state matrix based on the updated output vector or auxiliary vector.

10. A system (100) for training a neural network, the system comprising: - A data interface (120) for accessing data (050) representing parameters of a neural network, the neural network including at least one optimization layer, wherein the optimization layer is used to determine the values ​​of one or more output neurons from the values ​​of one or more input neurons through joint optimization, the data including one or more parameters of the joint optimization; and for accessing a training dataset (030) including one or more input instances, wherein the input instances include images; - A processor subsystem (140) is configured to train the neural network to classify images, wherein training the neural network includes determining a neural network loss gradient with respect to the one or more input neurons and / or the one or more parameters of the joint optimization from the loss gradient with respect to the one or more output neurons of the optimization layer in such a way as: - Determine the loss gradient with respect to the output vectors of the one or more output neurons from the gradients with respect to the output neurons; and - Determine the loss gradient with respect to the input vector of the one or more input neurons from the gradient with respect to the output vector, and determine the neural network loss gradient with respect to the input neurons from therein, and / or determine the loss gradient with respect to the one or more parameters from the gradient with respect to the output vector; The output vector of the one or more output neurons is calculated from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters.

11. The system (100) of claim 10, wherein joint optimization includes repeatedly updating the output vector or auxiliary vector based on the current values ​​of the output vector and auxiliary vector and the input vector, and wherein the processor subsystem is configured to determine a loss gradient with respect to the input vector and / or one or more weights via backpropagation through the repeated updates.

12. A computer-implemented method (600) for applying a neural network to an input instance, wherein the input instance includes an image, the method comprising: - Access data representing a neural network, the neural network including at least one optimization layer, wherein the optimization layer is used to determine the values ​​of one or more output neurons from the values ​​of one or more input neurons through joint optimization, the data including one or more parameters of the joint optimization; - Obtain an input instance; - Applying the neural network to input instances to classify images includes: - Obtain the values ​​of the one or more input neurons in the optimization layer, and determine the input vector of the one or more input neurons from them; - The output vector of the one or more output neurons is computed from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters; - Determine the values ​​of the one or more output neurons from the corresponding calculated output vector.

13. A computer-implemented method (700) for training a neural network, the method comprising: - Access data representing parameters of a neural network, the neural network including at least one optimization layer, wherein the optimization layer is used to determine the values ​​of one or more output neurons from the values ​​of one or more input neurons through joint optimization, the data including one or more parameters of the joint optimization; and a training dataset including one or more input instances, wherein the input instances include images; - Training the neural network to classify images, wherein training the neural network includes determining the neural network loss gradient with respect to the one or more input neurons and / or the one or more parameters of the joint optimization from the loss gradient with respect to the one or more output neurons of the optimization layer in such a way as: - Determine the loss gradient with respect to the output vectors of the one or more output neurons from the gradients with respect to the output neurons; and - Determine the loss gradient with respect to the input vector of the one or more input neurons from the gradient with respect to the output vector, and determine the neural network loss gradient with respect to the input neurons from therein, and / or determine the loss gradient with respect to the one or more parameters from the gradient with respect to the output vector; The output vector of the one or more output neurons is calculated from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters.

14. A computer-readable medium (800) comprising transient or non-transient data representing instructions that, when executed by a processor system, cause the processor system to perform the method according to claim 12 or claim 13.

15. A computer-readable medium (800) comprising transient or non-transient data representing parameters of a neural network, the neural network including at least one optimization layer, wherein the optimization layer is configured to determine values ​​of one or more output neurons from values ​​of one or more input neurons through joint optimization, the data including one or more parameters of the joint optimization, and wherein applying the neural network to an input instance comprises: - Obtain the values ​​of the one or more input neurons in the optimization layer, and determine the input vector of the one or more input neurons from them; - The output vector of the one or more output neurons is computed from the determined input vector by jointly optimizing at least the output vector with respect to the input vector to solve a semidefinite program defined by the one or more parameters; - Determine the values ​​of the one or more output neurons from the corresponding calculated output vector. The input instances include images, and a neural network is applied to classify the images.

Images