Control by scalable and efficient inference based on nonlinear tensor network

Abstract

To provide a device and a method for monitoring and controlling a machine, a system and a process having an advantage of a neural network (NN) and more scalable and complicated than the NN.SOLUTION: A method 10 includes: converting a predetermined machine learning routine having a form of an NN into a machine learning routine having a form of a tensor NN (TNN) (20); for execution of routine conversion from the NN into the TNN, converting one or more linear layers of a plurality of linear layers of a neural network into respective tensor networks, so that each linear layer to be converted becomes the tensor network (21); converting one or more nonlinearity applicable for the converted layer of the NN into the nonlinearity applicable for each tensor of each TNN obtained from the conversion 21 (22); and for generation 30 of at least one output 40 relating to a target machine, system or process by inference, inputting a data set to a converted TNN-based routine (31).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the field of machine learning for the control of machines, systems, and / or processes. In particular, the present invention relates to the provision and use of nonlinear tensor networks as machine learning routines or algorithms for inference and control of machines, systems, and / or processes. [Background technology]

[0002] Deep learning technology has grown significantly in popularity over the past few years. Its ability to solve a wide variety of problems is enormous. Many industries and applications are therefore benefiting from deep learning by incorporating such techniques into their computing devices. For example, applications such as predictive analytics, anomaly detection, image classification, and object recognition have seen significant improvements through the adoption of machine learning routines, especially deep learning routines.

[0003] Indeed, many machines, systems, processes, or targets may be partially or completely monitored and controlled by machine learning routines. Such routines enable the detection of target states whenever a new situation or configuration occurs that has not previously occurred for the target. Classical monitoring and control schemes, on the other hand, map several parameters and / or measurements about the target and use at least some combination of them to derive the corresponding target state based on their values falling within one range or another. This results in substandard monitoring and control, as information about the target and subsequent control commands are based on inaccurate state detection.

[0004] This problematic behavior of classical methods is exacerbated the more parameters and measurements that must be considered to derive the target state, making accurate monitoring and control impossible.

[0005] Deep learning is suitable for characterizing a target at any point in time. Target features, with or without feature value simplification, can be fed into a deep learning routine and processed to perform inferences about the target's state. Furthermore, such inferences can include control commands, or they can be derived after determining the target's status; in either case, the control commands can be tailored to a more accurate status of the target.

[0006] One of the main drawbacks of machine learning routines in the form of neural networks, one of the most widely used forms of machine learning, is that they are memory-greedy, requiring a lot of processing resources and memory allocation to store all of the data generated by the routine, especially intermediate feature vectors. As a result, computing devices and systems are unable to cope with the requirements of such neural networks as they become large and complex. As more complex targets are monitored and controlled, the complexity of the neural networks also limits their use or requires simplified versions to be viable. Summary of the Invention [Problem to be solved by the invention]

[0007] There is growing interest in finding machine learning routines that have the advantages of neural networks but are more scalable than neural networks, allowing for the monitoring and control of complex machines, systems and processes. [Means for solving the problem]

[0008] explanation A first aspect relates to an apparatus or system as claimed in claim 1 attached hereto, namely an apparatus or system including at least one classical processor and / or at least one quantum processor, which may be configured to perform at least: Converting a predetermined machine learning routine in the form of a neural network (also referred to in this disclosure as a NN) into a tensorized neural network (also referred to in this disclosure as a TNN), the predetermined machine learning routine being associated with a target machine or system or process; and Generating at least one output related to the target machine, system, or process, the at least one output being inferred by the transformed predetermined machine learning routine when the dataset is input thereto.

[0009] For the conversion of a routine from a NN to a TNN, the device or system is configured to perform at least the following: Converting one or more layers of the plurality of layers of the NN into one or more tensor networks, respectively; and Transforming one or more first nonlinearities (also referred to in this disclosure as global nonlinearities) applicable to the transformed one or more layers of the NN into one or more second nonlinearities (also referred to in this disclosure as local nonlinearities) applicable to each tensor of each of the one or more tensor networks.

[0010] The device or system incorporates machine learning routines in the form of TNNs, which can at least reduce the memory requirements of similar machine learning routines in the form of NNs.

[0011] A TNN contains one or more tensor networks, which are linear structures that can provide a complete and scalable description of the simulation of complex quantum many-body systems.

[0012] Although TNNs have been studied in the past, they have all been characterized by memory scalability issues and inconsistent inference. These issues arise because NNs have a huge amount of intermediate feature data, including sparse tensor network components, that are generated as data passes through the NN. In this sense, TNNs can compress machine learning routines like NNs, but they do not reduce the memory required during training or inference, making them ineffective at scaling up machine learning routines.

[0013] The feature vectors of NNs could not be compressed due to the NN's existing nonlinearities, which are pointwise applicable to the feature data. As mentioned above, tensor networks are linear structures. Therefore, tensor networks are limited in this respect. This is because, in order to replicate the applicability of nonlinearities, which are highly relevant in the inference performed by NNs, it is necessary to decompose the compressed data into larger data structures, apply nonlinearities to them, compress them again until more nonlinearities are applied, and then repeat the same process. The relevance of nonlinearities is due to the high nonlinearity content in various applications and tasks, including, but not limited to, image classification.

[0014] This device or system addresses this limitation by generating transformed versions of one or more nonlinearities that are applied to the tensors within each tensor network instead of being applied to the tensors within the respective tensor network. By associating the first nonlinearity(s) with each linear layer that is transformed into the resulting tensor network, while transforming them into a second nonlinearity(s), the nonlinearities can be applied to the tensor network without first requiring decomposition of the feature vectors. In other words, the generated feature data is maintained in a compact tensor network representation during the entire forward pass through the network. In this way, the memory requirements and, therefore, the computational resources required for the aforementioned process of decomposing, applying the nonlinearity(s), and compressing again can be reduced.

[0015] Converting a layer(s) of a NN into a tensor network(s) can be done according to processes known in the art.

[0016] The transformation of the global nonlinearity(s) of the NN into the local nonlinearity(s) is preferably done so that the resulting nonlinearities partially or completely reproduce the behavior of the global nonlinearity(s).

[0017] At least one generated output may be provided as an output of the device or system, for example in the form of data or a user-perceptible signal, and another device or system, or even a person, may then process such output(s) to provide one or more inputs to the target or its controller, or in the form of one or more instructions for automatic configuration or reconfiguration of the target or its controller.

[0018] In some embodiments of this aspect, the device or system is further configured to add at least one gauge optimization to the transformed neural network.

[0019] As known in the art, gauge optimization allows for the modification of tensors in a tensor network without modifying the data represented by the tensor network, i.e., the reduction of the tensor network with the modified tensors remains the same. In this sense, gauge optimization can be implemented that modifies the tensors but not the tensor network, in addition to resolving one or more identities, particularly for the internal connections of the tensors. This can result in a reduction in the computational resources required to run machine learning routines in the form of TNNs and / or an improvement in the numerical accuracy of the machine learning routines by adapting the tensors to a more stable form, for example, as known in the art.

[0020] In some embodiments of this aspect, the at least one gauge optimization is selected from a predetermined set of gauge optimizations.

[0021] Depending on the global nonlinearity(s) being transformed, it may be established that a particular gauge optimization achieves good results in terms of the similarity of the local nonlinearity(s) to the global nonlinearity(s), especially when the transformation from the global nonlinearity(s) to the local nonlinearity(s) is not sufficiently accurate. Additionally or alternatively, the gauge optimization may include adjustable parameters to increase the similarity between the local nonlinearity(s) and the global nonlinearity(s). Regarding the latter, in some embodiments, at least one gauge optimization includes trainable parameters that tune the respective gauge optimization. Furthermore, the device or system is configured to train a predetermined machine learning routine in the form of a TNN using the set of training data such that at least one of the one or more first nonlinearities applies a nonlinearity (e.g., a nonlinear coefficient or term) to the tensor that is more similar to the respective transformed second nonlinearity than if the gauge optimization had not been applied, and the training tunes the trainable parameters of the at least one gauge optimization. Therefore, gauge optimization can be optimized as well, so that machine learning routines with more accurate inferences can be obtained.

[0022] A second aspect relates to an apparatus or system as claimed in claim 7 attached hereto, namely an apparatus or system including at least one classical processor and / or at least one quantum processor, which may be configured to perform at least: providing a predetermined machine learning routine in the form of a tensorized neural network, the predetermined machine learning routine being associated with a target machine or system or process; and Generate at least one output related to the target machine, system, or process, where the at least one output is inferred by a provided predetermined machine learning routine when a dataset is input thereto.

[0023] A tensorized neural network includes at least the following: a plurality of layers, each layer including a respective tensor network containing intermediate feature data relating to a target machine or system or process; and One or more per-layer nonlinearities that can be applied to each tensor in the tensor network in each layer.

[0024] The apparatus or system of this embodiment, like the first embodiment, includes a machine learning routine in the form of a TNN, which has reduced memory requirements compared to a similar routine in the form of a NN.

[0025] The machine learning routine can be directly provided in the form of a TNN, i.e., it is not necessary to first provide the routine in the form of a NN and then convert it to the form of a TNN. The routine has a TNN defined from the beginning using multiple tensor networks similar to layers of a NN, and the device or system reduces the computational burden of providing a TNN-based machine learning routine without first providing a NN.

[0026] The number of tensors in each tensor network and the connection dimension of the tensor network can be selected according to predetermined values, depending, for example, on the type of inference required, and / or the type of target being monitored or controlled, and / or the number of features being fed to the machine learning routine to provide at least one output.

[0027] In contrast to known machine learning routines based on TNNs, the device or system of this embodiment has one or more local nonlinearities applied to each tensor, such that the linear structures in a TNN behave nonlinearly, thereby having behavior similar to that of a NN in which a global nonlinearity exists for each layer. The number and types of nonlinearities in a tensor network can be selected according to a predetermined value or can depend on one or more of the aforementioned criteria, i.e., the type of inference, the type of target, and / or the number of features to be considered.

[0028] A third aspect relates to an apparatus or system including at least one classical processor and / or at least one quantum processor, which may be configured to perform at least the following: Transforming one or more first nonlinearities (i.e., global nonlinearities) applied to a layer having one or more tensors of a given machine learning routine into one or more second nonlinearities (i.e., local nonlinearities) applicable to each tensor of the one or more tensors in the respective layer; and Generate at least one output related to the target machine, system, or process, which at least one output is inferred by the transformed predetermined machine learning routine when the dataset is input thereto.

[0029] The predetermined machine learning routine is or includes a tensorized neural network. In this manner, the apparatus or system converts one or more existing TNNs with global nonlinearities into one or more TNNs with local nonlinearities, thereby achieving the advantages discussed above in connection with the previous embodiment.

[0030] In some embodiments of any one of the second and third aspects, the apparatus or system is further configured to add at least one gauge optimization to the tensorized neural network.

[0031] Gauge optimization may also be included in or directly converted from a TNN-based routine to provide the local nonlinearity(s). For example, a gauge optimization may include one or more trainable parameters that can be varied during the training process.

[0032] In an embodiment of any one of the first, second and third aspects, the device or system is further configured to train a predetermined machine learning routine in the form of a tensorized neural network with the training dataset.

[0033] Training of TNN-based routines can be done in any manner known in the art, including the same manner in which NN-based routines are trained. In this regard, widely available libraries such as, but not limited to, PyTorch and TensorFlow, have automatic differentiation capabilities and can train routines.

[0034] In some embodiments of any one of the first, second, and third aspects, the apparatus or system is further configured to acquire at least a portion of the dataset from at least one or more sensors and / or one or more computing devices, wherein the one or more sensors and / or one or more computing devices may be communicatively coupled to a target machine or system, or a controller of a target process, or a controller of the target machine or system.

[0035] The device or system can be coupled to a target, or a computing device or its controller, to use a machine learning routine, where features about the target can be detected and considered by the routine to infer at least one output. Similar configurations are possible for training the machine learning routine, and other data sources can also be used.

[0036] In some embodiments of any one of the first, second and third aspects, the device or system is further configured to provide at least one instruction for operating one or more actuators and / or controllers of the target machine or system or process based on the at least one output.

[0037] A fourth aspect is directed to a system as claimed in claim 11 attached hereto, i.e. a system which may include a controller which includes or is communicatively coupled (or couplable) to an apparatus or system according to the first aspect or the second aspect or the third aspect, the system further configuring a target machine or system.

[0038] A fifth aspect relates to a method according to the appended claim 12, namely a method which may include at least: Transforming a predetermined machine learning routine in the form of a neural network into a tensorized neural network, and the predetermined machine learning routine is associated with a target machine or system or process by: converting one or more layers of the plurality of layers of the neural network into one or more tensor networks, respectively; and Transforming one or more nonlinearities applicable to the transformed one or more layers of the neural network into one or more nonlinearities applicable to each tensor of each of the one or more tensor networks; and Generating at least one output related to the target machine or system or process, the at least one output being inferred by the transformed predetermined machine learning routine when the dataset is input thereto.

[0039] The method may be performed, for example, in an apparatus or system as described with reference to any one of the first, second and third aspects, i.e. an apparatus or system including at least one classical processor and / or at least one quantum processor.

[0040] In some embodiments of the first aspect or the fifth aspect, the predetermined machine learning routine is a trained machine learning routine that is converted into a TNN.

[0041] A routine converted into a TNN-based machine learning routine may originate from an already trained NN-based machine learning routine, thereby improving at least one output produced by the TNN-based routine because the tensor network to which the layer is converted is influenced by the trained neural network. Even if the NN-based routine is trained, further training of the TNN-based routine can produce better results.

[0042] In some embodiments of the first aspect or the fifth aspect, the transformed one or more layers of the plurality of layers of the NN include all layers of the plurality of layers of the NN.

[0043] To fully utilize the advantages of TNNs, the entire neural network can be replaced by a TNN with multiple tensor networks and nonlinearities.

[0044] In some embodiments of this aspect, the method further includes adding at least one gauge optimization to the transformed neural network.

[0045] In some embodiments of this aspect, the at least one gauge optimization is selected from a predetermined set of gauge optimizations.

[0046] In some embodiments of this aspect, the at least one gauge optimization includes a trainable parameter that tunes each of the gauge optimizations. Further, the method also includes training a predetermined machine learning routine in the form of a TNN using the training data set such that at least one of the one or more first nonlinearities applies a nonlinearity (e.g., a nonlinear factor or term) to the tensor that is more similar to the respective transformed second nonlinearity than if the gauge optimization had not been applied, and the training tunes the trainable parameter of the at least one gauge optimization.

[0047] A sixth aspect relates to a method as claimed in claim 13, which may include at least: providing a predetermined machine learning routine in the form of a tensorized neural network, the predetermined machine learning routine associated with a target machine or system or process, the tensorized neural network including: a plurality of layers, each layer comprising a respective tensor network containing intermediate feature data relating to a target machine or system or process; and One or more nonlinearities per layer, applied to each tensor in the tensor network in each layer, and Generating at least one output related to the target machine or system or process, the at least one output being inferred by a provided predetermined machine learning routine when a dataset is input thereto.

[0048] The method may be performed, for example, in an apparatus or system as described with reference to any one of the first, second and third aspects, i.e. an apparatus or system including at least one classical processor and / or at least one quantum processor.

[0049] A seventh aspect relates to a method comprising at least: Transforming one or more first nonlinearities applied to a layer having one or more tensors of a given machine learning routine into one or more second nonlinearities applicable to each tensor of the one or more tensors of the respective layer; and Generating at least one output related to the target machine or system or process, the at least one output being inferred by a transformed predetermined machine learning routine upon inputting the dataset thereto.

[0050] The method may be performed, for example, in an apparatus or system as described with reference to any one of the first, second and third aspects, i.e. an apparatus or system including at least one classical processor and / or at least one quantum processor.

[0051] In an embodiment of any one of the fifth, sixth and seventh aspects, the method includes training a predetermined machine learning routine in the form of a tensorized neural network using a training dataset.

[0052] In some embodiments of any one of the fifth, sixth, and seventh aspects, the method includes obtaining at least a portion of the dataset from at least one or more sensors and / or one or more computing devices.

[0053] In some embodiments of any one of the fifth, sixth, and seventh aspects, the method includes providing at least one instruction for operating one or more actuators and / or controllers of the target machine or system or process based on the at least one output.

[0054] In some embodiments of any one of all the aforementioned aspects, the target machine or system or process comprises any one of a computing device or system, a factory line or machine thereof, a factory, a factory line or factory production process, a transportation means or automatic control unit thereof, an automated transportation control process, an electric power grid or network, an energy power plant (e.g., a wind farm, a solar power plant, a hydrogen production plant, etc.), an electric power station, an electric power generation process, an electric energy distribution process, etc.

[0055] In some embodiments of any one of all the aforementioned aspects, the at least one output includes any one of: a prediction of a failure of a machine (e.g., a factory line or machine thereof, a vehicle, a computing device or system, etc.); a predictive maintenance decision for a machine; a production of energy (e.g., by an energy power plant, a power plant, a solar power plant, a wind power plant, etc.); a production of a substance (e.g., hydrogen by a hydrogen production plant) or an object (e.g., a commodity by a factory); an operation of a control unit of a vehicle;

[0056] In some embodiments of any one of all of the aforementioned aspects, the at least one quantum processor includes or is part of a variational quantum circuit on one or more universal gate-based quantum computers based on superconducting qubits, cold atoms, ion traps, photonic systems, quantum dots, etc.

[0057] In some embodiments of any one of all the aforementioned aspects, the one or more local and global nonlinearities include at least one exponential function, and / or at least one rectified linear function (ReLU), and / or at least one hyperbolic tangent function, and / or at least one sigmoid function.

[0058] As an example, in the case of an exponential function, as a result of running a machine learning routine, every entry (i.e., component) of every tensor in the transformed tensor network with features related to the target is replaced by the respective exponential function, e.g., exp(ax), where a is a modifiable parameter and x is an entry. As another example, in the case of a ReLU function, every entry of every tensor in the transformed tensor network with features related to the target has negative entries replaced by zero, e.g., max(x,0), where x is an entry.

[0059] It should be noted that apart from the functions mentioned above, other functions are possible as well and are within the scope of the present disclosure.

[0060] Similar advantages to those described in relation to the first, second and third aspects also apply to the fourth, fifth, sixth and seventh aspects.

[0061] An eighth aspect relates to a computer program comprising instructions which, when executed by at least one computing device having at least one classical processor, cause the at least one computing device to perform the steps of a method as disclosed in the fifth aspect or the sixth aspect or the seventh aspect.

[0062] In some embodiments, the computer program is embodied on a non-transitory computer-readable storage medium having the computer program stored thereon.

[0063] A ninth aspect relates to a data carrier signal carrying a computer program according to the eighth aspect.

[0064] To complete the description and provide a better understanding of the present disclosure, a set of drawings are provided. The drawings form an integral part of this specification, illustrate embodiments of the present disclosure, and should not be construed as limiting the scope of the disclosure, but merely as examples of how the disclosure may be practiced. The drawings include the following figures: [Brief explanation of the drawings]

[0065] [Figure 1] FIG. 1 shows an apparatus or system according to an embodiment. [Figure 2] 2 and 3 show a method according to an embodiment. [Figure 3] 2 and 3 show a method according to an embodiment. [Figure 4A] Figures 4A and 4B show the NN-based machine learning routine. [Figure 4B] Figures 4A and 4B show the NN-based machine learning routine. [Figure 5] FIG. 5 illustrates a TNN-based machine learning routine according to an embodiment. [Figure 6] FIG. 6 partially illustrates a TNN-based machine learning routine according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0066] Detailed Description FIG. 1 shows an apparatus or system 1 according to an embodiment.

[0067] The device or system 1 includes one or more classical processors 2 and / or one or more quantum processors 3, i.e., there may be only one type of processor or both types of processors. The device or system 1 also includes at least one memory 4 for storing data, such as machine learning routines in the form of TNNs. The at least one memory 4 can also be used to store data generated during the execution of the machine learning routines, in particular feature vectors and inputs and / or outputs. Furthermore, the at least one memory 4 can store a computer program in the form of instructions that, when executed, performs a method according to the present disclosure.

[0068] In the case of a system, the system may include one or more devices adapted to provide machine learning routines in the form of TNNs according to the present disclosure.

[0069] If one or more quantum processors 3 are part of the device or system 1, converters for converting electrical signals into the type of signal processed by the quantum processors 3, and vice versa, may be located as part of the device or system 1 or externally, and in the latter case at least one data communication module may be used for transmitting data to and receiving data from the converters.

[0070] In some embodiments, one or more of the at least one classical processor includes or is part of at least one field programmable gate array (i.e., FPGA), and the at least one FPGA stores and / or executes a predetermined machine learning routine in the form of a TNN to generate at least one output.

[0071] In some embodiments, one or more classical processors of the at least one classical processor are distinct from or do not include an FPGA, and the one or more classical processors store and / or execute predetermined machine learning routines in the form of a neural network.

[0072] FIG. 2 illustrates a method 10 according to an embodiment.

[0073] The method 10 comprises the step 20 of converting a predetermined machine learning routine in the form of a neural network into a predetermined machine learning routine in the form of a tensorized neural network, the machine learning routine relating to a target machine or system or process to be monitored and optionally controlled.

[0074] To convert a routine from a neural network to a TNN, method 10 includes step 21 of converting one, some, or all of the neural network's linear layers into respective tensor networks, such that each linear layer of the converted NN becomes a tensor network. If the NN has already been trained, the converted layers are trained layers, so the TNN-based routine already benefits from the previous training. The decomposition of the tensor network can be selected for each layer of the NN (and for each layer of the TNN), and the size and number of tensors can be selected for each tensor network depending, for example, on the characteristics of the problem at hand (e.g., target type, output type and number, number of features, etc.), the computational resources required to run the TNN, and the desired accuracy of the output(s) provided. The size of the tensor also affects the number of learnable parameters; the smaller the size, the fewer the number of parameters. In fact, it has been shown that TNNs according to the present disclosure can reproduce the accuracy of a NN with fewer learnable parameters than the NN itself. Further, method 10 includes a step 22 of converting one, some, or all of the nonlinearities applicable to the transformed layers of the NN into nonlinearities applicable to each tensor of the respective tensor network resulting from the converting step 21. Such resulting nonlinearities are applied locally to each component of the respective tensor, thereby simulating the nonlinear behavior of the NN-based routine without having to expand the contracted tensor network to apply the nonlinearities.

[0075] Once the conversion 20 step is complete, in some embodiments, method 10 includes training the converted machine learning routine (not shown) to improve the results provided by the routine. The training can be any method known in the art and can depend on the type of data the routine generates, and thus can be supervised, reinforcement, or unsupervised. For the routine to operate properly, it should be trained at least once while in NN form and / or after the conversion has been performed.

[0076] Method 10 also includes inputting 31 a dataset into the transformed routine, i.e., the TNN-based routine, for inference 30 to generate at least one output 40 related to the target machine, system, or process. Because the tensor network in the routine is reduced and nonlinearities are applied directly to the tensors, the transformed routine can benefit from reduced memory requirements, facilitating the use of machine learning routines in more complex applications. In this regard, in some embodiments, a TNN occupies less space in at least one memory than a NN. That is, the space required in memory to store the TNN, and thus the machine learning routine based on the TNN, is less than the space required to store the NN, and thus the machine learning routine based on the NN.

[0077] Additionally, method 10 includes repeating input 31 and output generation 30, as indicated by the dashed arrow lines. At each pass, the routine may also adjust the values of tensors in the tensor network and the values of neurons in the linear layer of the neural network, if these are to be preserved and therefore not converted to a tensor network.

[0078] The generated output(s) provide information about the target, thereby enabling monitoring of the target, particularly its status based on characteristics input into the routine. The output / scan also serves to control the target. To that end, method 10 may also include generating at least one instruction 50 for operating one or more actuators and / or controllers of the target machine or system or process based on previously generated output 40.

[0079] FIG. 3 shows a method 11 according to an embodiment.

[0080] The method 11 includes providing a predetermined machine learning routine in the form of a tensorized neural network, the machine learning routine relating to a target machine or system or process to be monitored and optionally controlled.

[0081] The routine includes multiple layers in the form of tensor networks, each containing multiple tensors, each of which ultimately contains intermediate feature data about the target machine, system, or process. The routine also includes, for each layer, i.e., for each tensor network, one or more nonlinearities that can be applied to each tensor in each tensor network.

[0082] Method 11 includes a step 29 of training a TNN-based machine learning routine to improve the results provided by the routine. The training can be any method known in the art and can be supervised, reinforcement, or unsupervised depending on the type of data the routine generates.

[0083] The method 11 also includes inputting 31 the dataset into the TNN-based routine for generating 30 at least one output 40, as described with reference to Figure 2. The method 11 may also include generating at least one instruction 50 for operation, as described with reference to Figure 3.

[0084] In some embodiments of methods 10, 11, and any one of the aforementioned aspects, one or more of the at least one output is a value for: Predictive maintenance of targets such as factory lines, energy power plants, etc., or Weather forecasting for energy power plants such as wind farms, solar power plants, or Forecasting the amount of electricity generated by an energy power plant; or For example, optimization of a target such as a factory line, an energy power plant, or an electricity market. The optimization can be done by, for example, optimizing the resources of the target, the energy consumption by the target, or Optimizing cost functions in machine learning algorithms (both supervised and unsupervised) such as neural networks, or Solve combinatorial optimization problems with or without constraints, or Transport modes, e.g., air, sea, road, etc., for air / sea / road traffic optimization; or Portfolio optimization in finance, or Optimizing cost functions in computer vision systems, factories, and other environments.

[0085] In some embodiments of methods 10, 11, and any one of the aforementioned aspects, the connection dimension is selected to be a predetermined value. For example, in some embodiments, each tensor network of the tensorized neural network includes a connection dimension of 50 or less, preferably 10 or less, and more preferably 5 or less.

[0086] The connection dimension is a value that defines a tensor network and establishes the level of correlation between tensors, thus affecting the numerical accuracy of the output provided by the routine. A larger connection dimension results in more accurate output, but requires more computational resources to run the routine. The aforementioned connection dimension values have been found to provide a good tradeoff between accuracy and computational resources.

[0087] In some embodiments of methods 10, 11, and any one of the aforementioned aspects, the number of tensors in each tensor network is selected to be a predetermined value. For example, in some embodiments, each tensor network of the tensorized neural network includes a number of tensors of 50 or less, preferably 10 or less, and more preferably 5 or less.

[0088] In some implementations of method 10 and any one of the first and fifth aspects, the number of tensors in the transformed tensor network is less than or equal to two times the number of neurons in each linear layer of the neural network.

[0089] 4A and 4B show different representations of a NN-based machine learning routine 100 known in the art.

[0090] The NN-based routine 100 includes an input layer 60 having a number of features related to a target, multiple linear layers 70a-70n each containing neurons 71a-71n, a global nonlinearity 80a-80n for each linear layer, and an output layer 40 having one or more outputs 41a-41n. In each linear or hidden layer 70a-70n, intermediate features are generated in each pass of the NN-based routine 100 and then globally processed by a respective global nonlinearity 80a-80n. The global nonlinearity may be, for example, a ReLU function, a sigmoid function, etc.

[0091] FIG. 5 illustrates a TNN-based machine learning routine 110 according to an embodiment.

[0092] The TNN-based routine 110 has one or more input layers 60, one or more layers in the form of tensor networks 75a-75n, each containing tensors 75a1-75a4, 75n1-75n3, and local nonlinearities 85a-85n, each containing local nonlinearities 85a1-85a4, 85n1-85n4 applicable to the respective tensors; thus, the routine 110 is a locally activated TNN, and further comprises an output layer 40 having one or more outputs 41a-41n.

[0093] As data enters from input layer(s) 60, it passes through tensors 75a1-75a4, 75n1-75n3, each of which is locally modified by a respective local nonlinearity 85a1-85a4, 85n1-85n4 (e.g., hyperbolic tangent function, ReLU function, exponential function, etc.) before reaching the tensor of a subsequent tensor network or the output of one or more outputs 40. Such a structure allows for efficient generation and processing of intermediate features and reduces memory requirements.

[0094] Regarding memory requirements, the table below qualitatively shows the amount of memory required for similar machine learning routines with different types of networks. [Table 1]

[0095] where D is the number of inputs and outputs per neuron, k is the number of neurons or tensors per layer, X is the connection dimension per tensor network layer (i.e., the dimension of the indices of the tensors connected in the tensor network layer), and Φ is the connection dimension per nonlinearity layer (i.e., the dimension of the indices of the tensors connected in the nonlinearity layer). As can be seen, X and Φ are adjustable parameters when providing a TNN-based routine, whether providing the TNN-based routine directly or in a form converted from an NN-based routine. By adjusting these parameters, the memory requirements of a TNN-based routine can be lower than those of a similar NN-based routine or a TNN-based routine with global nonlinearity.

[0096] The table below provides a qualitative comparison of different types of networks in terms of the processing time and computational resources required to run machine learning routines. [Table 2]

[0097] In some embodiments, each nonlinearity in the TNN-based routine 110 may be a tensor, and each set of nonlinearities 85a-85n associated with a tensor network 75a-75n may also be defined as a tensor network, which may simplify the processing of machine learning routines due to the way tensor networks are processed in classical and / or quantum processors, particularly the contraction of tensor networks.

[0098] While the TNN and NN-based routines 110, 100 have been illustrated with two layers and three and four tensors or neurons, respectively, it should be noted that these are merely examples of the present disclosure and that additional layers and other numbers of tensors and / or neurons are also possible within the scope of the present disclosure. Additionally, while the tensor network of the TNN-based routine 110 has been illustrated as one-dimensional, it should be noted that this is merely an example of the present disclosure and that other dimensions are also possible within the scope of the present disclosure.

[0099] FIG. 6 partially illustrates a TNN-based machine learning routine according to an embodiment.

[0100] Part of this routine is a tensor network layer, as shown with reference to Figure 5. In this case, the tensor network includes at least three tensors 75n1 to 75n3. In the overall TNN-based routine, such a tensor network is connected to other layers, such as an input layer, an output layer, a linear layer (if not all linear layers of the NN-based routine are converted to tensor networks), and / or a tensor network layer.

[0101] In this example, the tensor network layer includes gauge optimizations 90a, 90b', 90b'', and 90c placed between adjacent tensors of the same tensor network. The gauge optimizations can be tuned or trained to better reproduce the desired nonlinearity applied to the tensors.

[0102] In this specification, the terms "comprises," "comprises," and derivatives thereof (such as "including," "comprises," etc.) should not be understood in an exclusive sense, i.e., these terms should not be interpreted as excluding the possibility that what is described and defined may include additional elements, steps, etc.

[0103] However, the present disclosure is clearly not limited to the particular example(s) or embodiment(s) described herein, but encompasses all modifications that may occur to those skilled in the art (e.g., with regard to selection of materials, dimensions, components, configurations, etc.) within the general scope of the invention as defined by the claims.

Claims

1. 1. An apparatus comprising at least one classical processor, or at least one quantum processor, or a combination thereof, a predetermined machine learning routine in the form of a neural network associated with a target machine or system or process; converting one or more layers of the neural network into one or more tensor networks, respectively; and transforming one or more first nonlinearities applicable to the transformed one or more layers of the neural network into one or more second nonlinearities applicable to each tensor of each of the one or more tensor networks; and convert it into a tensorized neural network by generating at least one output related to the target machine or system or process, the at least one output being inferred by the transformed predetermined machine learning routine upon inputting a dataset thereto; An apparatus configured as follows.

2. 2. The apparatus of claim 1, wherein the predetermined machine learning routine is a trained machine learning routine that is converted into the tensorized neural network.

3. 10. The apparatus of claim 1, further configured to add at least one gauge optimization to the transformed neural network, wherein the at least one gauge optimization is selected from a set of predetermined gauge optimizations.

4. 4. The apparatus of claim 3, wherein the at least one gauge optimization includes a trainable parameter that adjusts each of the gauge optimizations, and the apparatus or system is further configured to train the predetermined machine learning routine in the form of the tensorized neural network with a training dataset, wherein the training adjusts the trainable parameter of the at least one gauge optimization so that the nonlinearity applied by at least one of the one or more second nonlinearities more closely resembles a transformed version of the one or more first nonlinearities.

5. 10. The apparatus of claim 1, further comprising at least one memory adapted to store the neural network and the tensorized neural network, wherein the tensorized neural network occupies less space in the at least one memory than the neural network.

6. 2. The apparatus of claim 1, wherein the transformed one or more layers of the plurality of layers of the neural network include all layers of the plurality of layers of the neural network.

7. 10. The apparatus of claim 1, further configured to train the predetermined machine learning routine in the form of the tensorized neural network with a training dataset.

8. The apparatus of claim 1 , further configured to obtain at least a portion of the data set from at least one or more sensors or one or more computing devices or a combination thereof.

9. 10. The apparatus of claim 1, wherein the apparatus is further configured to provide at least one instruction for operation of one or more actuators or controllers or combinations thereof of the target machine or system or process based at least on the at least one output.

10. 10. The apparatus of claim 1, wherein the target machine or system or process comprises one of a computing device or system, a factory line or machine thereof, a factory, a production process, a vehicle or automated control unit thereof, an automated transportation control process, an electrical power grid or network, an energy power plant, an electric power station, an electrical power generation process, and an electrical energy distribution process.

11. 10. The apparatus of claim 1, wherein the at least one output comprises one of a prediction of machine failure, a predictive maintenance decision for a machine, an energy production, a material or object production, and an operation of a control unit of a vehicle.

12. 1. An apparatus comprising at least one classical processor or at least one quantum processor or a combination thereof, providing a predetermined machine learning routine associated with a target machine or system or process in the form of a tensorized neural network, said tensorized neural network comprising: a plurality of layers, each layer comprising a respective tensor network containing intermediate feature data relating to the target machine or system or process; one or more per-layer nonlinearities applicable to each tensor of the tensor network in each of the layers; generating at least one output related to the target machine or system or process, the at least one output being inferred by the provided predetermined machine learning routine when a dataset is input thereto; An apparatus configured as follows.

13. 13. The apparatus of claim 12, further configured to train the predetermined machine learning routine in the form of the tensorized neural network with a training dataset.

14. 13. The apparatus of claim 12, wherein the apparatus is further configured to obtain at least a portion of the data set from at least one or more sensors or one or more computing devices or a combination thereof.

15. 13. The apparatus of claim 12, wherein the apparatus is further configured to provide at least one instruction for operation of one or more actuators or controllers or combinations thereof of the target machine or system or process based at least on the at least one output.

16. 13. The apparatus of claim 12, wherein the target machine or system or process comprises one of a computing device or system, a factory line or machine thereof, a factory, a production process, a vehicle or automated control unit thereof, an automated transportation control process, an electrical power grid or network, an energy power plant, an electric power station, an electrical power generation process, and an electrical energy distribution process.

17. 13. The apparatus of claim 12, wherein the at least one output comprises one of a prediction of machine failure, a predictive maintenance decision for a machine, an energy production, a material or object production, and an operation of a control unit of a vehicle.

18. a predetermined machine learning routine in the form of a neural network associated with a target machine or system or process; converting one or more layers of the neural network into one or more tensor networks, respectively; and transforming one or more nonlinearities applicable to the transformed one or more layers of the neural network into one or more nonlinearities applicable to each tensor of each of the one or more tensor networks; and converting it into a tensorized neural network by generating at least one output related to the target machine or system or process, wherein the at least one output is inferred by the transformed predetermined machine learning routine upon inputting a dataset thereto; A method comprising:

19. 20. The method of claim 18, wherein the target machine or system or process comprises one of a computing device or system, a factory line or machine thereof, a factory, a production process, a vehicle or automatic control unit thereof, an automated transportation control process, an electrical power grid or network, an energy power plant, an electric power station, a power generation process, and an electrical energy distribution process.

20. 20. The method of claim 18, wherein the at least one output comprises one of a prediction of machine failure, a predictive maintenance decision for a machine, an energy production, a material or object production, and an operation of a control unit of a vehicle.

Images