Multidimensional deep neural network

Abstract

An artificial intelligence (AI) system is disclosed. The AI ​​system includes: an input interface for receiving input data; a memory for storing a multidimensional neural network having a sequence of deep neural networks (DNNs) having an inner DNN and an outer DNN; a processor configured to submit input data to the multidimensional neural network to produce the output of the outer DNN; and an output interface for rendering at least one function of the output. Each DNN processes the input data sequentially by a sequence of layers along a first dimension of data propagation. The DNNs are arranged from the inner DNN to the outer DNN along a second dimension of data propagation. Furthermore, the DNNs are connected such that the output of at least one layer of a DNN is combined with the input of at least one layer of a subsequent DNN in the DNN sequence.

Description

Technical Field

[0001] This invention relates generally to artificial neural network technology, and in particular to methods and systems including multidimensional deep neural networks. Background Technology

[0002] Artificial Neural Networks (ANNs) are computational systems inspired by the biological neural networks of the human brain. These systems are widely used in various fields, such as Natural Language Processing (NLP), image processing, and computer vision. Typically, an ANN is a directed, weighted graph of interconnected neurons (nodes). These interconnected neurons are grouped into layers. Each layer of an ANN performs a mathematical transformation (e.g., a non-linear transformation) on the input data to generate output data. An ANN can have input layers, hidden layers, and output layers to process the input data. Between each layer is an activation function used to determine the ANN's output. To improve the accuracy of processing input data, some ANNs have multiple hidden layers. Such ANNs with multiple layers between the input and output layers are called Deep Neural Networks (DNNs). The interconnected neurons of a DNN contain data values. When a DNN receives input data, the input data propagates forward through each layer. Each layer computes an output and provides that output as input to the next layer. Therefore, the input data is propagated in a feedforward manner. For example, a feedforward DNN performs function approximation, filtering a weighted combination of inputs through a non-linear activation function. Nonlinear activation functions are organized into a cascade of fully connected hidden layers.

[0003] Such DNNs need to be trained to perform tasks across various domains. However, training a DNN becomes a tedious process as the number of hidden layers increases to achieve better approximation. For example, the activation functions in a DNN can cause complex problems, such as the vanishing gradient problem during backpropagation of the objective function gradient between layers. Backpropagation determines the gradient of the loss function relative to the weights in the DNN. However, a large number of hidden layers in a DNN can cause the gradient to become zero (i.e., the vanishing gradient problem), which may be far from the optimal value. Furthermore, due to the large number of hidden layers, DNNs may encounter difficulties in optimizing the weights of neurons. This can delay the training process of the DNN and may slow down the improvement of the DNN model parameters, thus affecting the accuracy of the DNN. The vanishing gradient problem during DNN training can be addressed by introducing residual neural network layers into the DNN.

[0004] Residual Neural Networks (ResNets) utilize skip connections to add the outputs of the first few layers of a DNN to the inputs of other non-adjacent layers. Typically, ResNets are implemented with two or three layers of skip connections. Furthermore, ResNets only allow layer skips in the forward direction of input propagation. This prevents the formation of loops or cycles, which are problematic during DNN training and inference. However, in some cases, the forward propagation direction of the input can be a less than ideal constraint. This deficiency can be compensated for by increasing the number of hidden layers. Therefore, as additional hidden layers are added, the number of parameters may increase, while simultaneously propagating the input in the forward direction. This increase in the number of parameters can also delay the training process, which is undesirable.

[0005] Therefore, a technical solution is needed to overcome the above limitations. More specifically, there is a need to train neural networks with multiple hidden layers in an efficient and feasible manner, while avoiding the vanishing gradient problem and the problem of increasing parameter number. Summary of the Invention

[0006] Some implementations aim to provide an artificial neural network (ANN), such as a deep neural network (DNN) with a deep architecture, having multiple hidden layers that allow connections between layers regardless of their respective positions within the ANN. The DNN has multiple layers, each of which can be connected to non-adjacent layers. Furthermore, or alternatively, some implementations aim to increase the number of hidden layers in the DNN without increasing the number of training parameters. Additionally, or alternatively, some implementations aim to provide a DNN architecture that allows the reuse of outputs from different layers to improve DNN performance without increasing the number of parameters.

[0007] Some implementations are based on the understanding of the advantages of sharing information between layers of a DNN in both data propagation directions. For example, while outputs from the earlier layers of a DNN can be added to the inputs of other adjacent and non-adjacent layers, it is also beneficial to have outputs computed in later layers help to better process the input data or intermediate outputs from earlier layers. In this way, data can be exchanged in both directions, adding additional flexibility to data processing. However, propagating data in both directions can create logical loops, jeopardizing the training and execution of the DNN.

[0008] Some implementations are based on the understanding that the loop problem can be solved by cloning or copying the parameters of a DNN and unfolding the DNN in a direction different from the data propagation direction. For example, some implementations are based on the understanding that a sequence of hidden layers processing inputs sequentially can provide insightful information to a parallel sequence of hidden DNN layers that also process the same inputs sequentially. In some implementations, both sequences are feedforward neural networks with the same parameters. In this way, having multiple hidden layer sequences does not increase the number of parameters. In some implementations, at least some layers of one hidden layer sequence are connected to at least some layers of another hidden layer sequence to combine at least some intermediate outputs of the hidden layer sequence with at least some inputs of the other hidden layer sequence. Each hidden layer sequence corresponds to a DNN. The arrangement direction of the hidden layer sequence (i.e., the DNN) is different from the propagation direction of the inputs in each layer of the DNN. For this purpose, the DNNs in the DNN sequence are interconnected. For example, at least some layers of the first DNN are connected to at least some layers of subsequent DNNs. As used herein, two layers are connected when at least one function of the output of one layer forms at least a portion of the input of another connected layer. The connections between DNNs combine to form a single neural network, such as a multidimensional neural network. As used in this paper, in a multidimensional neural network, input data propagates in multiple directions, from the input layer of the DNN to the output layer and across the sequence of DNNs that form the multidimensional neural network.

[0009] In various implementations, the multidimensional neural network can have a different number of DNNs. In one implementation, the multidimensional neural network includes two DNNs (as an inner DNN and an outer DNN). Each of the DNNs (i.e., the inner DNN and the outer DNN) includes one or more intermediate (hidden) layers. When a layer of the inner DNN is connected to a layer of the outer DNN, the layers are connected at the input / output level to maintain the size of the inner and outer DNN layers. For example, the output of the inner DNN layer can be added to the input of the outer DNN layer to combine them.

[0010] The layers of the inner DNN and the layers of the outer DNN have multiple connections. For example, all layers of the inner DNN can be connected to all layers of the outer DNN. This connection pattern is referred to as fully connected in this paper, making the multidimensional neural network fully connected. Alternatively, the multidimensional neural network can be partially connected. For example, in a partially connected multidimensional neural network, one or more layers of the inner DNN can be connected to multiple layers of the outer DNN. Furthermore, or alternatively, in a partially connected multidimensional neural network, multiple layers of the inner DNN can be connected to one layer of the outer DNN.

[0011] Different implementations use different connection patterns, allowing for the adaptation of multidimensional neural networks to various applications. For example, in some implementations, the output of a given layer in the inner DNN may contribute only to the input of a unique layer in the outer DNN. In some implementations, the outputs of two given layers in the inner DNN may contribute to the input of the same layer in the outer DNN.

[0012] In addition to the different connection patterns between layers of different DNNs in a multidimensional neural network, some implementations also use different types of connections. For example, various implementations use hard connections, soft connections, or a combination thereof. For instance, in hard connections, the output of an inner DNN layer is added as a whole to the input of an outer DNN layer. That is, these layers are either connected or not connected. If the layers are connected, the output of one layer is combined with the input of another layer without additional scaling and / or weight multiplication. If the layers are not connected, the output of that layer is not added to the input of other layers.

[0013] Therefore, according to the principle of hard connections, the output of an inner DNN layer may or may not contribute to the input of an outer DNN layer. The principle of data propagation based on hard connections differs from the propagation principle between layers of a single DNN. Thus, hard connections allow for decoupling of data propagation in different directions. In turn, this decoupling allows for finding better hard connection patterns based on the trained DNN parameters, adding additional flexibility to the architecture of multidimensional neural networks.

[0014] In some implementations, during the training of a multidimensional neural network, hard connection patterns are selected from multiple connection patterns. For each selected connection pattern, the corresponding multidimensional neural network is trained. The best-performing multidimensional neural network is selected from all trained networks. More specifically, the selection of hard connection patterns is based on a search algorithm (e.g., a random search algorithm). This random search algorithm randomly selects a number of connection patterns from multiple connections and trains a model for each connection pattern. A model is then selected based on performance metrics on the validation set (e.g., accuracy, F1 score, BLEU score, etc.). For example, one or more connection patterns with high scores can be selected for runtime execution. In some cases, the selected connection patterns can be manipulated with minor modifications.

[0015] Alternatively, in soft-connection types, only a portion of the output of one layer is combined with the input of another layer. Specifically, the output of a soft connection between one layer and another is "weighted" before being added to the input of the other layer. The weights of the soft connections may vary for different types of soft connections.

[0016] In some other implementations, multiple connections may correspond to a soft-connection pattern. In the case of a soft-connection pattern, the output of an inner DNN layer, along with its weights, is added to the input of each layer of the outer DNN. In some exemplary implementations, these weights in the soft-connection pattern may be associated with all connections or subsets of connections between layers of the inner DNN and layers of the outer DNN. These weights may represent the connection strength between a given layer of the inner DNN and a given layer of the outer DNN. The output of a given layer of the inner DNN may be scaled by a factor depending on a set of connection weights before being combined with the input of a given layer of the outer DNN. In some implementations, during the training of the multidimensional neural network, the connection weights are trained simultaneously with the parameters of the DNN. In this way, unlike hard connections, the estimation of soft connections or soft-connection weights can be implemented as a process integrated with the training of the neural network. Therefore, the process of establishing soft connections is more in line with the principles of neural networks.

[0017] For example, in some implementations, multidimensional neural networks are fully connected using soft connections. Fully connected networks reflect what the network designer considers the maximum possible connectivity. The nature of soft connections allows training to determine which connections are more important than others.

[0018] For example, in some implementations, the training weights of soft connections can be pruned by retaining only a subset of connections based on their weight values. For instance, only connections with weights above a threshold can be retained, or only the connection with the highest weight among all connections from a given layer of the inner DNN, or only the connection with the highest weight among all connections entering a given layer of the outer DNN. After the connections are pruned, the network can be further trained using only the residual connections, and the weights of the residual connections are trained simultaneously. In another implementation, the residual soft connections can be converted to hard connections, and the resulting network can be further trained.

[0019] In another implementation, the multidimensional neural network includes one or more hidden DNNs between an inner DNN and an outer DNN. The DNNs of the multidimensional neural network are connected in a forward direction from the inner DNN to the outer DNN. For example, the input propagates forward from the inner DNN to the outer DNN. This forward propagation of input prevents loops or cycles between layers while allowing subsequent layers of one DNN to connect to the preceding layer of a subsequent DNN. Therefore, adding hidden DNNs to an existing ANN provides a deep architecture (i.e., a multidimensional neural network) without increasing the number of parameters or creating any loops in the corresponding layers.

[0020] In one exemplary embodiment, a multidimensional neural network forms a multithreaded transducer (MPT) architecture for NLP applications, such as language machine translation. The MPT includes an inner network and an outer network. The inner network corresponds to the inner DNN of the multidimensional neural network, and the outer network corresponds to the outer DNN of the multidimensional neural network. The outer network utilizes features of the layers of the inner network by adding the outputs of layers of the inner network to the raw input of at least one layer of the outer network. In the MPT, the same parameters of the inner network are shared with the outer network. Because the same parameters are shared between the inner and outer networks, the number of parameters is not increased. The MPT also performs feature refinement iteratively, which significantly improves the performance of machine translation. Furthermore, the MPT can be combined with self-attention networks and convolutional neural networks or feedforward neural networks for machine translation. In some exemplary embodiments, the MPT can be generated by searching a search space of multiple possible connection patterns (e.g., heuristic-based search). Heuristic-based search can be performed using evolutionary search algorithms. In some exemplary embodiments, the MPT may include connection weights that determine the connection strength between layers of the inner network and layers of the outer network of the MPT. The connection weights can be learned along with other neural network parameters. Furthermore, or alternatively, the MPT model for machine translation includes layers with a dual network or path, consisting of a self-attention subnetwork and a feedforward neural (FFN) subnetwork (e.g., a convolutional neural network). This dual combination of self-attention and FFN subnetworks can achieve better performance than a pure self-attention network.

[0021] Therefore, one embodiment discloses a computer-based artificial intelligence (AI) system. The AI ​​system includes: an input interface configured to receive input data; a memory configured to store a multidimensional neural network having a sequence of deep neural networks (DNNs), including an inner DNN and an outer DNN; a processor configured to submit the input data to the multidimensional neural network to produce the output of the outer DNN; and an output interface configured to render at least one function of the output of the outer DNN. In the multidimensional neural network, each DNN includes a sequence of layers, and corresponding layers of different DNNs have the same parameters. Each DNN is configured to process the input data sequentially by the layer sequence along a first dimension of data propagation. The DNNs in the DNN sequence are arranged along a second dimension of data propagation, starting from the inner DNN and continuing to the outer DNN. The DNNs in the DNN sequence are connected such that the output of an intermediate or final layer of a DNN is combined with the input of at least one layer of a subsequent DNN in the DNN sequence. The multidimensional neural network receives input data submitted by the processor to produce the output of the outer DNN.

[0022] Therefore, another embodiment discloses a method for generating the output of a multidimensional neural network. The method includes accepting input data via an input interface. The method includes submitting the input data to a multidimensional neural network having a sequence of DNNs, the sequence including an inner DNN and an outer DNN. Each DNN includes a sequence of layers, and corresponding layers of different DNNs have the same parameters. Each DNN is configured to process the input data sequentially by the layer sequence along a first dimension of data propagation. The DNNs in the DNN sequence are arranged along a second dimension of data propagation, starting from the inner DNN and continuing to the outer DNN. The DNNs in the DNN sequence are connected, and at least one of the intermediate or final outputs of a DNN is combined with the input of at least one layer of a subsequent DNN in the DNN sequence. The method includes generating the output of the outer DNN. The method also includes at least one function for rendering the output of the outer DNN.

[0023] Embodiments of this disclosure will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of embodiments of this disclosure. Attached Figure Description

[0024] [ Figure 1 ]

[0025] Figure 1 A schematic block diagram of an artificial intelligence (AI) system according to some embodiments of the present disclosure is shown.

[0026] [ Figure 2A ]

[0027] Figure 2A A block diagram is shown of a multidimensional neural network stored in the memory of an AI system according to an exemplary embodiment of the present disclosure.

[0028] [ Figure 2B ]

[0029] Figure 2B A block diagram of a multidimensional neural network stored in the memory of an AI system according to another exemplary embodiment of this disclosure is shown.

[0030] [ Figure 3 ]

[0031] Figure 3 The present disclosure illustrates the processing pipeline of an AI system for generating the output of a multidimensional neural network according to some embodiments of the present disclosure.

[0032] [ Figure 4A ]

[0033] Figure 4AAn exemplary schematic diagram of an encoder formed by a multidimensional neural network according to some exemplary embodiments of the present disclosure is shown.

[0034] [ Figure 4B ]

[0035] Figure 4B A block diagram of a multi-threaded converter (MPT) according to an exemplary embodiment of the present disclosure is shown, wherein the inner network layer of the MPT has a hard-connected mode between the outer network layer of the MPT.

[0036] [ Figure 4C ]

[0037] Figure 4C A block diagram is shown of an MPT with a search-based hard-connected pattern for a certain machine translation application according to an exemplary embodiment of the present disclosure.

[0038] [ Figure 4D ]

[0039] Figure 4D A block diagram of an MPT with a weighted soft connection pattern between an inner DNN layer and an outer DNN layer is shown according to another exemplary embodiment of the present disclosure.

[0040] [ Figure 5 ]

[0041] Figure 5 A block diagram is shown of the attention module in each layer of each DNN in a multidimensional neural network according to some embodiments of the present disclosure.

[0042] [ Figure 6A ]

[0043] Figure 6A A block diagram illustrating a connection pattern between two layers of two DNNs according to an exemplary embodiment of the present disclosure, the connection pattern combining the final output of one DNN layer with the input of a subsequent DNN layer.

[0044] [ Figure 6B ]

[0045] Figure 6B A block diagram illustrating a connection pattern between two layers of two DNNs according to another exemplary embodiment of the present disclosure, the connection pattern combining the final output of one DNN layer with the input of a subsequent DNN layer.

[0046] [ Figure 6C ]

[0047] Figure 6CA block diagram illustrating a connection pattern between two layers of two DNNs according to another exemplary embodiment of the present disclosure, the connection pattern combining the intermediate output of one DNN layer with the input of a subsequent DNN layer.

[0048] [ Figure 6D ]

[0049] Figure 6D A block diagram is shown illustrating a connection pattern between two layers of two DNNs according to some embodiments of the present disclosure, which combines the intermediate output of one DNN layer with the input of a subsequent DNN layer.

[0050] [ Figure 7 ]

[0051] Figure 7 An ablation study table of an encoder corresponding to an MPT architecture according to an exemplary embodiment of this disclosure is shown.

[0052] [ Figure 8A ]

[0053] Figure 8A A comparison table is shown between an exemplary embodiment of MPT according to this disclosure and prior art methods.

[0054] [ Figure 8B ]

[0055] Figure 8B The advantages of a multi-threaded converter equipped with a multi-dimensional neural network according to some embodiments of this disclosure are illustrated.

[0056] [ Figure 9 ]

[0057] Figure 9 A flowchart illustrating a method for generating the output of a multidimensional neural network according to some embodiments of the present disclosure is shown.

[0058] [ Figure 10 ]

[0059] Figure 10 A block diagram of an AI system according to some embodiments of the present disclosure is shown.

[0060] [ Figure 11A ]

[0061] Figure 11A The following diagram illustrates an environment for machine translation applications according to some embodiments of the present disclosure.

[0062] [ Figure 11B ]

[0063] Figure 11BThis illustration shows a representation of a cooperative operating system using a machine translation application according to some embodiments of this disclosure. Detailed Implementation

[0064] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a full understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, apparatuses and methods are shown only as block diagrams to avoid obscuring the scope of the present disclosure.

[0065] As used in this specification and claims, the terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used with a list of one or more components or other items, should be understood to be open-ended, meaning that the list should not be construed as excluding other, additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the phrases and terms used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or restrictive effect.

[0066] Overview

[0067] In recent years, the architecture or structure of neural networks has evolved from recurrent neural networks (RNNs) to long short-term memory (LSTM) and convolutional neural networks (CNNs) with convolutional sequence architectures and transformers. Generally, convolutional sequence architectures and transformers are widely used in natural language processing, such as language representation learning. For computer vision applications, the neural architecture of a neural network corresponds to multi-path methods to achieve efficient information flow between the layers of the neural network. For NLP applications, the neural architecture corresponds to sequence-based neural architectures. Sequence-based neural architectures utilize the features of the last layer (i.e., the output layer) of the neural network, which provides a limited information flow. Some implementations are based on gaining insights from multi-path-based neural architectures.

[0068] Specifically, some implementations are based on the understanding of the advantages of sharing information between layers of a DNN in both directions of data propagation. For example, while the outputs from the earlier layers of a DNN can be added to the inputs of other adjacent and non-adjacent layers, it is also beneficial to have the outputs of later layers help to better process the input data or intermediate outputs from earlier layers. In this way, data can be exchanged in both directions, adding additional flexibility to data processing. However, propagating data in both directions can create logical loops, jeopardizing the training and execution of the DNN.

[0069] Some implementations are based on the understanding that the looping problem can be solved by cloning or copying the parameters of a DNN and unfolding it in a direction different from the data propagation direction. For example, some implementations are based on the understanding that a sequence of hidden layers that process the input sequentially can provide insightful information to a parallel sequence of hidden DNN layers that also process the same input sequentially.

[0070] To this end, some exemplary implementations disclose multi-level fusion mechanisms combining residual connections and dense connections to obtain robust neural architectures for applications such as NLP applications, computer vision applications, or combinations thereof. Residual connections enable skip connections between features of a neural network layer and other non-adjacent layers. Dense connections enable all possible connections between layers of the neural network. Residual connections and multi-level connections are implemented based on operations such as cascading, addition, and loop fusion. Residual connections and dense connections can efficiently combine feature information from lower and higher layers of the neural network. More specifically, residual connections allow gradients (or vectors) to flow through the neural network without crossing non-linear activation functions between layers. In this way, residual connections can skip one or more layers of the neural network. This prevents the vanishing gradient problem in neural networks. Prevention of the vanishing gradient problem improves the training process of neural networks.

[0071] Several application examples of multi-level fusion include object detection and machine translation. However, due to the limited capabilities of cascading, addition, and loop fusion operations, this model lacks the ability to capture multi-level information.

[0072] Some implementations are based on determining the optimal structure for constructing parametric models (such as image models for computer vision applications or language models for NLP applications). To this end, the optimal structure is determined using Neural Architecture Search (NAS) algorithms. Additionally, or alternatively, reinforcement learning and evolutionary learning algorithms can be used for neural architecture search. Some implementations randomly sample the outputs (such as output features) of different layers of the neural network during the training phase to determine the optimal structure. This results in training multiple architectures simultaneously and provides a form of regularization to prevent overfitting of features or parameters in the neural network. The parametric model (i.e., the optimal structure) can be obtained by using the output features of the internal networks of the optimal neural architecture. Such a parametric model can be obtained with lower computational cost because multiple architectures are trained simultaneously.

[0073] Figure 1A schematic block diagram of an artificial intelligence (AI) system 100 according to some embodiments of the present disclosure is shown. The AI ​​system 100 includes an input interface 102, a memory 104 storing a multidimensional neural network 106, a processor 108, and an output interface 110. The multidimensional neural network 106 has a sequence of deep neural networks (DNNs) including an inner DNN and an outer DNN. Each DNN includes a sequence of layers, and corresponding layers of different DNNs have the same parameters. That is, the inner DNN shares the same parameters with the outer DNN. The same parameters are shared through different layers (or paths) between the inner and outer DNNs. Sharing the same parameters prevents an increase in the number of parameters and provides regularization for the parameters of the multidimensional neural network 106. Parameter regularization can prevent overfitting of the parameters. Each DNN is configured to process input data sequentially by the layer sequence along a first dimension of data propagation. The DNNs in the DNN sequence are arranged along a second dimension of data propagation, starting from the inner DNN and continuing to the outer DNN. The DNNs in the DNN sequence are connected such that at least one intermediate output of a layer of one DNN in the DNN sequence is combined with the input of at least one layer of a subsequent DNN. In one exemplary embodiment, each DNN in the multidimensional neural network 106 includes an attention module. Each attention module includes a self-attention subnetwork and a feedforward subnetwork, and also includes residual connections around the self-attention subnetwork and residual connections around the feedforward subnetwork.

[0074] Processor 108 is configured to submit input data to multidimensional neural network 106 to produce the output of an external DNN. In some embodiments, processor 108 is configured to randomly sample the outputs of different layers of multidimensional neural network 106 during the training phase. This results in training multiple architectures for different applications at once and improves the processing time efficiency of AI system 100. In some embodiments, processor 108 is configured to establish connections between one or more pairs of DNN layers and subsequent DNN layers of the multidimensional neural network based on multiple connections. These connections can have different patterns and different types. Different connection patterns connect different layers of adjacent DNNs. Different types of connections include hard connections and soft connections, as described below.

[0075] Output interface 110 is configured to render at least one function of the output of an external DNN. For example, the output function corresponds to a parametric model of an application (such as an NLP application, a computer vision application, or a combination thereof). More specifically, the function can be another DNN that accepts the output of the external DNN and outputs class labels for classification tasks (such as optical character recognition, object recognition, and speaker recognition). Furthermore, the function can be a decoder network that accepts the output of the external DNN and generates word sequences for sentence generation tasks (such as speech recognition, machine translation, and image captioning).

[0076] Figure 2A A block diagram of a multidimensional neural network 106 stored in memory 104 of an AI system 100 according to an exemplary embodiment of the present disclosure is shown. In one embodiment, the multidimensional neural network 106 includes an inner DNN 200 and an outer DNN 210. The inner DNN 200 includes a sequence of layers, such as an input layer 202 and an output layer 208, with one or more intermediate or hidden layers, such as hidden layers 204 and 206, between the input layer 202 and the output layer 208. Similarly, the outer DNN 210 includes a sequence of layers, such as an input layer 212 and an output layer 218, with one or more hidden layers, such as hidden layers 214 and 216, between the input layer 212 and the output layer 218.

[0077] DNNs 200 and 210 include corresponding layers, i.e., layers with the same parameters. For example, layer 202 corresponds to layer 212, layer 204 corresponds to layer 214, layer 206 corresponds to layer 216, and layer 208 corresponds to layer 218. The corresponding layers are arranged in the same order, such that at least some parts of the structures of DNNs 200 and 210 are identical to each other. In this way, parameter variations in the multidimensional neural network 106 are reduced, which increases the flexibility of its structure.

[0078] The inner DNN 200 is configured to process the input data sequentially along the first dimension 220 of data propagation, consisting of layers (i.e., DNN layers 202, 204, and 206). Similarly, the outer DNN 210 is configured to process the input data sequentially along the first dimension 220 of data propagation, consisting of layers (i.e., DNN layers 212, 214, and 216). The inner DNN 200 and the outer DNN 210 are arranged along the second dimension 222 of data propagation.

[0079] The layers of the inner DNN 200 (i.e., input layer 202, hidden layers 204 and 206, and output layer 208) are connected at the input / output level to the layers of the outer DNN 210 (i.e., input layer 212, hidden layers 214 and 216, and output layer 218). In one exemplary embodiment, the layers of the inner DNN 200 have multiple connections to the layers of the outer DNN 210 (i.e., input layer 212, hidden layers 214 and 216, and output layer 218). These multiple connections correspond to... Figure 2A The diagram shows multiple hard connections arranged in pattern 200a (hereinafter referred to as the hard connection pattern). For example, the output of an intermediate layer of the inner DNN 200 (such as hidden layer 204 or hidden layer 206) is added to the input of a layer of the outer DNN 210 (such as hidden layer 214 or hidden layer 216). In some exemplary embodiments, the output of one layer of the inner DNN 200 is added as input to any other layer of the inner DNN 200 via residual connections.

[0080] Different implementations can use different connection patterns 200a to adapt the multidimensional neural network to different applications. For example, in some implementations, the output of a given layer of the inner DNN may contribute only to the input of a unique layer of the outer DNN. In some implementations, the outputs of two given layers in the inner DNN may contribute to the input of the same layer of the outer DNN. For example, all layers of the inner DNN may be connected to all layers of the outer DNN. Such a connection pattern is referred to herein as fully connected, making the multidimensional neural network fully connected. Alternatively, the multidimensional neural network may be partially connected. For example, in a partially connected multidimensional neural network, one or more layers of the inner DNN may be connected to multiple layers of the outer DNN. Furthermore, or alternatively, in a partially connected multidimensional neural network, multiple layers of the inner DNN may be connected to one layer of the outer DNN.

[0081] In addition to different connection patterns between layers of different DNNs in a multidimensional neural network, some implementations also use different types of connections. For example, various implementations use hard connections, soft connections, or a combination thereof. For example, in hard connections, the output of an inner DNN layer is added as a whole to the input of an outer DNN layer. That is, these layers are either connected or not connected. If the layers are connected, the output of one layer is combined with the input of another layer without additional scaling and / or weight multiplication. If the layers are not connected, the output of that layer is not added to the input of other layers. Pattern 200a shows an exemplary pattern of hard connections.

[0082] Therefore, according to the principle of hard connections, the output of an inner DNN layer may or may not contribute to the input of an outer DNN layer. The principle of data propagation based on hard connections differs from the propagation principle between layers of a single DNN. Thus, hard connections allow for the decoupling of data propagation in different directions. In turn, this decoupling allows for the discovery of better hard connection patterns based on the trained DNN parameters, adding additional flexibility to the architecture of multidimensional neural networks.

[0083] In some implementations, during the training of a multidimensional neural network, hard connection patterns are selected from multiple connection patterns. For each selected connection pattern, the corresponding multidimensional neural network is trained. The best-performing multidimensional neural network is then selected from all trained networks. More specifically, the selection of hard connection patterns is based on a search algorithm (e.g., a random search algorithm). This random search algorithm randomly selects a certain number of connection patterns from multiple connections and trains a model for each connection pattern. A model is then selected based on performance metrics on the validation set (e.g., accuracy, F1 score, BLEU score, etc.).

[0084] In some implementations, new connection patterns can be selected to be included in the search algorithm. Predetermined connection patterns can be identified based on scores associated with each pattern. For example, one or more predetermined connection patterns with high scores can be selected as new connection patterns. In some cases, the selected predetermined connection patterns can be operated with minor modifications.

[0085] Alternatively, in soft-connection types, only a portion of the output of one layer is combined with the input of another layer. Specifically, the output of a soft connection between one layer and another is "weighted" before being added to the input of the other layer. The weights of the soft connections may vary for different types of soft connections.

[0086] In one exemplary embodiment, residual connections allow connections in one layer of the inner DNN 200 to jump to other non-adjacent layers of the inner DNN 200. For example, the output of input layer 202 can be added as input to hidden layer 206 by skipping hidden layer 204 based on residual connections. In some exemplary embodiments, the output of input layer 202 triggers an activation function when the output of input layer 202 is added as input to hidden layer 206. Such an activation function may be a rectified linear activation unit (ReLU), which can transform any nonlinearity in the output of input layer 202. Furthermore, in some exemplary embodiments, layers 202 to 208 of the inner DNN 200 and corresponding layers 212 to 218 of the outer DNN 210 share the same parameters (e.g., weight values ​​or feature vectors). Layers 212 to 218 of the outer DNN 210 compute data inputs to provide output 224.

[0087] In another implementation, the multidimensional neural network 106 may have one or more hidden DNNs between the inner DNN 200 and the outer DNN 210, such as Figure 2B As shown in the image.

[0088] Figure 2B A block diagram of a multidimensional neural network 106 stored in memory 104 of an AI system 100 according to another exemplary embodiment of the present disclosure is shown. The multidimensional neural network 106 includes one or more hidden DNNs, such as hidden DNN 224 and hidden DNN 226 between an inner DNN 200 and an outer DNN 210. Figure 2B As shown, each hidden DNN processes the input data sequentially along the first dimension 220 of the data propagation (e.g., Figure 3The input data 302 is arranged along the second dimension 222 of the data propagation from the inner DNN 200 to the outer DNN 210.

[0089] The hidden DNNs (i.e., hidden DNNs 224 and 226) of the multidimensional neural network 106 are connected in the forward direction (i.e., along the second dimension 222). The connections between hidden DNNs (such as DNNs 224 and 226) along the second dimension 222 prevent loop connections or circular connections between layers in the multidimensional neural network 106. Furthermore, the connections along the second dimension 222 can increase the number of hidden DNNs between the inner DNN 200 and the outer DNN 210 to provide accurate output. Increasing the number of hidden DNNs (i.e., hidden DNNs 224 and 226) does not increase the number of parameters, because the same parameters are shared among the inner DNN 200, hidden DNNs 224 and 226, and the outer DNN 210.

[0090] In one exemplary implementation, the inner DNN 200 provides output to any other DNN (e.g., hidden DNN 226) via residual connections. Residual connections allow skipping one or more hidden DNNs (e.g., hidden DNN 224) and adding the output of one layer of the inner DNN 200 to one layer of the hidden DNN 226. For example, the output of the inner DNN 200 can be added as input to the hidden DNN 226 by skipping hidden DNN 224 based on residual connections.

[0091] Figure 3 This diagram illustrates a processing pipeline 300 of an AI system 100 according to some embodiments of the present disclosure for generating the output of a multidimensional neural network 106. Initially, input data 302 is provided to the AI ​​system 100 via an input interface 102. In some cases, the input data 302 may include a training dataset for training the multidimensional neural network 106. Furthermore, the input data 302 may vary depending on the application type. For example, the input data 302 may include image data or video data for image processing or computer vision applications. For NLP applications, the input data 302 may correspond to speech data or text data. The processing pipeline 300 includes operations 304 to 310 performed by the AI ​​system 100.

[0092] At operation 304, processor 108 obtains input data 302 from input interface 102. At operation 306, processor 108 submits input data 302 to multidimensional neural network 106. At operation 308, multidimensional neural network 106 processes input data 302. In some embodiments, multidimensional neural network 106 processes input data 302 to provide the output of one of DNNs 200, 224, 226, and 210 as input to subsequent DNNs of DNNs 200, 224, 226, and 210. In some exemplary embodiments, input data 302 may be processed using a predetermined connection pattern. In some cases, the predetermined connection pattern may correspond to a hard connection pattern optimized during random search during the training of multidimensional neural network 106. In some other cases, the predetermined connection pattern may correspond to a soft connection pattern that is learned simultaneously with the parameters of multidimensional neural network 106 during the training of multidimensional neural network 106. At operation 310, multidimensional neural network 106 renders a function of the output of external DNN 210. The output is provided as output data 312 via output interface 110. In some exemplary embodiments, the function of the output of the external DNN 210 includes an encoded form of the input data to be generated as the output of the AI ​​system 100 via output interface 110. Additionally, the generated output can be displayed through a graphical representation or visualization of the output interface. In one exemplary embodiment, the input data may be processed by a decoder to produce decoded data as output.

[0093] Exemplary Implementation

[0094] Figure 4A An exemplary schematic diagram 400 of an encoder in a multidimensional neural network 106 according to some exemplary embodiments of the present disclosure is shown. In some embodiments, the multidimensional neural network 106 forms an encoder 402 in an encoder-decoder architecture of a neural network (e.g., a neural network for machine translation of one language into another). In one illustrative example scenario, the AI ​​system 100 trains the multidimensional neural network 106 for machine translation. For example, the multidimensional neural network 106 is trained using a training dataset. The training dataset corresponds to language pairs, such as English-German or English-French language pairs. The training dataset may include multiple sentence pairs, such as 4.5 million sentence pairs. During training, the multidimensional neural network 106 generates a symbol dictionary (e.g., 32,000 symbols) based on a byte-pair encoding (BPE) algorithm. The multidimensional neural network 106 extracts sentences of approximately the same length to group or batch them.

[0095] Alternatively, the AI ​​system 100 may determine the optimal connection pattern from multiple connections. In some implementations where the connection pattern is a hard connection pattern, the optimal connection pattern may be determined based on a random search algorithm. This random search algorithm randomly selects a certain number of connection patterns from multiple connections. The pattern is selected based on a performance metric of validation data prepared for the target application. For example, the performance metric could be the recognition accuracy for a classification application and the F1 or BLEU score for a machine translation application.

[0096] Figure 4B A block diagram of a multithreaded converter (MPT) 402 according to an exemplary embodiment of the present disclosure is shown, wherein the inner network layers of the MPT have a hard-connected mode or a soft-connected mode between the outer network layers of the MPT. In this exemplary MPT, the layers of the multidimensional neural network 106 are formed by sub-networks with attention module architectures. The MPT 402 is fully connected, meaning that the outputs of all attention modules of the DNN 408 are added to the inputs of all attention modules of the DNN 410.

[0097] The connection mode of the MPT 402 can be formed by hard connections and / or soft connections. (See reference) Figure 4C This further explains how to determine the optimal connection pattern in hard-connected architectures. In some implementations where connections are soft connections, the optimal connection pattern can be determined by simultaneously optimizing the weights of the soft connections and other parameters of the neural network. (See references.) Figure 4D This section further explains how to determine the optimal soft connection mode within the soft connection pattern.

[0098] In some implementations, MPT 402 forms an encoder for machine translation. MPT 402 includes an internal network and an external network. The internal network corresponds to an internal DNN 200, and the external network corresponds to an external DNN 210. Similar to the sharing of the same parameters between the internal DNN 200 and the external DNN 210, the internal and external networks share the same parameters. The output of a layer in the internal network is added via a residual connection, which allows the output to be added to the input of a layer in the external network within MPT 402. Furthermore, in some implementations, during training, MPT 402 can randomly sample features from the last layer (i.e., the output layer) of either the internal or external network for the application (e.g., machine translation). In some implementations, MPT 402 can use the output of the external DNN 210 for applications, such as machine translation applications.

[0099] For machine translation, source statement 406A (e.g., an English statement) is provided to MPT 402 as input data (e.g., input data 302) via input interface 102. For example, the source statement may be provided as speech input, text input, or a combination thereof. MPT 402 translates source statement 406A into target statement 406B (e.g., a German statement). In one exemplary embodiment, input interface 102 tags the input statement to form source statement 406A, which is sent to layer 202 of internal DNN 200 and layer 212 of external DNN 210. The input statement may be tagged based on byte-pair encoding (BPE) and further transformed into a vector representation by a word embedding layer. The vector representation may include an LC-dimensional vector, where L corresponds to the statement length, i.e., the number of tags in the statement, and C corresponds to the word embedding dimension. Furthermore, the position of each word in source statement 406A is encoded into a position embedding space and added to the vector representation to form the final source statement sequence 406A, which is used as input to MPT 402. Then, MPT 402 computes an encoding from the input, which is obtained as the output 224 of layer 218 of the external DNN 210. In one embodiment, the encoding computed by MPT 402 is provided to decoder 404. Decoder 404 computes the target statement 406B from the encoding and provides the target statement as output via output interface 110. The target statement 406B can be provided as speech output, text output, or a combination thereof.

[0100] Figure 4B The diagram illustrates a block diagram of an exemplary embodiment of an MPT 402 according to the present disclosure, which has a hard-connected pattern between layers of an internal network 408 (e.g., an internal DNN 200) and layers of an external network 410 (e.g., an external DNN 210) for a machine translation application. The internal network 408 replicates the external network 410 to share the same parameters. For example, the internal network 408 shares the same weights as the external network 410. The internal network 408 and the external network 410 correspond to those referenced above. Figure 1 and Figure 2AThe described DNN. Each network 408 and 410 includes an attention module layer, which are sequentially connected along a dimension (e.g., a first dimension 220). For example, the inner network 408 includes attention modules 408A, 408B, 408C, and 408D connected in sequence. Similarly, the outer network 410 includes attention modules 410A, 410B, 410C, and 410D connected in sequence. Additionally, the output of the inner network 408 propagates forward (e.g., a second dimension 222). The output of each of the attention modules 408A through 408D in the inner network 408 is provided as input to each of the attention modules 410A through 410D in the outer network 410. The output of each of the attention modules 408A through 408D is added together with the original input of each of the attention modules 410A through 410D. The overall output of the internal network 408 is provided as the first thread output, and the overall output of the external network 410 is provided as the second thread output.

[0101] like Figure 4B As shown, each output of attention modules 408A to 408D is added to attention modules 410A to 410D via multiple connections. This allows for iterative refinement of features without increasing the number of parameters, thereby improving the performance of MPT 402. In some implementations, MPT 402 may be trained for machine translation using only the output of external network 410. In other implementations, two network models, namely internal network 408 and external network 410, are trained for machine translation in a single training session by using either the output of internal network 408 or the output of external network 410. This allows for dynamic selection of the parameter model for various applications based on computational requirements. In one case, computational requirements can be manually determined to select the parameter model. In other cases, computational requirements can be automatically determined, selecting the parameter model in an automatic manner.

[0102] Figure 4CA block diagram is shown of an optimal model 412 of an exemplary embodiment of MPT 402 according to this disclosure, which has a partially connected pattern with hard connections selected for a machine translation application. The optimal model 412 corresponds to the optimal connected pattern determined from a fully connected search space based on a randomized neural architecture search algorithm. This neural architecture search is based on a randomized search algorithm. In one exemplary scheme, consider a connected search space with all connected patterns, where the output of a layer in the inner network 408 is added to the input of a layer in the outer network 410, with the constraint that no two outputs can be added to the same input. This reduces the search space to a set of permutations on {0,...,N-1}, where N represents the number of layers in the inner network 408, such that the search space size is N!. Without this constraint, the search space grows exponentially to N!. N .

[0103] The i-th hard MPT architecture can be used via the i-th permutation using the image [τ0] corresponding to the sequence [0, ..., N]. (i) ,…,τ N (i) The exponential sequence is used to represent the i-th hard MPT architecture. In the i-th hard MPT architecture, the layer τ in the internal network 408... k (i) The output is added to the input of the k-th attention module in the outer network 410. For example, for the inner network 408 with N hard = 6 attention modules 408A to 408F, the optimal model 412 of MPT 402 is obtained using the connection pattern [0,4,1,5,2,3], where the output of the 0th inner layer is added to the input of the 0th outer layer, the output of the 4th inner layer is added to the input of the 1st outer layer, the output of the 1st inner layer is added to the input of the 2nd outer layer, and so on. The connection pattern [0,1,2,3,4,5] represents the default architecture of MPT 402 during setup.

[0104] The outputs of one or more attention modules 408A to 408F in the internal network 408 are combined with the input of one of the attention modules 410A to 410F in the external network 410. For example, the input of attention module 410A is combined with the output of attention module 408A. The connection between attention modules 408A to 408F and attention modules 410A to 410F can be configured from any output of the intermediate or output layer of the internal network 408 to any input of the input layer or intermediate layer of the external network 410.

[0105] Figure 4DA block diagram of MPT 418 according to some other exemplary embodiments of the present disclosure is shown. In some exemplary embodiments, MPT 418 utilizes soft connections to scale the output of the internal network 408 using weights before adding the scaled output to the connection layer of the external network 410. For example, attention module 408A may be connected to each of the attention modules 410A to 410D of the external network 410 using weights 418d w1, w2, w3, and w4. In an illustrative example scheme, weight w1 (e.g., 0.2) may be used to scale the input of attention module 410A, weight w2 (e.g., 0.4) may be used to scale the input of attention module 410B, weight w3 (e.g., 0.6) may be used to scale the input of attention module 410C, and weight w4 (e.g., 0.8) may be used to scale the input of attention module 410D.

[0106] Different implementations can use the weights 418d directly or indirectly. For example, in one implementation, each soft connection has an associated weight, and this implementation directly uses the weights to scale the contribution of the inner layer to the corresponding outer layer. Therefore, the weight of each soft connection represents its strength. In another implementation, the weight w of each soft connection between layer j of the inner DNN and layer k of the outer DNN... kj Instead of being used directly to determine connection strength, it is fed back to a function (such as the softmax function), so that the strength of each connection depends on the weights of other connections.

[0107] In some cases, MPT 418 is fully connected using soft connections. The weights are learned during the training of the residual connections between each layer pair between attention modules 408A-408D and attention modules 410A-410D. For example, by... The output of the k-th attention module in the external network 410 can be calculated as:

[0108]

[0109] Where AttModule(.) represents an attention module (e.g., attention modules 410A to 410D), which includes a self-attention network and a feedforward neural network. It is the output of the j-th inner layer, and α kj This represents the weight of the connection from the j-th inner layer to the k-th outer layer. The connection weights utilize a learnable parameter w. kj α is calculated using softmax. kj =exp(w kj ) / ∑ j exp(w kj ), to force 0≤α kj ≤1 and ∑j (α kj ) = 1.

[0110] In some exemplary implementations, the MPT can be trained during training based on the random minimization of the input sequence S and the target sequence T. During training, the decoder (e.g., Figure 4C The decoder 416) is applied to the output of the external network 410. To obtain the objective function Decoder 416 corresponds to Figure 4A Decoder 404. In some exemplary embodiments, the internal network 408 and the external network 410 can be optimized so that the outputs of the internal network 408 and the external network 410 can be used for downstream tasks, such as machine translation. The internal network 408 and the external network 410 can be optimized by minimizing the output L(S) of the internal network 408. N-1 The output of ,T) and external network 410 The sum of the outputs of the inner network 408 or the outer network 410 can be used for optimization. In some cases, the output of the inner network 408 or the outer network 410 can be randomly minimized for optimization. Furthermore, the sum of the outputs of the inner network 408 can be kept consistent with the output of the outer network 410. This can be useful in applications involving switching between high-computational-cost and low-computational-cost mechanisms without increasing the number of parameters.

[0111] Each layer of the attention modules 408A to 408D and attention modules 410A to 410D in the corresponding inner network 408 and outer network 410 includes a self-attention network and a feedforward neural network, which will refer to Figure 5 Further description.

[0112] Figure 5 A block diagram 500 illustrates an attention module 502 in each layer of each DNN in a multidimensional neural network 106 according to some embodiments of the present disclosure. For example, the architecture of attention module 502 corresponds to the architectures of attention modules 408A to 408D and attention modules 410A to 410D. Attention module 502 includes a self-attention subnetwork 504 and a feedforward neural subnetwork (FFN) 506, the feedforward neural subnetwork including residual connections therebetween. The self-attention subnetwork 504 learns information relations in a pairwise manner. For example, the self-attention subnetwork 504 learns relations about words in NLP applications (such as machine translation).

[0113] In one exemplary scheme, the self-attention subnetwork 504 receives input, for example, from S∈R. L×C The statement S is represented by a self-attention subnetwork 504 that translates S into a bond (Si) through a linear transformation. k ), query (S q Sum of values ​​(S)v ). By using S k and S q The attention value between words is calculated using self-attention aggregation of information from other words for each word in S. For key K, query Q, and value V, the attention value can be calculated using formula (2):

[0114]

[0115] Attention values ​​are determined by the feature dimension d. k Square root modulation. After aggregating information from other words in the self-attention network 504, the FFN subnetwork 506 combines the information in a position-wise manner. In some embodiments, the self-attention subnetwork 504 corresponds to multi-head attention. The stacking of this self-attention subnetwork 504 and FFN subnetwork 506 constitutes the attention module 502, which processes the input S as follows:

[0116] S mid =Attention(S) q ,S k ,S v (3)

[0117] S out =FFN(S mid (4)

[0118] in,

[0119] S mid These are features derived from intermediate layers (e.g., one of attention modules 408B to 408E or one of attention modules 410B to 410E) within each of the inner network 408 and the outer network 410; and

[0120] S out The output is provided by the FFN subnet 506.

[0121] In some implementations, during the decoding phase of the output encoding, self-attention is performed on the embedding representation T of each target statement, followed by common attention and FFN. The decoding phase can be represented as follows, where SA represents self-attention:

[0122]

[0123]

[0124] In an encoder-decoder architecture, the encoder and decoder share a word embedding layer. This is done to obtain the representation of the next word (i.e., in decoder 404). After that, A linear transformation and softmax operation are applied to obtain the probability of the next word. Then, the cross-entropy loss based on the probability of the next word is used to train all the connected networks using the backpropagation technique of the ANN.

[0125] Figure 6A A block diagram illustrating a connection pattern 600A between two layers of two DNNs according to an exemplary embodiment of the present disclosure, which combines the final output of a DNN layer with the input of a subsequent DNN layer. The attention module 608 of the inner network (e.g., inner network 408) includes a self-attention subnetwork 602A (e.g., self-attention subnetwork 504) and a feedforward neural subnetwork 602B (e.g., feedforward neural subnetwork 506), as... Figure 6A As shown in the diagram. Attention module 608 is representative of each attention module 408A through 408D. Self-attention subnetwork 602A is connected to FFN subnetwork 602B via a sublayer (e.g., addition and regularization sublayer 604A). Addition and regularization sublayer 604A is a layer regularization step combined with residual connections. In a similar manner, attention module 610 of the outer network (e.g., outer network 410) includes self-attention subnetwork 606A (e.g., self-attention subnetwork 504) and feedforward neural subnetwork 606B (e.g., feedforward neural subnetwork 506). Attention module 610 is representative of each attention module 410A through 410D. Self-attention subnetwork 606A provides output to addition and regularization sublayer 614A, and FFN subnetwork 606B provides output to addition and regularization sublayer 614B. The input to the self-attention subnetwork 606A is calculated as the sum of the output 618A of the feedforward neural subnetwork 602B after the addition and regularization sublayer 604B and the input 612 of the attention module 610.

[0126] In one exemplary embodiment, the final output 618A of attention module 608 is added to the input of attention module 610 prior to the residual connection associated with the self-attention sub-network 606A of attention module 610. The residual connection of self-attention sub-network 606A includes output 618A, i.e., the sum of output 618A and input 612 is added to the output of self-attention sub-network 606A in the addition and regularization sub-layer 614A.

[0127] Figure 6BA block diagram illustrating a connection pattern 600B between two layers of two DNNs according to another exemplary embodiment of the present disclosure, which combines the final output of a DNN layer with the input of a subsequent DNN layer. In one exemplary embodiment, the final output 618A of attention module 608 is added to the input of attention module 610 after a residual connection associated with a self-attention subnetwork 606A of attention module 610. The input of self-attention subnetwork 606A is computed as the sum of the output 618A of feedforward neural subnetwork 602B after the addition and regularization sublayer 604B and the input 612 of attention module 610. The residual connection of self-attention subnetwork 606A does not include output 618A; that is, only input 612 is added to the output of self-attention subnetwork 606A in the addition and regularization sublayer 614A.

[0128] Figure 6C A block diagram illustrating a connection pattern 600C between two layers of two DNNs according to another exemplary embodiment of the present disclosure, which combines the final output of a DNN layer with the input of a subsequent DNN layer. In one exemplary embodiment, the intermediate output 618B of the self-attention subnetwork 602A after the addition and regularization sublayer 604A of the attention module 608 is added to the input of the attention module 610 before a residual connection associated with the self-attention subnetwork 606A of the attention module 610 of the external network 410. The input of the self-attention subnetwork 606A is calculated as the sum of the intermediate output 618B of the self-attention subnetwork 602A after the addition and regularization sublayer 604A and the original input 612 of the attention module 610. The residual connection of the self-attention subnetwork 606A includes the intermediate output 618B, i.e., the sum of the intermediate output 618B and the original input 612, added to the output of the self-attention subnetwork 606A in the addition and regularization sublayer 614A.

[0129] Figure 6DA block diagram illustrating a connection pattern 600B between two layers of two DNNs according to another exemplary embodiment of the present disclosure, which combines the intermediate output of a DNN layer with the input of a subsequent DNN layer. In one exemplary embodiment, the intermediate output 618B of the self-attention subnetwork 602A after the addition and regularization sublayer 604A of attention module 608 is added to the input 612 of attention module 610 after a residual connection associated with the self-attention subnetwork 606A of attention module 610. The input of self-attention subnetwork 606A is calculated as the sum of the intermediate output 618B of self-attention subnetwork 602A after the addition and regularization sublayer 604A and the input 612 of attention module 610. The residual connection of self-attention subnetwork 606A does not include the intermediate output 618B; that is, only the original input 612 is added to the output of self-attention subnetwork 606A in the addition and regularization sublayer 614A.

[0130] Figure 7 Table 700 shows ablation studies of different models of MPT with various connection modes according to embodiments of the present disclosure. Table 700 shows the evaluation of different MPT models based on the BLEU score index (a commonly used index in machine translation). Different models include models such as: base converter 702 (as reported in the article by Vaswani et al.), base converter 718 (the applicant's reimplementation of the base converter by Vaswani et al.), MPT model 704 with hard connection modes [0,1,2,3,4,5], MPT model 706 based on connection mode 600D, MPT model 708 based on connection mode 600C, MPT model 710 based on connection mode 600B, MPT model 712 based on connection mode 600A, worst-case hard-connected search MPT model 714, best-case hard-connected search MPT model 412, and MPT model 720 with soft connections (e.g., MPT model 418). The average performance 716 of all hard-connected search MPT models is also shown.

[0131] Table 700 shows that combining information before initiating residual connections results in better performance. The performance difference between the best model (i.e., MPT converter 412) and the worst performing model (i.e., base converter 702) is 0.8, with MPT converter 412 achieving 28.4 and base converter 702 achieving 27.3. Analysis of the different models in Table 700 was conducted to identify factors affecting the MPT performance of different implementations. Furthermore, based on these factors (such as network search), performance tends to improve when features from deeper layers of the inner network 408 are added to features in the outer network 410, except when features from the last layer of the inner network 408 are added to features in the first layer (i.e., layer 410A of the outer network 410). Additionally, performance also improves when features from shallower layers of the inner network 408 are directly linked to deeper layers of the outer network 410.

[0132] The MPT 402 with hard connections and the MPT 418 with soft connections can achieve better performance than evolution converters, which will be discussed later. Figure 8A This will be described.

[0133] Figure 8A Table 800 shows a comparison between an exemplary embodiment of the present disclosure, MPT 402 with hard connections and MPT 418 with soft connections, and prior art methods (collectively referred to as machine translation method 802). The comparison is performed using datasets (e.g., EN-DE and EN-FR translation datasets). Method 802 includes basic transformers (BTs), large transformers, evolutionary transformers, contextual max pooling, contextual attention, deep contextual recurrent neural networks (RNNs), linear combination + BT, dynamic combination + BT, dynamic routing + BT, and EM routing + BT.

[0134] Evolutionary Transformers perform architecture search over a larger search space using evolutionary algorithms. Architecture search can be performed based on the size of the self-attention head, the number of layers, different cascades between convolutions and self-attention networks, dense residual fusion, and the architecture search is performed jointly on the encoder and decoder of the encoder-decoder architecture neural network. Evolutionary Transformers can also include a larger search space than MPT 402. MPT 402 with hard-connected patterns performs a random search over a finite search space. The reduced search space allows MPT 402 to achieve better performance than Evolutionary Transformers. MPT 418 can estimate the optimal connection pattern without random search, which provides better performance than Evolutionary Transformers. As shown in Table 800, with fewer parameters (i.e., 61.2 for EN-DE and 111.4 for EN-FR), MPT 402 achieves a BLEU score of 28.4 on the EN-DE dataset and 41.8 on the EN-FR dataset. Similarly, with a smaller number of parameters (i.e., 61.2 for EN-DE and 111.4 for EN-FR), MPT 418 achieves a BLEU score of 28.4 on the EN-DE dataset and 41.6 on the EN-FR dataset. However, with a larger number of parameters (i.e., 64.1 for EN-DE and 221.2 for EN-FR), the evolutionary transformer achieves a BLEU score of 28.2 on the EN-DE dataset and 41.3 on the EN-FR dataset.

[0135] The Context Max Pooling Transformer combines features from all layers in the encoder network based on addition, loop fusion, concatenation, or attention operators. Furthermore, operators like concatenation and loop fusion can significantly increase the number of parameters. For example, the Context Max Pooling Transformer has 106.9 million parameters, which is more than the MPT 402. Therefore, the MPT 402 can achieve much better performance than the Context Max Pooling Transformer with a smaller number of parameters. Similarly, Dynamic Combination with Bitwise Barrier (BT) and Dynamic Routing with BT share the same concept as the Context Max Pooling Transformer. Dynamic Combination with BT and Dynamic Routing with BT also utilize a multi-layer information fusion mechanism based on the Expectation-Maximization (EM) algorithm. However, Dynamic Combination with BT and Dynamic Routing with BT increase the number of parameters, such as 113.2 million and 125.8 million respectively.

[0136] It is worth noting that MPT 402 and 418 can also be compared to deeper converters, which have more layers but only one dimension, i.e., no DNN sequence and no data propagation along a second dimension. For example, a “deeper” converter with 12 layers performs roughly as well as an MPT with 6 layers, but the deeper converter uses more parameters and therefore has more memory.

[0137] Figure 8B This illustrates the advantages of a multi-threaded converter 806 equipped with a multi-dimensional neural network 106 having multiple DNNs 810 and 812 compared to other converter structures. For example... Figure 8B As shown and combined Figure 7 and Figure 8A It is noted that the MPT according to various embodiments can (outperform or at least match) the performance of a larger converter 808, for example, with more layers and / or more parameters per layer, resulting in more units and more attention heads.

[0138] Figure 9 A flowchart 900 illustrates a method for generating the output of a multidimensional neural network 106 according to some embodiments of the present disclosure. At block 902, input data from the AI ​​system 100 is received via an input interface (e.g., input interface 102).

[0139] At box 904, input data is submitted to a multidimensional neural network 106, which has a sequence of deep neural networks (DNNs) (including an inner DNN and an outer DNN). Each DNN includes a sequence of layers, with corresponding layers of different DNNs having the same parameters. Each DNN is configured to process the input data sequentially along a first dimension of data propagation by the layer sequence. The DNNs in the DNN sequence are arranged along a second dimension of data propagation, starting from the inner DNN and continuing to the outer DNN. The DNNs in the DNN sequence are connected such that at least one output of an intermediate or final layer of a DNN is combined with the input of at least one layer of a subsequent DNN in the DNN sequence, as described above. Figure 1 as well as Figure 2A and Figure 2B As described in the description.

[0140] At box 906, the output of the external DNN is generated. At box 908, at least one function that renders the output of the external DNN is executed. The output of the external DNN is rendered via output interface 110.

[0141] Figure 10 A block diagram of an AI system 1000 according to some embodiments of the present disclosure is shown. The AI ​​system 1000 corresponds to... Figure 1AI system 1000 includes an input interface 1002, a processor 1004, a memory 1006, a network interface controller (NIC) 1012, an output interface 1018, and a storage device 1022. The memory 1006 is configured to store a multidimensional neural network 1008. The multidimensional neural network 1008 has a sequence of deep neural networks (DNNs), including an inner DNN (e.g., inner DNN 200) and an outer DNN (e.g., outer DNN 210). Each DNN (i.e., inner DNN 200 and outer DNN 210) is configured to process input data, i.e., input data 1016, sequentially by a sequence of layers (e.g., layers 202 to 208) along a first dimension of data propagation (e.g., first dimension 220). The DNNs (e.g., inner DNN 200 and outer DNN 210) are arranged along a second dimension of data propagation (e.g., second dimension 222) from inner DNN 200 to outer DNN 210. Each layer of the DNNs in the DNN sequence (e.g., layers 202 to 208) is connected, such that at least one intermediate output of a layer of a DNN is combined with the input of at least one layer of a subsequent DNN in the DNN sequence (see reference). Figure 2A and Figure 2B In some other embodiments, the multidimensional neural network 1008 has at least one hidden DNN (such as hidden DNNs 224 and 226) arranged along the second dimension 222 between the inner DNN 200 and the outer DNN 210. In some embodiments, the multidimensional neural network 1008 forms an encoder (e.g., MPT 402) in an encoder-decoder architecture of a neural network. In some other embodiments, the multidimensional neural network 1008 with a soft-connected mode forms an encoder (e.g., MPT 418). The encoder provides the output of the outer DNN 210, which corresponds to the encoded form of the input data 1016. The encoded form of the input data 1016 is processed by a decoder (e.g., decoder 416) to produce the output of the AI ​​system 1000. Additionally, each layer of each DNN in the multidimensional neural network 1008 includes an attention module (e.g., attention module 502). Each attention module includes a self-attention subnetwork (e.g., self-attention subnetwork 504) and a feedforward subnetwork (e.g., feedforward subnetwork 506).

[0142] Input interface 1002 is configured to accept input data 1016. In some embodiments, AI system 1000 receives input data 1016 via network 1014 using NIC 1012. In some cases, input data 1016 may be online data received via network 1014. In other cases, input data 1016 may be recorded data stored in storage device 1022. In some embodiments, storage device 1022 is configured to store a training dataset for training multidimensional neural network 1008.

[0143] Processor 1004 is configured to submit input data 1016 to multidimensional neural network 1008 to produce the output of external DNN 210. At least one function provided via output interface 1018 is rendered from the output of external DNN 210. Output interface 1018 is further connected to output device 1020. Some embodiments of output device 1020 include, but are not limited to, monitors, display screens, and projectors.

[0144] Figure 11A An environment 1100 for machine translation applications according to some embodiments of the present disclosure is illustrated. Environment 1100 is depicted as including machine translation devices, such as machine translation devices 1104A, 1104B, and 1104C distributed at different remote locations. Each of the machine translation devices 1104A, 1104B, and 1104C can be operated by corresponding operators 1102A, 1102B, and 1102C, who may speak different languages. These machine translation devices 1104A, 1104B, and 1104C can be interconnected via network 1014. Each of the machine translation devices 1104A, 1104B, and 1104C includes an AI system 1000. The AI ​​system 1000 may have training datasets corresponding to different language pairs (e.g., EN-DE, EN-FR). Language pairs used for machine translation can be combinations of different languages ​​(such as English, French, Spanish, German, Italian, Chinese, Hindi, Arabic, Portuguese, Indonesian, Korean, Russian, Japanese, etc.). In some cases, the training dataset may include multiple sentence pairs, such as 4.5 million sentence pairs. In some exemplary embodiments, each of the machine translation devices 1104A, 1104B, and 1104C may include a language detector (not shown in the original text). Figure 11A (As shown in the figure), it is used to detect each language of operators 1102A, 1102B, and 1102C. In addition or alternatively, the AI ​​system 1000 may include a speech-to-text conversion program (not shown) stored in memory 1006 for translating speech input into text output.

[0145] Each of the machine translation devices 1104A, 1104B, and 1104C may include a corresponding interface controller 1106A, interface controller 1106B, and interface controller 1106C. For example, interface controllers 1106A, 1106B, and 1106C may be located in a NIC 1012 connected to the display, speaker, and microphone of the machine translation devices 1104A, 1104B, and 1104C. Interface controllers 1106A, 1106B, and 1106C may be configured to convert the speech signals of the corresponding operators (i.e., operators 1102A, 1102B, and 1102C) received as input data 1016 from network 1014. Network 1014 may be the Internet, a wired communication network, a wireless communication network, or a combination of at least two of these.

[0146] Input data 1016 is processed by each of machine translation devices 1104A, 1104B, and 1104C. The processed input data 1016 is translated into the desired language by the corresponding machine translation devices 1104A, 1104B, and 1104C. The translated speech is provided as output to the corresponding operators 1102A, 1102B, and 1102C. For example, operator 1102A sends an English speech signal to operator 1102B using machine translation device 1104A. The English speech is received by machine translation device 1104B. Machine translation device 1104B translates the English into German speech. The translated speech is provided to operator 1102B. Furthermore, in some exemplary embodiments, machine translation devices 1104A, 1104B, and 1104C can store / record the dialogue between operators 1102A, 1102B, and 1102C to a storage unit, such as storage device 1022. The dialogue can be stored as audio data or text data using a computer-executable voice-text program stored in memory 1006 or storage device 1022.

[0147] In this way, operators speaking different languages ​​in different locations can communicate effectively using machine translation devices equipped with an AI system 1000. This communication enables operators to perform collaborative operations, such as... Figure 11B As shown and described.

[0148] Figure 11B It is used according to some embodiments of this disclosure. Figure 11AThe representation 1108 is of a collaborative operating system 1110 for machine translation applications. The collaborative operating system 1110 can be deployed as part of a product assembly / manufacturing production line. The collaborative operating system 1110 may include a voice-to-text program (computer-executable voice-to-text program), an AI system 1000 with an MPT 402 architecture, a NIC 1012 connected to a display 1112, a camera, a speaker, and input devices (microphone / pointing devices) via a network 1014. In this case, the network 1014 can be a wired network, a wireless network, or the Internet.

[0149] Some implementations are based on the understanding that when operator 1114 speaks a different language than other operators (e.g., operators 1102A, 1102B, and 1102C working in a single facility in a different country or on a manufacturing line built in different facilities), the cooperative operating system 1110 can provide a process data format based on a predetermined language for maintaining / recording the entire process data of the manufacturing line. In this case, even if operators 1102A, 1102B, 1102C, and 1114 use different instruction languages, the process data format can be recorded in their individual languages.

[0150] The NIC 1012 of the AI ​​system 1000 can be configured to communicate with a manipulator (e.g., robot 1116) via a network 1014. The robot 1116 may include a manipulator controller 1118 and a sub-manipulator 1120 connected to a manipulator status detector 1122, wherein the sub-manipulator 1120 is configured to assemble a workpiece 1124 for manufacturing components of a product or finalizing the product. Additionally, the NIC 1012 can be connected to an object detector 1126 via the network 1014. The object detector 1126 can be arranged to detect the status of the workpiece 1124, the sub-manipulator 1120, and the manipulator status detector 1122 connected to the manipulator controller 1118 arranged in the robot 1116. The manipulator status detector 1122 detects a manipulator status signal (S) and transmits it to the manipulator controller 1118. The manipulator controller 1118 then provides process flow or instructions based on the manipulator status signal (S).

[0151] Display 1112 can show a process flow or instructions representing the process steps for assembling a product based on a (pre-designed) manufacturing method. This manufacturing method can be received via network 1014 and stored in memory 1006 or storage device 1022. For example, when operator 1114 inspects the condition of assembled parts or assembled products (while simultaneously performing quality control processing according to a format (e.g., process record format), audio input can be provided via the microphone of cooperative operating system 1110 to record the quality inspection. The quality inspection can be based on the product manufacturing process and product specifications that can be displayed on display 1112. Operator 1114 can also provide instructions to robot 1116 to operate the product assembly line. Using a voice-text program stored in memory 1006 or storage device 1022, cooperative operating system 1108 can use a voice-text program to store the results confirmed by operator 1114 in memory 1006 or storage device 1022 as text data. This result can be stored along with a timestamp and an item number assigned to each assembled part or assembled product for manufacturing product records. Furthermore, the cooperative operating system 1108 can transmit records to the manufacturing central computer via network 1014. Figure 11B (not shown in the image), thus integrating the entire process data of the assembly line to maintain / record product quality.

[0152] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an advantageous description of implementing one or more exemplary embodiments. Various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the subject matter set forth in the appended claims.

[0153] Specific details are set forth in the following description to provide a full understanding of the embodiments. However, it will be understood by those skilled in the art that embodiments of the invention can be practiced without these specific details. For example, processes, techniques, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the various figures denote the same elements.

[0154] Furthermore, various implementations can be described as processes, depicted as flowcharts, data flow diagrams, data flow graphs, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or concurrently. Moreover, the order of operations can be rearranged. A process may terminate upon completion of its operations, but there may be additional steps not discussed or included in the diagram. Furthermore, not all operations can occur in all implementations within any particularly described process. A process can correspond to a method, function, program, subroutine, subroutine, etc. When a process corresponds to a function, the termination of that function can correspond to the function returning from a calling function or the main function.

[0155] Furthermore, implementations of the disclosed subject matter can be carried out, at least in part, manually or automatically. Manual or automatic implementation can be performed, or at least assisted in, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments performing the necessary tasks can be stored in a machine-readable medium. One or more processors can perform the necessary tasks.

[0156] The various methods or processes outlined in this article can be coded as software that can execute on one or more processors employing various operating systems or platforms. Furthermore, such software can be written using any of a large number of suitable programming languages ​​and / or programming or scripting tools, and can be compiled into executable machine language code or intermediate code for execution on a framework or virtual machine. Typically, the functionality of program modules can be combined or distributed as needed in various implementations.

[0157] The embodiments of this disclosure can be embodied as methods, of which embodiments have been provided. The actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which the order of action execution differs from that illustrated; this may include actions performed simultaneously, even if shown sequentially in the illustrative embodiments. Furthermore, the use of ordinal terms such as "first" and "second" in the claims to modify the claim elements themselves does not imply any priority, hierarchy, or order of one claim element relative to another, nor does it imply a temporal order of the actions of the method; rather, it serves merely as a label to distinguish one claim element with a certain name from another element with the same name (but using ordinal terms) to differentiate these claim elements.

[0158] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of this disclosure. Therefore, the appended claims cover all such changes and modifications within the true spirit and scope of this disclosure.

Claims

1. An audio processing system, the audio processing system comprising a computer-based artificial intelligence (AI) system, the AI ​​system comprising: An input interface configured to accept input data, wherein the input data includes audio signals, and the audio signals include spoken words; A memory configured to store a multidimensional neural network having a sequence of deep neural networks (DNNs), the DNN sequence including an inner DNN and an outer DNN, each DNN including a sequence of layers, the layer sequence including corresponding layers of different DNNs having the same parameters, the parameters being weight values, each DNN being configured to process the input data sequentially by the layer sequence along a first dimension of data propagation, the DNNs in the DNN sequence being arranged along a second dimension of data propagation from the inner DNN to the outer DNN, wherein the DNNs in the DNN sequence are connected such that the output of at least one intermediate layer of a first DNN is combined with the input of at least one earlier layer of a subsequent DNN in the DNN sequence, the subsequent DNN being a DNN in the DNN sequence located after the first DNN, the earlier layer being a layer in the subsequent DNN located before the layer corresponding to the at least one intermediate layer; A processor configured to submit the input data to the multidimensional neural network to generate the output of the external DNN; and An output interface configured to render at least one function of the output of the external DNN, the output function including transcription of the audio signal, such that the audio processing system is an automatic speech recognition (ASR) system.

2. The audio processing system according to claim 1, wherein, The multidimensional neural network has at least one hidden DNN, which is arranged between the inner DNN and the outer DNN along a second dimension of the data propagation.

3. The audio processing system according to claim 1, wherein, One or more layers of the internal DNN are connected to multiple layers of the external DNN, or multiple layers of the internal DNN are connected to one layer of the external DNN, or a combination of both.

4. The audio processing system according to claim 1, wherein, Multiple layers of the inner DNN are connected to layers of the subsequent DNN via multiple soft connections. The outputs of the multiple layers of the inner DNN are scaled based on the weights of the soft connections, and then the scaled outputs are added to the inputs of the layers of the outer DNN.

5. The audio processing system according to claim 4, wherein, The weights of the soft connection are trained simultaneously with the parameters of the multidimensional neural network.

6. The audio processing system according to claim 1, wherein, The multidimensional neural network forms the encoder in the encoder-decoder architecture of the multithreaded converter MPT, such that the output of the external DNN includes the encoding of the input data processed by the decoder to generate the output of the AI ​​system via the output interface, wherein each layer of each DNN in the multidimensional neural network includes an attention module, and each attention module includes a self-attention subnetwork followed by a feedforward subnetwork.

7. The audio processing system according to claim 6, wherein, Each of the DNNs in the multidimensional neural network includes a residual connection before each attention module and between the self-attention subnetwork and the feedforward subnetwork, further configured as one of the following: The connection between two layers of two DNNs in the DNN sequence combines the output of the first DNN layer with the input of the subsequent DNN layer, wherein the output is added to the input of the subsequent DNN layer before the residual connection of the self-attention subnetwork of the attention module of the subsequent DNN layer; The connection between two layers of two DNNs in the DNN sequence combines the output of the first DNN layer of the DNN sequence with the input of the subsequent DNN layer, wherein the output is added to the input of the subsequent DNN layer after the residual connection of the self-attention subnetwork of the attention module of the subsequent DNN layer; The connection between two layers of two DNNs in the DNN sequence combines the intermediate output of the first DNN layer with the input of the subsequent DNN layer, wherein the output is added to the input of the subsequent DNN layer before the residual connection of the self-attention subnetwork of the attention module of the subsequent DNN layer; The connection between two layers of the two DNNs in the DNN sequence combines the intermediate output of the first DNN layer with the input of the subsequent DNN layer, wherein the output is added to the input of the subsequent DNN layer after the residual connection of the self-attention subnetwork of the attention module of the subsequent DNN layer.

8. A method performed by an audio processing system for generating the output of an external deep neural network (DNN) for a multidimensional neural network, wherein, The method uses a processor coupled to stored instructions for implementing the method, wherein the instructions, when executed by the processor, perform at least some steps of the method, the steps including: The system accepts input data via an input interface, wherein the input data includes audio signals, and the audio signals include spoken words; The input data is submitted to the multidimensional neural network, which has a deep neural network (DNN) sequence, including an inner DNN and an outer DNN. Each DNN includes a layer sequence, which includes corresponding layers of different DNNs with the same parameters, the parameters being weight values. Each DNN is configured to process the input data sequentially along a first dimension of data propagation by the layer sequence. The DNNs in the DNN sequence are arranged along a second dimension of data propagation, starting from the inner DNN and ending at the outer DNN. The DNNs in the DNN sequence are connected, and at least one output of an intermediate layer of the first DNN is combined with the input of at least one earlier layer of a subsequent DNN in the DNN sequence. The subsequent DNN is the DNN in the DNN sequence that is located after the first DNN, and the earlier layer is the layer in the subsequent DNN that is located before the layer corresponding to the at least one intermediate layer. Generate the output of the external DNN; and At least one function that renders the output of the external DNN, the function of the output including the transcription of the audio signal, such that the audio processing system is an automatic speech recognition (ASR) system.

9. The method according to claim 8, wherein, One or more layers of the inner DNN are connected to multiple layers of the subsequent DNN via multiple soft connections, the soft connections scaling the output of the one or more layers of the inner DNN and then adding the scaled output to the multiple layers of the outer DNN; and wherein the soft connections scale the output based on weights that are trained simultaneously with the parameters of the multidimensional neural network; or wherein the deep neural network sequence is entirely connected by the soft connections such that all layers of the inner DNN are connected to all layers of the subsequent DNN with different weights, the weights being determined by training simultaneously with the parameters of the multidimensional neural network.

10. The method according to claim 8, further comprising the step of: An encoder is formed in the encoder-decoder architecture of the neural network based on the multidimensional neural network, such that the output of the external DNN includes the encoding of the input data processed by the decoder to produce the output of the AI ​​system. Each layer of each DNN in the multidimensional neural network includes an attention module, and each attention module includes a self-attention subnetwork followed by a feedforward subnetwork. The residual connections in each of the DNNs in the multidimensional neural network are analyzed before each attention module and between the self-attention subnetwork and the feedforward subnetwork of the attention module.

Images