Neural network method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a neural network method and device, electronic equipment and storage medium, the method comprising: processing a training sample through a neural network model to obtain a network loss of the neural network; performing back propagation according to the network loss to obtain an update gradient of a plurality of network nodes of the neural network, and optimizing the back propagation of at least one network node; and obtaining an updated neural network based on the update gradient of the neural network. The neural network training method according to the embodiments of the present disclosure can optimize the back propagation of at least one network node, determine the update gradient by using the optimized network node, reduce the amount of computation in the back propagation, save memory resources, and improve the training efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a neural network method and apparatus, electronic device and storage medium. Background Technology

[0002] Forward propagation and backpropagation are common training processes in neural networks. Forward propagation determines the output of the neural network and the network loss, while backpropagation uses the network loss as feedback to iteratively adjust the parameters of each layer in the neural network. Backpropagation has a similar or even greater computational cost than forward propagation, consuming more memory resources. Summary of the Invention

[0003] This disclosure presents a neural network method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a neural network training method is provided, comprising: processing training samples through a neural network model to obtain a network loss of the neural network model; performing backpropagation based on the network loss to obtain update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes; and obtaining an updated neural network model based on the update gradients of the neural network model.

[0005] The neural network training method according to embodiments of the present disclosure can optimize the backpropagation of at least one network node and use the optimized network node to determine the update gradient, thereby reducing the amount of computation in backpropagation, saving memory resources, and improving training efficiency.

[0006] In one possible implementation, the method further includes: determining at least one network node requiring optimized backpropagation from a plurality of network nodes included in the neural network model based on the backpropagation graph of the neural network model; and / or determining at least one network node requiring optimized backpropagation from the plurality of network nodes based on the number of network nodes connected to each of the plurality of network nodes.

[0007] This method optimizes the backpropagation mechanism of network nodes and improves training efficiency.

[0008] In one possible implementation, the optimized backpropagation of at least one of the plurality of network nodes includes: performing gradient backpropagation on a first network node among the plurality of network nodes using optimized computational code, wherein the number of back input nodes connected to the first network node is 1.

[0009] In one possible implementation, the optimized computation code is the code after removing the accumulation and memory clearing processes.

[0010] In this way, the accumulation process and memory clearing process can be eliminated, and the optimized operation code can be used for backpropagation, reducing redundant calculations and improving training efficiency.

[0011] In one possible implementation, the optimized backpropagation of at least one of the plurality of network nodes includes: performing gradient backpropagation on a second network node among the plurality of network nodes using an optimized caching method, wherein, in the optimized caching method, the reverse output information of the reverse input node of the second network node or the reverse output information of the second network node is not cached.

[0012] In this way, redundant cached data can be removed to save cache space, improve memory resource utilization efficiency, and thus enhance training efficiency.

[0013] In one possible implementation, the number of reverse input nodes and reverse output nodes of the second network node is 1 each.

[0014] In one possible implementation, the method further includes: setting first identification information and second identification information for each of the plurality of network nodes, wherein the first identification information is used to identify whether the output information of the forward input node of the network node is consistent with the buffer information of the reverse output, and the second identification information is used to identify whether the input information of the forward output of the network node is consistent with the buffer information of the reverse input; and determining the second network node among the plurality of network nodes based on the first identification information and the second identification information of each of the plurality of network nodes.

[0015] In this way, first and second identification information can be set to determine network nodes that can be optimized, thereby improving training efficiency.

[0016] The optimized backpropagation for at least one of the plurality of network nodes includes:

[0017] In one possible implementation, gradient backpropagation is performed on at least two of the plurality of network nodes using an optimized backpropagation execution sequence, wherein the gradient backpropagation is performed in parallel on the at least two network nodes in the optimized backpropagation execution sequence.

[0018] In this way, the execution order of multiple output functions in a neural network can be determined by constructing an execution sequence, and output functions with the same index value can be executed in parallel, which can improve the execution efficiency of training and the utilization of processing resources.

[0019] In one possible implementation, the method further includes: obtaining an optimized backpropagation execution sequence based on the data dependencies between the plurality of network nodes.

[0020] In one possible implementation, obtaining an optimized backpropagation execution sequence based on the data dependencies between the plurality of network nodes includes: determining a backpropagation node sequence based on the number of output nodes of the network nodes and the data dependencies of the back output nodes of the network nodes, wherein the input function of the network node with index value n in the node sequence is the output function of the network node with index value n-1, where n is an integer greater than 1; and determining the execution sequence based on the node sequence.

[0021] According to one aspect of this disclosure, a neural network training apparatus is provided, comprising: a network loss module for processing training samples through a neural network model to obtain a network loss of the neural network model; a backpropagation module for performing backpropagation based on the network loss to obtain update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes; and an update module for obtaining an updated neural network model based on the update gradients of the neural network model.

[0022] In one possible implementation, the apparatus further includes: a first determining module, configured to determine, based on the backpropagation graph of the neural network model, at least one network node from a plurality of network nodes included in the neural network model that requires optimized backpropagation; and / or a second determining module, configured to determine, based on the number of network nodes connected to each of the plurality of network nodes, at least one network node from the plurality of network nodes that requires optimized backpropagation.

[0023] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on a first network node among the plurality of network nodes using optimized computational code, wherein the number of back input nodes connected to the first network node is 1.

[0024] In one possible implementation, the optimized computation code is the code after removing the accumulation and memory clearing processes.

[0025] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on the second network node among the plurality of network nodes using an optimized caching method, wherein, in the optimized caching method, the reverse output information of the reverse input node of the second network node or the reverse output information of the second network node is not cached.

[0026] In one possible implementation, the number of reverse input nodes and reverse output nodes of the second network node is 1 each.

[0027] In one possible implementation, the apparatus further includes: a setting module, configured to set first identification information and second identification information for each of the plurality of network nodes, wherein the first identification information is used to identify whether the output information of the forward input node of the network node is consistent with the buffer information of the reverse output node, and the second identification information is used to identify whether the input information of the forward output node of the network node is consistent with the buffer information of the reverse input node; and a third determining module, configured to determine the second network node among the plurality of network nodes based on the first identification information and the second identification information of each of the plurality of network nodes.

[0028] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on at least two of the plurality of network nodes using an optimized backpropagation execution sequence, wherein the at least two network nodes perform gradient backpropagation in parallel in the optimized backpropagation execution sequence.

[0029] In one possible implementation, the apparatus further includes an execution sequence module for obtaining an optimized backpropagation execution sequence based on the data dependencies between the plurality of network nodes.

[0030] In one possible implementation, the execution sequence module is further configured to determine the backpropagation node sequence based on the number of output nodes of the network node and the data dependencies of the reverse output nodes of the network node, wherein the input function of the network node with index value n in the node sequence is the output function of the network node with index value n-1, where n is an integer greater than 1; and to determine the execution sequence based on the node sequence.

[0031] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0032] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0035] Figure 1 A flowchart of a network training method according to an embodiment of the present disclosure is shown;

[0036] Figure 2 A schematic diagram illustrating the forward propagation of a neural network according to an embodiment of the present disclosure is shown;

[0037] Figure 3 A schematic diagram illustrating backpropagation of a neural network according to an embodiment of the present disclosure is shown;

[0038] Figure 4 A schematic diagram illustrating backpropagation of a neural network according to an embodiment of the present disclosure is shown;

[0039] Figure 5 A schematic diagram illustrating data dependency during backpropagation according to an embodiment of this disclosure is shown;

[0040] Figure 6A and Figure 6B A schematic diagram illustrating the function execution order according to embodiments of the present disclosure is shown;

[0041] Figure 7 A schematic diagram illustrating backpropagation according to an embodiment of the present disclosure is shown;

[0042] Figure 8 A schematic diagram illustrating the application of a neural network training method according to an embodiment of the present disclosure is shown.

[0043] Figure 9 A block diagram of a neural network training apparatus according to an embodiment of the present disclosure is shown;

[0044] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0045] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0046] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0047] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0048] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0049] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0050] Figure 1 A flowchart illustrating a neural network training method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the method includes:

[0051] In step S11, the training samples are processed by a neural network model to obtain the network loss of the neural network model;

[0052] In step S12, backpropagation is performed based on the network loss to obtain the update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes.

[0053] In step S13, the updated neural network model is obtained based on the update gradient of the neural network model.

[0054] The neural network training method according to embodiments of the present disclosure can optimize the backpropagation of at least one network node and use the optimized network node to determine the update gradient, thereby reducing the amount of computation in backpropagation, saving memory resources, and improving training efficiency.

[0055] In one possible implementation, the neural network training method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0056] In one possible implementation, the neural network may include a BP neural network (backpropagation neural network), a convolutional neural network, a recurrent neural network, a recurrent neural network, etc. This disclosure does not limit the type of neural network. The training samples include at least one of images, videos, speech, and text. For example, images and videos can be used as training samples for a convolutional neural network, and speech and text can be used as training samples for a recurrent neural network. This disclosure does not limit the type of training samples.

[0057] In one possible implementation, in step S11, the neural network can be an untrained neural network or a pre-trained neural network with a certain level of accuracy. The neural network can be trained using training samples. Training samples can be input into the neural network and forward propagated, processing the training samples using the current network parameters to obtain a prediction result. The obtained prediction result may contain errors.

[0058] In one possible implementation, the network loss of the neural network can be determined by the prediction results and the annotations of the training samples. The training samples may include annotation information; for example, in a sample image, the region where the target object is located can be annotated. The neural network can determine the region where the target object is located in the sample image, and then the network loss can be determined by the difference between the annotated region and the region predicted by the neural network. As another example, the semantic information of sample text can be annotated, and the neural network can determine and predict the semantic information of the sample text. Then, the network loss of the neural network can be determined by the difference between the predicted semantic information and the annotated semantic information. In the examples, network losses of types such as cross-entropy loss and set similarity loss can be determined based on the above differences. This disclosure does not limit the type of network loss.

[0059] In one possible implementation, in step S12, the network loss can be backpropagated using an optimized backpropagation method, and the update gradients of multiple network nodes can be determined during the backpropagation process. Then, the network parameters can be adjusted using the gradient descent method to reduce the network loss and make the prediction results of the neural network more accurate, thereby improving the performance and accuracy of the neural network.

[0060] Figure 2 A schematic diagram illustrating the forward propagation of a neural network according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, during the forward propagation of a neural network, the data of each network node can be determined by the input function of the network node during the forward propagation process. For example, the data of network node B = F1 (the data of network node A), where F1 is the input function of network node B or the output function of network node A; the data of network node C = F2 (the data of network node B); the data of network node D = F3 (the data of network node B); and the data of network node E = F4 (the data of network node C) + F5 (the data of network node D) + F6 (the data of network node A).

[0061] Figure 3 A schematic diagram of backpropagation of a neural network according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, during the backpropagation process of a neural network, the gradient of each network node can be determined by the input function of the network node during the backpropagation process. For example, the gradient of network node C = dF4 (gradient of network node E), the gradient of network node D = dF5 (gradient of network node E), the gradient of network node B = dF3 (gradient of network node D) + dF2 (gradient of network node C), and the gradient of network node A = dF1 (gradient of network node B) + dF6 (gradient of network node E).

[0062] like Figure 2 and Figure 3 As shown, the computation graph for determining the gradients of multiple network nodes in backpropagation is usually the dual graph of the computation graph for determining the data of each node in forward propagation. Therefore, during training, there may be a lot of redundant information cached, consuming more memory resources, and there may be redundant computational processing in the computation graph, which may increase computational overhead and reduce running efficiency.

[0063] Furthermore, in backpropagation, the backpropagation process is usually performed in the reverse order of forward propagation, which may cause two or more operations that have no data dependency to be processed in parallel, thus reducing processing efficiency.

[0064] In one possible implementation, at least one network node among a plurality of nodes in a neural network model that can be optimized can be identified. The method further includes: determining, based on a backpropagation graph of the neural network model, at least one network node from the plurality of network nodes included in the neural network model that requires optimization of backpropagation; and / or determining, based on the number of network nodes connected to each of the plurality of network nodes, at least one network node from the plurality of network nodes that requires optimization of backpropagation.

[0065] In one possible implementation, at least one network node requiring optimization for backpropagation can be determined from the plurality of network nodes based on the number of network nodes each network node is connected to. In an example, the network node requiring optimization can be determined by the number of network nodes connected to the output of the network node during backpropagation (i.e., the network node pointed to by the output function during backpropagation).

[0066] In one possible implementation, during backpropagation, the gradient value of a network node is equal to the sum of the calculated results of the input function pointing to that network node, the calculated value of which can be determined by the input function and the gradient value substituted into the input function (i.e., the gradient value of the network node pointing to that network node).

[0067] Figure 4 A schematic diagram of backpropagation of a neural network according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, the gradient of network node B = dF(gradient of network node C) + dG(gradient of network node D), and the gradient of network node A = dH(gradient of network node B). That is, numerically, the gradient of network node A is determined by its input function dH and the gradient value of node B pointing to node A; that is, the gradient of network node A = dH(gradient of network node B). However, since it is uncertain whether other input functions contribute to the gradient of this network node (for example, if network node A has more than one input function, but two or more, similar to network node B), the computation graph of a network node is usually reserved for cases where there is more than one input function; that is, computational processing is performed when the number of input functions is greater than one.

[0068] In the example, when calculating the gradient of a single-in-degree node A (a network node with only one input function), accumulation processing is typically reserved. For example, accumulation processing can be reserved in the computer-executable computation code, i.e., the gradient of network node A is incremented by dH (the gradient of network node B). When the number of input functions is greater than one, the gradients of all input functions can be accumulated. Alternatively, the initial gradient of network node A can be cleared to prevent the addition of irrelevant values ​​(e.g., due to accumulation processing, all input values ​​of network node A are accumulated, including the calculated values ​​of the input functions and the initial gradient value). To ensure the accuracy of the gradient value of network node A, the initial gradient value of network node A needs to be cleared to zero, and only the calculated values ​​of the input functions are accumulated.

[0069] While the above processing method ensures the accuracy of calculations for multi-in-degree nodes (i.e., nodes with more than one input node in backpropagation), it introduces redundant computational overhead for single-in-degree nodes (i.e., nodes with only one input node in backpropagation), thus reducing processing efficiency. This decrease in efficiency can be particularly significant for neural networks with a large number of single-in-degree nodes.

[0070] In one possible implementation, the training process can be optimized based on the input and output functions of multiple network nodes in the neural network. In this example, the gradient calculation process for single-in-degree nodes can be optimized to improve training efficiency. At least one network node requiring optimization for backpropagation can be determined from the multiple network nodes based on the number of network nodes each node connects to. That is, a first network node with a single input function is determined, i.e., a node connected to only one backpropagation input node. The backpropagation of this first node can then be optimized. For example, the computation code for this node can be optimized. The optimized computation code is the code after removing the accumulation and memory clearing processes.

[0071] Furthermore, the optimized backpropagation of at least one of the plurality of network nodes includes: performing gradient backpropagation on a first network node among the plurality of network nodes using optimized computational code, wherein the number of backpropagation input nodes connected to the first network node is 1. That is, the optimized computational code no longer performs the aforementioned accumulation and memory clearing processes, and the optimized computational code can be used to perform backpropagation on the first network node. For example, the gradient value of the first network node can be made equal to the computation result of the backpropagation input function.

[0072] In the example, such as Figure 4As shown, during backpropagation, if network node A has only one input function, then network node A is a single-in-degree node (a node with only one input function). Similarly, in forward propagation, network node B is a single-in-degree node. In the optimization training process, the single-in-degree nodes during backpropagation can be determined; that is, network node A is the first network node.

[0073] In the example, the computational code of the first network node can be optimized. In the example, the computational code of the first network node A includes the aforementioned accumulation processing and initial gradient value clearing processing. The computational code for accumulation processing and initial gradient value clearing processing can be removed to optimize the backpropagation process of the first network node A, reduce redundant computation, and improve training efficiency.

[0074] For example, the accumulation process can be eliminated. For a single-in-degree node A, the gradient of network node A can be made equal to dH (the gradient of network node B) without the need for accumulation. Furthermore, since the accumulation process is eliminated, when calculating the gradient value of network node A, it is directly set to equal the calculated value of the input function, without adding the initial gradient of A. Therefore, there is no need to clear the initial gradient value of network node A.

[0075] In this way, the accumulation process and memory clearing process can be eliminated, and the optimized operation code can be used for backpropagation, reducing redundant calculations and improving training efficiency.

[0076] In one possible implementation, the update gradients of the network nodes of the neural network model can be obtained according to the optimized backpropagation method. Then, in step S13, the neural network model is trained using these update gradients to obtain the updated neural network model.

[0077] In one possible implementation, the computation graph for determining the gradients of multiple network nodes during backpropagation is typically the dual graph of the computation graph for determining the data of each node during forward propagation. This may result in redundant information being cached during training, consuming more memory resources. Redundant information can be removed to improve memory resource utilization and thus training efficiency.

[0078] In one possible implementation, forward propagation does not rely on any data from backward propagation, but backward propagation usually relies on data cached during forward computation. Therefore, backward propagation computation has a higher memory overhead. Redundant cached data can be optimized to reduce the amount of memory required during training, improve the utilization efficiency of memory resources, and thus improve training efficiency.

[0079] In one possible implementation, the computation graph that determines the gradients of multiple network nodes during backpropagation is typically the dual graph of the computation graph that determines the data of each node during forward propagation. For example, the input function of multiple network nodes during backpropagation is the partial derivative of the output function of the network node during forward propagation. The data that the partial derivative depends on during computation is not only the gradient of the input during backpropagation, but also at least one of the input values, output values ​​and intermediate results computed during forward propagation, which needs to be cached and used during backpropagation.

[0080] Figure 5 This diagram illustrates data dependencies during backpropagation according to embodiments of the present disclosure, such as... Figure 5 As shown, for network node B, its gradient value is gB, its input function is dG, and its output function is dF. Both its input and output functions have a degree of 1, meaning their in-degree and out-degree are both 1. The input function dG depends on the input of function G during forward propagation, i.e., the data of network node B. Similarly, the output function dF depends on the output of function F during forward propagation, i.e., the data of network node B. Therefore, the cached data dependent on the input and output functions of network nodes with in-degree and out-degree both being 1 is numerically identical. Cacheing both the input and output function dependencies may cause data redundancy. Deleting one copy of the cached data and retaining only one copy can release the cache, improve memory resource utilization, and thus enhance training efficiency.

[0081] In one possible implementation, based on the backpropagation graph of the neural network model and the number of network nodes connected to each network node, at least one network node requiring optimized backpropagation can be determined from the plurality of network nodes. Then, an optimized caching method is used to backpropagate the map of the second network node requiring optimization. The optimized backpropagation of at least one network node from the plurality of network nodes includes: using an optimized caching method to perform gradient backpropagation on the second network node from the plurality of network nodes, wherein, in the optimized caching method, the back-output information of the back-input nodes or the back-output information of the second network node are not cached. The number of back-input nodes and back-output nodes of the second network node is both 1.

[0082] In one possible implementation, during backpropagation, the cached information of the input function or output function of a network node can be information cached during forward propagation. The second network node can be determined using identification information set during forward propagation. The method further includes: setting first identification information and second identification information for each of the plurality of network nodes, wherein the first identification information is used to identify whether the output information of the forward input node of the network node is consistent with the cached information of the reverse output, and the second identification information is used to identify whether the input information of the forward output of the network node is consistent with the cached information of the reverse input; and determining the second network node among the plurality of network nodes based on the first identification information and the second identification information of each of the plurality of network nodes.

[0083] In one possible implementation, during forward propagation, the computational data of the input and output functions of multiple network nodes can be cached for use during backpropagation. The output information of the input functions of network nodes in forward propagation (i.e., the output information of the forward input nodes) is consistent with the cached information of the output functions in backpropagation, and the input information of the output functions of network nodes in forward propagation (i.e., the input information of the forward output nodes) is consistent with the cached information of the input functions in backpropagation. In the example, such as... Figure 5 As shown, during forward propagation, the output information of the input function F of network node B can include the data of network node B, which is consistent with the data that the output function dF of network node B depends on during backward propagation; that is, the data of network node B. Similarly, during forward propagation, the input information of the output function G of network node B can include the data of network node B, which is consistent with the data that the input function dG of network node B depends on during backward propagation; that is, the data of network node B. In backward propagation, the data that the output function dG of network node B depends on is consistent with the data that the output function dF depends on, indicating data redundancy and increased cache usage.

[0084] In one possible implementation, during the forward propagation process, a first identifier corresponding to the output information of the input function and a second identifier corresponding to the input information of the output function can also be set.

[0085] In one possible implementation, the first identifier can indicate whether the cached data (i.e., the cached information relied upon for reverse output) is consistent with the output information of the forward input function. If they are consistent, the first identifier is true. If the network node has only one input function, the cached data is consistent with the output information of the input function. If the network node has more than one input function, the cached data is the output information of multiple input functions (e.g., the output information of multiple input functions can be cached separately, or the sum of the output information of multiple input functions can be cached, etc.). That is, the cached data is different from the output information of each input function. Therefore, the first identifier can indicate whether the number of input functions of the network node is 1. That is, if the number of input functions of the network node is 1, the first identifier is true; if the in-degree of the network node is greater than 1, the first identifier is false. Furthermore, if there is no need to cache the output information of the input functions (e.g., the data of the network node is no longer subjected to further computation), the first identifier is false.

[0086] In one possible implementation, the second identifier can indicate whether the cached data (i.e., the cached information relied upon during reverse input) is consistent with the input information of the forward output function. If they are consistent, the second identifier is true. If the network node has only one output function, the cached data is consistent with the input information of the output function. If the network node has more than one output function, the cached data consists of the input information of multiple output functions (e.g., the input information of multiple output functions can be cached separately, i.e., the amount of cached data is multiple, while the amount of data required for the input information of each output function is 1). The cached data is different from the input information of each output function. Therefore, the second identifier can indicate whether the number of output functions of the network node is 1. That is, if the number of output functions of the network node is 1, the second identifier is true; if the number of output functions of the network node is greater than 1, the second identifier is false. Furthermore, if there is no need to cache the input information of the output functions (e.g., the data of the network node is no longer subjected to further computation), the second identifier is false.

[0087] In one possible implementation, the second network node can be determined from multiple network nodes. Specifically, the second network node can be identified from among the multiple network nodes in the neural network model where both the first and second identification information are true. For example, Figure 5 Network node B in the network.

[0088] In one possible implementation, during backpropagation, since the output cache information of the input function of the first node is consistent with the input cache information of the output function, one of them can be deleted to free up cache space.

[0089] In one possible implementation, an optimized caching method is used to perform gradient backpropagation on the second network node among the plurality of network nodes. That is, after reducing redundant cached information, gradient backpropagation is performed on the second network node to obtain the updated gradient. In step S13, the updated gradient can be used to train the neural network model to obtain the updated neural network model.

[0090] In this way, redundant cached data can be removed to save cache space, improve memory resource utilization efficiency, and thus enhance training efficiency.

[0091] In one possible implementation, backpropagation is typically performed in the exact reverse order of forward propagation, which may result in two or more operations being unable to be processed in parallel, thus reducing processing efficiency.

[0092] Figure 6A and Figure 6B A schematic diagram illustrating the function execution order according to an embodiment of the present disclosure is shown.

[0093] like Figure 6A As shown, during the forward propagation process, the processing of node 3 requires the calculation result of function F1, the processing of node 4 requires the calculation result of function F3, the processing of node 5 requires the calculation result of function F2, and the processing of node 6 requires the calculation results of functions F4 and F5. Functions F1 and F2 can be executed simultaneously. However, since the processing of node 6 requires the results of both F4 and F5, function F5 cannot be executed immediately after F1 and F2. Instead, it must wait for function F3 to be executed, and node 4 must process the calculation result of F3 before F4 and F5 are executed simultaneously. Therefore, the execution order of the functions is: F1 and F2 are executed simultaneously first, then F3 is executed, and finally F4 and F5 are executed simultaneously.

[0094] like Figure 6B As shown, during backpropagation, if the execution order of functions is reversed compared to forward propagation, dF4 and dF5 are executed first, followed by dF3, and finally dF1 and dF2. That is, dF2 must wait for dF3 to finish before it can execute. This may cause dF2 and other subsequent nodes of node 2 to execute later, thus affecting training efficiency. Therefore, for the situation where dF2 must wait for dF3 to finish before it can execute, and cannot execute in parallel with dF3, optimization can be achieved through the backpropagation graph of the neural network model. For example, the execution sequence of each function can be set, and the index values ​​of dF2 and dF3 in the execution sequence can be set to the same value, so that dF2 and dF3 execute simultaneously. That is, compared to before optimization, the execution time of function dF2 is advanced, thereby reducing waiting time and improving training efficiency.

[0095] In one possible implementation, based on the backpropagation graph of the neural network model, at least one network node that needs to be optimized for backpropagation is determined from the multiple network nodes included in the neural network model. That is, in the backpropagation computation graph, network nodes that can be executed in parallel but are not executed in parallel are identified for optimization to improve training efficiency.

[0096] Figure 7 A schematic diagram illustrating backpropagation according to an embodiment of this disclosure is shown, such as... Figure 7 As shown, backpropagation typically executes the dual backpropagation in the reverse order of forward propagation. For example, if modules A, B, and C are all network structures containing one or more nodes, and in forward propagation, module C's operations are executed first, followed by module B's, and finally module A's, then in backpropagation, module C's backpropagation is executed first, followed by module B's, and finally module A's, ultimately yielding the output gradient. While these three modules share the same input information, the lack of parallel processing reduces training efficiency.

[0097] In one possible implementation, the use and training of neural networks typically employs a GPU (Graphics Processing Unit) as the main computing device. While GPUs can support parallel processing, the training sequence described above does not involve parallel processing, which reduces training efficiency and wastes GPU performance resources.

[0098] In one possible implementation, training efficiency can be improved through parallel processing. The optimized backpropagation of at least one of the plurality of network nodes includes: performing gradient backpropagation on at least two of the plurality of network nodes using an optimized backpropagation execution sequence, wherein the at least two network nodes perform gradient backpropagation in parallel in the optimized backpropagation execution sequence.

[0099] In one possible implementation, the aforementioned backpropagation execution sequence can represent the execution order of functions during backpropagation, allowing functions with the same index value to be executed in parallel. The method further includes obtaining an optimized backpropagation execution sequence based on the data dependencies between the multiple network nodes.

[0100] In one possible implementation, data dependencies may include the input functions of network nodes, i.e., the output functions of other nodes pointing to the network node. Based on the data dependencies between the plurality of network nodes, an optimized backpropagation execution sequence is obtained, including: determining a backpropagation node sequence based on the number of output nodes of the network node and the data dependencies of the network node's reverse output nodes, wherein the input function of the network node with index value n in the node sequence is the output function of the network node with index value n-1, where n is an integer greater than 1; and determining the execution sequence based on the node sequence.

[0101] In one possible implementation, a node sequence can be constructed, and the first or first group of nodes in the backpropagation (e.g., the node that outputs the result of the neural network model) can be added to the node sequence, with its index value set to 1. Subsequently, the data dependencies of the output nodes connected to the first group of nodes can be determined, i.e., whether the output node depends only on the first group of nodes. If the output node depends only on the first group of nodes, its index value in the node sequence can be set to 2. If the output node depends not only on the first group of nodes but also on other output nodes, then this node cannot execute simultaneously with the other output nodes and must wait for the other nodes to complete their computation before executing, thus its index value can be set to 3.

[0102] In one possible implementation, the network node with index 1 in the node sequence can be pointed to the output function of the output node with index 2, and the index value in the execution sequence can be set to 1.

[0103] In one possible implementation, the input function of the network node with index n in the node sequence is the output function of the network node with index n-1. That is, network nodes pointed to by the network node with index n-1 that only depend on the output function with index n-1 can be added to the node sequence, and their indices in the node sequence can be set to n. Subsequently, the output function of the network node with index n-1 pointing to the output node with index n can be set to n-1 in the execution sequence. The above method of constructing the execution sequence and node sequence can be iteratively executed, connecting all network nodes in the neural network to the node sequence and adding all output functions to the execution sequence. This yields the execution sequence used to represent the execution order of the functions during backpropagation.

[0104] In one possible implementation, backpropagation is performed using an optimized backpropagation execution sequence, that is, the output functions are executed according to their index values. In the example, output functions with the same index value can be executed in parallel to simultaneously determine the update gradient values ​​of the network nodes they point to. In step S13, the updated gradients can be used to train the neural network model, resulting in an updated neural network model.

[0105] In this way, the execution order of multiple output functions in a neural network can be determined by constructing an execution sequence, and output functions with the same index value can be executed in parallel, which can improve the execution efficiency of training and the utilization of processing resources.

[0106] The neural network training method according to embodiments of this disclosure can eliminate accumulation processing and memory zeroing processing, reduce redundant calculations, and delete redundant cached data to save cache space. Furthermore, by constructing an execution sequence, the execution order of multiple output functions in the neural network can be determined, and output functions with the same index value can be executed in parallel. This reduces the amount of computation in backpropagation, saves memory resources, improves training efficiency, and increases the utilization rate of processing resources.

[0107] Figure 8 This diagram illustrates an application of the neural network training method according to embodiments of the present disclosure, such as... Figure 8 As shown, the neural network includes eight nodes (A, B, C, D, E, F, G, H) and eight output functions (dF1, dF2, dF3, dF4, dF5, dF6, dF7, dF). The execution order of these eight output functions can be determined to ensure that they fully utilize GPU performance. In the example, network node A is added to the node sequence, and its index value is set to 1. Subsequently, network nodes B and C are added to the node sequence, and their index values ​​are set to 2. The output functions dF1 and dF2, which point to network nodes B and C respectively, are added to the execution sequence, and their index values ​​are set to 1.

[0108] In the example, network node B and network node C, which point to network node D, are added to the node sequence, and their index values ​​in the node sequence are set to 3. Subsequently, the output function dF3 of network node B pointing to network node D and the output function dF4 of network node C pointing to network node D are added to the execution sequence, and their index values ​​in the execution sequence are set to 2.

[0109] In the example, network node E, which is pointed to by network node D, is added to the node sequence, and its index in the node sequence is set to 4. Subsequently, the output function dF5, which is pointed to by network node D and is linked to by network node E, is added to the execution sequence, and its index in the execution sequence is set to 3.

[0110] In the example, network node F, which is pointed to by network node E, can be added to the node sequence, and its index in the node sequence can be set to 5. Then, the output function dF6 of network node F, which is pointed to by network node E, can be added to the execution sequence, and its index in the execution sequence can be set to 4.

[0111] In the example, network nodes G and H, which are pointed to by network node F, can be added to the node sequence, and their index values ​​in the node sequence can be set to 6. Subsequently, the output functions dF7 and dF8 of network nodes G and H, which are pointed to by network node F, can be added to the execution sequence, and their index values ​​in the execution sequence can be set to 5.

[0112] In one possible implementation, the above output functions can be executed according to the index values ​​in the execution sequence. For example, dF1 and dF2 can be executed in parallel, dF3 and dF4 can be executed in parallel, followed by dF5 and dF6, and then dF7 and dF8 can be executed in parallel.

[0113] In one possible implementation, the first network node (the network node with 1 reverse input node) among multiple network nodes can be determined, namely network nodes E and F, and the accumulation processing and memory clearing processing in the operation code of network nodes E and F can be cleared to reduce redundant calculations and improve training efficiency.

[0114] In one possible implementation, a second network node, namely network node E, with both in-degree and out-degree being 1, is determined. Since the cached data on which the reverse input function dF5 and the reverse output function dF6 of network node E depend are numerically consistent, that is, the output cache information of the reverse input function is numerically consistent with the input cache information of the reverse output function, one of them can be deleted to free up cache space and improve running efficiency.

[0115] In one possible implementation, the gradient of each network node can be determined in the manner described above, and the neural network can be trained using gradient descent to obtain a neural network model with high accuracy.

[0116] In one possible implementation, the neural network training method can save both memory and training time, and can complete more training iterations within a fixed amount of computing resources (e.g., the number of GPUs) and a fixed time to achieve higher accuracy.

[0117] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0118] In addition, this disclosure also provides a neural network training device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the neural network training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0119] Figure 9 A block diagram of a neural network training apparatus according to an embodiment of the present disclosure is shown, such as Figure 9 As shown, the device includes: a network loss module 11, used to process training samples through a neural network model to obtain the network loss of the neural network model; a backpropagation module 12, used to perform backpropagation based on the network loss to obtain the update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes; and an update module 13, used to obtain the updated neural network model based on the update gradient of the neural network model.

[0120] In one possible implementation, the apparatus further includes: a first determining module, configured to determine, based on the backpropagation graph of the neural network model, at least one network node from a plurality of network nodes included in the neural network model that requires optimized backpropagation; and / or a second determining module, configured to determine, based on the number of network nodes connected to each of the plurality of network nodes, at least one network node from the plurality of network nodes that requires optimized backpropagation.

[0121] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on a first network node among the plurality of network nodes using optimized computational code, wherein the number of back input nodes connected to the first network node is 1.

[0122] In one possible implementation, the optimized computation code is the code after removing the accumulation and memory clearing processes.

[0123] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on the second network node among the plurality of network nodes using an optimized caching method, wherein, in the optimized caching method, the reverse output information of the reverse input node of the second network node or the reverse output information of the second network node is not cached.

[0124] In one possible implementation, the number of reverse input nodes and reverse output nodes of the second network node is 1 each.

[0125] In one possible implementation, the apparatus further includes: a setting module, configured to set first identification information and second identification information for each of the plurality of network nodes, wherein the first identification information is used to identify whether the output information of the forward input node of the network node is consistent with the buffer information of the reverse output node, and the second identification information is used to identify whether the input information of the forward output node of the network node is consistent with the buffer information of the reverse input node; and a third determining module, configured to determine the second network node among the plurality of network nodes based on the first identification information and the second identification information of each of the plurality of network nodes.

[0126] In one possible implementation, the backpropagation module is further configured to perform gradient backpropagation on at least two of the plurality of network nodes using an optimized backpropagation execution sequence, wherein the at least two network nodes perform gradient backpropagation in parallel in the optimized backpropagation execution sequence.

[0127] In one possible implementation, the apparatus further includes an execution sequence module for obtaining an optimized backpropagation execution sequence based on the data dependencies between the plurality of network nodes.

[0128] In one possible implementation, the execution sequence module is further configured to determine the backpropagation node sequence based on the number of output nodes of the network node and the data dependencies of the reverse output nodes of the network node, wherein the input function of the network node with index value n in the node sequence is the output function of the network node with index value n-1, where n is an integer greater than 1; and to determine the execution sequence based on the node sequence.

[0129] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0130] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0131] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0132] This disclosure also provides a computer program product including computer-readable code, which, when executed on a device, causes a processor in the device to execute instructions for implementing the neural network training method provided in any of the above embodiments.

[0133] This disclosure also provides another computer program product for storing computer-readable instructions that, when executed, cause a computer to perform the operations of the neural network training method provided in any of the above embodiments.

[0134] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0135] Figure 10 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, or other terminal.

[0136] Reference Figure 10 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0137] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0138] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0139] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0140] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the edges of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0141] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0142] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0143] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0144] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0145] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0146] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0147] Figure 11 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server. (Refer to...) Figure 11The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0148] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0150] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0151] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0152] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0153] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0154] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0158] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0159] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural network training method, characterized in that, include: The training samples are processed by a neural network model to obtain the network loss of the neural network model. The training samples include at least one of images, videos, audio, and text. Backpropagation is performed based on the network loss to obtain the update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes. Based on the update gradient of the neural network model, the updated neural network model is obtained; The optimized backpropagation for at least one of the plurality of network nodes includes: Using an optimized caching method, gradient backpropagation is performed on the second network node among the plurality of network nodes. In the optimized caching method, the reverse output information of the reverse input node or the reverse output information of the second network node is not cached; the number of reverse input nodes and reverse output nodes of the second network node is 1.

2. The method according to claim 1, characterized in that, The method further includes: Based on the backpropagation graph of the neural network model, at least one network node requiring optimization for backpropagation is determined from the multiple network nodes included in the neural network model; and / or Based on the number of network nodes connected to each of the plurality of network nodes, at least one network node from the plurality of network nodes is determined that requires optimization of backpropagation.

3. The method according to claim 1 or 2, characterized in that, The optimized backpropagation for at least one of the plurality of network nodes includes: The gradient is backpropagated to the first network node among the plurality of network nodes using optimized computation code, wherein the number of back input nodes connected to the first network node is 1.

4. The method according to claim 3, characterized in that, The optimized computation code is the code after removing the accumulation process and memory clearing process.

5. The method according to claim 1, characterized in that, The method further includes: For each of the plurality of network nodes, a first identification information and a second identification information are set, wherein the first identification information is used to identify whether the output information of the forward input node of the network node is consistent with the buffer information of the reverse output, and the second identification information is used to identify whether the input information of the forward output of the network node is consistent with the buffer information of the reverse input. Based on the first identification information and the second identification information of each of the plurality of network nodes, the second network node is determined among the plurality of network nodes.

6. The method according to claim 1 or 2, characterized in that, The optimized backpropagation for at least one of the plurality of network nodes includes: Using an optimized backpropagation execution sequence, gradient backpropagation is performed on at least two of the plurality of network nodes, wherein the gradient backpropagation is performed in parallel on the at least two network nodes in the optimized backpropagation execution sequence.

7. The method according to claim 6, characterized in that, The method further includes: Based on the data dependencies between the multiple network nodes, an optimized backpropagation execution sequence is obtained.

8. The method according to claim 7, characterized in that, Based on the data dependencies between the multiple network nodes, an optimized backpropagation execution sequence is obtained, including: Based on the number of output nodes of the network node and the data dependency relationship of the reverse output nodes of the network node, the node sequence for backpropagation is determined, wherein the input function of the network node with index value n in the node sequence is the output function of the network node with index value n-1, where n is an integer greater than 1; The execution sequence is determined based on the node sequence.

9. A neural network training device, characterized in that, include: The network loss module is used to process the training samples through the neural network model to obtain the network loss of the neural network model. The training samples include at least one of images, videos, speech, and text. The backpropagation module is used to perform backpropagation based on the network loss to obtain the update gradients of multiple network nodes of the neural network model, wherein optimized backpropagation is performed on at least one of the multiple network nodes. An update module is used to obtain an updated neural network model based on the update gradient of the neural network model. The backpropagation module is further used to perform gradient backpropagation on the second network node among the plurality of network nodes using an optimized caching method, wherein, in the optimized caching method, the reverse output information of the reverse input node or the reverse output information of the second network node is not cached; the number of reverse input nodes and reverse output nodes of the second network node is 1.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images