Neural network training method, data processing method and related equipment

Abstract

The invention provides a neural network training method, a data processing method and related equipment, and the method comprises the steps: determining a target back propagation strategy from a plurality of preset back propagation strategies; for any iteration in the neural network training, according to the target back propagation strategy, a target back propagation path of gradient updating in the iteration is determined, and the target back propagation path comprises a part of propagation paths in all back propagation paths; and performing gradient updating on parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of the iteration. According to the embodiment of the invention, the cache data volume and the training calculation amount in the neural network training process can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a neural network training method, a data processing method and related equipment. Background Art

[0002] In the related art, when training a neural network, back propagation (BP) is usually used to update the gradient. However, back propagation requires saving all the intermediate state variables of the neural network (such as the activation of each layer), which will result in too much cached data. In particular, when training neural networks that process sequence data, such as recurrent neural networks and spiking neural networks, back propagation will be performed along time, which requires saving more intermediate state variables, making the problem of too much cached data more prominent. Summary of the invention

[0003] The present invention provides a neural network training method, a data processing method and related equipment.

[0004] In a first aspect, the present disclosure provides a neural network training method, which includes: determining a target back propagation strategy from a plurality of preset back propagation strategies; for any iteration in the neural network training, determining a target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes a portion of all back propagation paths; performing gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

[0005] In a second aspect, the present disclosure provides a data processing method, which includes: obtaining data to be processed; processing the data to be processed through a neural network to obtain a data processing result, wherein the neural network is trained through the above-mentioned neural network training method, and the data to be processed includes any one of voice, text, and video.

[0006] In a third aspect, the present disclosure provides a neural network training device, which includes: a strategy determination module, which is used to determine a target back propagation strategy from a plurality of preset back propagation strategies; a path determination module, which, for any iteration in the neural network training, determines the target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes some propagation paths in all back propagation paths; a gradient update module, which is used to perform gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

[0007] In a fourth aspect, the present disclosure provides a data processing device, which includes: a data acquisition module for acquiring data to be processed; a data processing module for processing the data to be processed through a neural network to obtain a data processing result, wherein the neural network is trained through the above-mentioned neural network training method, and the data to be processed includes any one of voice, text, and video.

[0008] In a fifth aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned neural network training method or data processing method.

[0009] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned neural network training method or data processing method when executed by a processor.

[0010] According to an embodiment of the present disclosure, when training a neural network, first, a target back propagation strategy is determined from a plurality of preset back propagation strategies, and then for any iteration in the neural network training, a target back propagation path for gradient update in this iteration is determined according to the target back propagation strategy, and the target back propagation path includes a portion of all back propagation paths, and then the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration. Since the target back propagation path includes a portion of all back propagation paths, when the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path, only the state variables of the network layers involved in the back propagation (i.e., the state variables of the network layers included in the back propagation path) need to be stored and calculated, without storing and calculating other state variables, thereby not only being able to significantly reduce the amount of cached data in the neural network training process, but also being able to reduce the amount of training calculations.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 A flowchart of a neural network training method provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a connection diagram of a processing state provided by an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram of a target reverse propagation path provided in an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a target reverse propagation path provided in an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a target reverse propagation path provided in an embodiment of the present disclosure;

[0018] Figure 6 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0019] Figure 7 A block diagram of a neural network training device provided in an embodiment of the present disclosure;

[0020] Figure 8 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0021] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0024] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0025] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0026] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0027] In order to solve the above technical problems, an embodiment of the present disclosure provides a neural network training method, the method comprising: determining a target back propagation strategy from a plurality of preset back propagation strategies; for any iteration in the neural network training, determining a target back propagation path for gradient update in this iteration according to the target back propagation strategy, the target back propagation path including some propagation paths among all back propagation paths; performing gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

[0028] According to an embodiment of the present disclosure, when training a neural network, first, a target back propagation strategy is determined from a plurality of preset back propagation strategies, and then for any iteration in the neural network training, a target back propagation path for gradient update in this iteration is determined according to the target back propagation strategy, and the target back propagation path includes a portion of all back propagation paths, and then the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration. Since the target back propagation path includes a portion of all back propagation paths, when the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path, only the state variables of the network layers involved in the back propagation (i.e., the state variables of the network layers included in the back propagation path) need to be stored and calculated, without storing and calculating other state variables, thereby not only being able to significantly reduce the amount of cached data in the neural network training process, but also being able to reduce the amount of training calculations.

[0029] The neural network training method according to the embodiment of the present disclosure can be executed by a server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The data processing method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0030] The neural network training method of the disclosed embodiment can be applied to a spiking neural network (Spiking Neural Network, SNN), improving the practicality of the spiking neural network. The neural network training method of the disclosed embodiment can also be applied to a neural network training scenario with a time dimension, which can be time-discrete and unfolded along time, and trained along the back propagation through time (BPTT).

[0031] Figure 1 A flowchart of a neural network training method provided by an embodiment of the present disclosure. Figure 1 , the method comprising:

[0032] Step S11, determining a target back propagation strategy from a plurality of preset back propagation strategies;

[0033] Step S12, for any iteration in the neural network training, determining a target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes a portion of all back propagation paths;

[0034] Step S13, performing gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

[0035] In some possible implementations, the neural network of the embodiments of the present disclosure is a neural network for processing sequence data, such as a spiking neural network (SNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. The sequence data includes speech, text, video, etc. The neural network of the embodiments of the present disclosure can be used to perform data processing tasks such as speech processing, text processing, and video processing. It should be noted that the present disclosure does not limit the specific types of data processing tasks performed by the neural network.

[0036] In some possible implementations, when training a neural network, first in step S11, a target back propagation strategy may be determined from a plurality of preset back propagation strategies, wherein the plurality of preset back propagation strategies include at least one of a preset shape path strategy, a fixed random path strategy, a dynamic random path strategy, and a hybrid strategy.

[0037] The preset shape path strategy refers to a strategy for determining a path based on a preset shape. The preset shape may be, for example, a plane shape such as a triangle, a trapezoid, a rectangle, etc., and the present disclosure does not limit the specific type of the preset shape. The preset shape may be generated by preset rules based on the number of layers of the neural network and the number of time steps of the sequence data processed by the neural network. For example, a triangle can be generated by L / N*nl≥0, where L represents the number of layers of the neural network, N represents the number of time steps of the sequence data processed by the neural network, l represents the number of layers included in the preset shape, and n represents the number of time steps included in the preset shape. For another example, when the preset shape is a trapezoid, it can be split into two parts, a triangle and a rectangle, to be generated separately.

[0038] The fixed random path strategy refers to a strategy that randomly generates a path and the path is fixed for each iteration. In other words, under the fixed random path strategy, a path is randomly generated before the first iteration, and the path is used for each subsequent iteration. The dynamic random path strategy refers to a strategy that dynamically generates a path randomly for each iteration. The hybrid strategy refers to a strategy that selects at least one of the preset shape path strategy, fixed random path strategy, dynamic random path strategy, and all path strategy for each iteration. Among them, the all path strategy refers to a strategy that uses all back propagation paths.

[0039] A strategy can be randomly selected from multiple preset back-propagation strategies as the target back-propagation strategy. A strategy can also be selected from multiple preset back-propagation strategies in a certain order as the target back-propagation strategy. For example, the training of a neural network includes 5 rounds, and each round includes multiple iterations. A strategy can be selected from multiple back-propagation strategies in order of the training rounds as the target back-propagation strategy: for the first round of training, a preset shape path strategy can be selected as the target back-propagation strategy; for the second round of training, a fixed random path strategy can be selected as the target back-propagation strategy; for the third round of training, a dynamic random path strategy can be selected as the target back-propagation strategy; for the fourth round of training, a mixed strategy can be selected as the target back-propagation strategy; for the fifth round of training, a preset shape path strategy can be selected as the target back-propagation strategy.

[0040] It should be noted that those skilled in the art may set the method for selecting the target back-propagation strategy according to actual conditions, and the present disclosure does not limit this.

[0041] In some possible implementations, after determining the target back propagation strategy, for any iteration in the neural network training, in step S12, the target back propagation path for gradient update in this iteration can be determined according to the target back propagation strategy. The target back propagation path includes some propagation paths in all back propagation paths. That is, when the target back propagation path is used for gradient update, other paths in all back propagation paths that are not included in the target back propagation path are disconnected.

[0042] The target back propagation path may be fixed, for example, when the target back propagation strategy is a fixed random path strategy, the target back propagation path for each iteration is fixed. The target back propagation path may also be non-fixed. For example, when the target back propagation strategy is a dynamic random path strategy, the target back propagation path for each iteration is dynamically randomly generated.

[0043] In some possible implementations, the target back propagation path for the gradient update in this iteration may be the same as the target back propagation path for the gradient update in the previous iteration, or may be different from the target back propagation path for the gradient update in the previous iteration, and the present disclosure does not impose any restrictions on this.

[0044] In some possible implementations, in step S13, the parameters of the neural network can be gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration. Among them, forward propagation refers to the process of the neural network gradually converting the input sequence data into output through the hierarchical structure and parameters. Back propagation refers to the process of backpropagating the network loss forward from the output layer of the neural network.

[0045] When training a neural network, the process of inputting a sample sequence into the neural network for processing to obtain the sequence processing result is forward propagation. Based on the sequence processing result, the network loss of this iteration can be determined, and then the process of propagating the network loss from back to front based on the target back propagation path is back propagation. Through forward propagation and back propagation based on the target back propagation path, the parameters of the neural network can be gradient updated to obtain the neural network of this iteration.

[0046] According to an embodiment of the present disclosure, when training a neural network, first, a target back propagation strategy is determined from a plurality of preset back propagation strategies, and then for any iteration in the neural network training, a target back propagation path for gradient update in this iteration is determined according to the target back propagation strategy, and the target back propagation path includes a portion of all back propagation paths, and then the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration. Since the target back propagation path includes a portion of all back propagation paths, when the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path, only the state variables of the network layers involved in the back propagation (i.e., the state variables of the network layers included in the back propagation path) need to be stored and calculated, without storing and calculating other state variables, thereby not only being able to significantly reduce the amount of cached data in the neural network training process, but also being able to reduce the amount of training calculations.

[0047] The following is a detailed description of the neural network training method according to an embodiment of the present disclosure.

[0048] In some possible implementations, the neural network of the embodiment of the present disclosure includes multiple network layers, and the network layer can be an input layer, an output layer, a convolution layer, an activation layer, a pooling layer, a fully connected layer, etc. The present disclosure does not limit the specific type of the network layer. Each iteration in the neural network training of the embodiment of the present disclosure includes multiple processing states. The processing state can be used to indicate the intermediate state of the neural network in the process of processing sequence data. Since the sequence data has a time characteristic, the sequence data is discretized according to time to obtain a plurality of ordered time steps. Then, after the neural network is expanded along the time dimension, the state of each network layer of the neural network at each time step is regarded as the intermediate state of the neural network in the process of processing the sequence data.

[0049] In some possible implementations, the target back-propagation path includes a target network layer in a target processing state, the target network layer is at least one of the plurality of network layers, and the target processing state is at least one of the plurality of processing states. The number of target processing states is less than the total number of processing states, that is, the target processing state is a portion of the plurality of processing states. The target network layer may be part or all of the plurality of network layers.

[0050] For example, assuming that the neural network includes 32 network layers, and the sample sequence used in this iteration is audio data with a length of 16 frames, then the number of network layers is 32, and the sample sequence is discretized in time to obtain 16 time steps, that is, each frame corresponds to a time step, and the number of processing states is 512 (obtained by 32×16). The target processing state in the target backpropagation path is a part of the 512 processing states, and the network layer corresponding to the target processing state is the target network layer.

[0051] In some possible implementations, the target back propagation strategy is a preset shape path strategy, which refers to a strategy for determining a path according to a preset shape. Step S12 includes: selecting a target processing state from a connection graph of processing states according to the preset shape, wherein the connection graph of the processing states is obtained by expanding the multiple processing states along the time dimension and the space dimension, the time dimension is used to indicate the time order of the sample sequence, and the space dimension is used to indicate the connection order of the multiple network layers; determining the target back propagation path for gradient update in this iteration according to the connection relationship of the target processing state in the connection graph.

[0052] The neural network of the embodiment of the present disclosure includes multiple processing states. A connection diagram of the processing states can be determined based on the sample sequence. The connection diagram of the processing states is obtained by expanding the multiple processing states of the neural network along the time dimension and the space dimension, wherein the time dimension is used to indicate the time order of the sample sequence, and the space dimension is used to indicate the connection order of multiple network layers of the neural network.

[0053] Figure 2 A schematic diagram of a connection diagram of a processing state provided by an embodiment of the present disclosure. Figure 2 , the neural network 21 includes four network layers, the sequence data is audio, the number of frames is 4, and each frame corresponds to a time step. Then, the neural network 21 includes 16 (through 4×4 calculation) processing states. After expanding these 16 processing states along the time dimension and the space dimension, a connection diagram 22 of the processing state is obtained. In the connection diagram 22 of the processing state, the rightward and upward arrows represent forward propagation, and the leftward and downward arrows represent reverse propagation. Figure 2 The back propagation paths shown in are all back propagation paths. It should be noted that there is only one copy of the parameters of the same network layer at different time steps, that is, the parameters are shared.

[0054] After obtaining the connection graph of the processing states, the target processing state can be selected from the connection graph of the processing states according to a preset shape, such as a triangle or a trapezoid, and then the target back propagation path for gradient update in this iteration can be determined based on the connection relationship of the target processing states in the connection graph.

[0055] Figure 3 A schematic diagram of a target reverse propagation path provided by an embodiment of the present disclosure. Figure 3 , the target backpropagation path (the path consisting of downward and leftward arrows) includes the target processing state in the triangular area in the connection graph of the processing state, that is, the target processing state is selected from the connection graph of the processing state according to the preset triangle. Compared with caching the state variables of all 16 processing states, using Figure 3 When the back propagation path shown in the figure is used, only the state variables of the 10 processing states connected by the downward arrow and the left arrow need to be cached, without caching the state variables of other processing states not included in the back propagation path, thereby reducing the amount of cached data.

[0056] Figure 4 A schematic diagram of a target reverse propagation path provided by an embodiment of the present disclosure. Figure 4 , the target back propagation path (the path consisting of downward and leftward arrows) includes the target processing state in the trapezoidal area in the connection diagram of the processing state, that is, the target processing state is selected from the connection diagram of the processing state according to the preset trapezoid. Compared with caching the state variables of all 16 processing states, using Figure 4 When the back propagation path shown in the figure is used, only the state variables of the 12 processing states connected by the downward arrow and the left arrow need to be cached, without caching the state variables of other processing states not included in the back propagation path, thereby reducing the amount of cached data.

[0057] from Figure 3 and Figure 4 It can be seen that Figure 3 The preset shape in is a triangle, and in the target back-propagation path, the last layer of the neural network only calculates the gradient of the last time step; Figure 4 The default shape in is a trapezoid, and in the target backpropagation path, the last layer of the neural network calculates the gradient of the last two time steps.

[0058] In the embodiments of the present disclosure, when the target back propagation strategy is a preset shape path strategy, the target processing state can be selected from the connection graph of the processing states according to the preset shape, and the target back propagation path for gradient update in this iteration can be determined according to the connection relationship of the target processing states in the connection graph. Thus, the target back propagation path can be determined based on the preset shape, which is simple and fast and can improve the processing efficiency when determining the target back propagation path.

[0059] In some possible implementations, the target back-propagation strategy is a fixed random path strategy, which refers to a strategy in which a path is randomly generated and the path is fixed for each iteration. The step S12 may include: determining the target random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the multiple processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate the time order of a sample sequence, and the space dimension is used to indicate the connection order of the multiple network layers; determining the target random path as the target back-propagation path for each iteration of the neural network.

[0060] When the target back propagation strategy is a fixed random path strategy, the target back propagation path for each iteration is fixed when the neural network is trained. In other words, in this case, the target back propagation path only needs to be randomly generated once before the first iteration, and can be used directly in subsequent iterations. That is, after the path is randomly generated for the first time, the path is fixed, and the path will not change after multiple subsequent uses.

[0061] When determining the target back propagation path, the multiple processing states of the neural network can be first expanded along the time dimension and the space dimension to obtain a connection diagram of the processing state; then, the target random path can be determined from the connection diagram of the processing state by a breadth-first search or depth-first search with probability judgment. For example, the node for calculating the network loss can be determined based on the connection diagram of the processing state; then, starting from the node for calculating the network loss, the nodes connected to it are traversed with a preset probability; the traversed nodes are marked as nodes for performing back propagation, and the nodes that are not traversed are not used as nodes for performing back propagation, and so on, so as to obtain the target random path. It should be noted that those skilled in the art can set the specific method for determining the target random path according to the actual situation, and the present disclosure does not limit this.

[0062] The target random path is one or more. The target random path is a connected path, which can avoid the situation where the back propagation error cannot be obtained due to disconnection, and improve the comprehensive effect and efficiency of error back propagation. The target random path needs to be determined before the forward propagation is performed in this iteration in order to control the cache of the state variable.

[0063] After obtaining the target random path, the path can be used as the target back propagation path for each iteration of the neural network. In other words, the target back propagation path used for each iteration of the neural network is the same or fixed.

[0064] Figure 5 A schematic diagram of a target reverse propagation path provided by an embodiment of the present disclosure. Figure 5 , the target backpropagation path (the path consisting of downward and leftward arrows) includes 14 target processing states. Compared with caching state variables for all 16 processing states, using Figure 5 When the target back propagation path is shown, only the state variables of the 14 processing states connected by the downward arrow and the left arrow need to be cached, without caching the state variables of the processing states not included in the target back propagation path, thereby reducing the amount of cached data.

[0065] In the embodiments of the present disclosure, when the target back propagation strategy is a fixed random path strategy, the target random path can be determined from the connection diagram of the processing state, and the target random path can be determined as the target back propagation path for each iteration of the neural network, so that the target back propagation path can be determined by means of a random path, and the target back propagation path is used in each iteration, which is simple and fast and can improve the processing efficiency when determining the target back propagation path.

[0066] In some possible implementations, the target back-propagation strategy is a dynamic random path strategy, which refers to a strategy for dynamically randomly generating a path in each iteration, and the step S12 includes: determining a first random path from a connection graph of processing states, the connection graph of processing states being obtained by expanding the multiple processing states along a time dimension and a space dimension, the time dimension being used to indicate a time order of a sample sequence, and the space dimension being used to indicate a connection order of the multiple network layers, the first random path being randomly generated based on the connection graph before each iteration, and a starting node of the first random path being connected to a node for calculating network loss; and determining the first random path as the target back-propagation path for gradient update in this iteration.

[0067] When the target back propagation strategy is a dynamic random path strategy, the target back propagation path used by the neural network for each iteration is a dynamically randomly generated path, and is dynamically generated before each iteration. When determining the target back propagation path, the multiple processing states of the neural network can be first expanded along the time dimension and the space dimension to obtain a connection graph of the processing state, and then the first random path is determined from the connection graph of the processing state by a breadth-first search or depth-first search with probability judgment. In other words, the first random path is randomly generated based on the connection graph of the processing state before each iteration, and the starting node of the first random path is connected to the node for calculating the network loss. After obtaining the first random path, the first random path can be determined as the target back propagation path for gradient update in this iteration.

[0068] It should be noted that the present disclosure does not limit the method for obtaining the first random path. In addition, when the target back propagation strategy is a dynamic random path strategy, the connectivity of the first random path does not need to be strictly satisfied, that is, it is also possible to be non-connected, as long as the following conditions are met: the starting node of the first random path is connected to the node for calculating the network loss, otherwise the back propagation error cannot be obtained, that is, it is required that there is no isolated back propagation path that is not connected to the node for calculating the network loss.

[0069] In the embodiments of the present disclosure, when the target back propagation strategy is a dynamic random path strategy, it is possible to determine a first random path from a connection diagram of processing states, and then determine the first random path as the target back propagation path for gradient update in this iteration without restricting the proportion of processing states included in the target back propagation path. This is simple and fast, and can improve the processing efficiency when determining the target back propagation path.

[0070] In some possible implementations, determining a first random path from a connection graph of a processing state includes: determining a first node according to the connection graph of the processing state, wherein the first node is a node for calculating network loss; starting from the first node, traversing nodes connected to the first node with a preset probability, and determining the traversed nodes connected to the first node as second nodes; and determining a path including the second node as a first random path.

[0071] When determining the first random path, the first node may be determined according to the connection graph of the processing state, and the first node is the node for calculating the network loss. The node for calculating the network loss may be a node in the connection graph of the processing state, for example, a node where the processing state of the last network layer is located, or a node where the processing state of other network layers is located. The node for calculating the network loss may also be other nodes connected to the connection graph, and the present disclosure does not limit this.

[0072] After the first node is determined, the nodes connected to the first node may be traversed starting from the first node with a preset probability (for example, the preset probability is 0.4), and the traversed nodes may be determined as the second nodes. Then the path including the second node may be determined as the first random path.

[0073] In the embodiments of the present disclosure, a first node can be determined based on a connection diagram of a processing state, where the first node is a node for calculating network loss; then starting from the first node, nodes connected to the first node are traversed with a preset probability, and the traversed nodes connected to the first node are determined as second nodes; then, the path containing the second node is determined as a first random path, so that the first random path can be quickly determined, thereby improving processing efficiency.

[0074] In some possible implementations, the target back propagation strategy is a dynamic random path strategy, and determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: determining a first number of target processing states according to a preset first ratio and the number of the multiple processing states, the first ratio referring to the ratio of the number of the target processing states to the number of the multiple processing states; determining a second random path from a connection graph of the processing states according to the first number, the connection graph of the processing states being obtained by expanding the multiple processing states along a time dimension and a space dimension, the time dimension being used to indicate the time order of the sample sequence, and the space dimension being used to indicate the connection order of the multiple network layers; determining the second random path as the target back propagation path for gradient update in this iteration.

[0075] When the target back propagation strategy is a dynamic random path strategy, the first number of target processing states can be determined according to a preset first ratio and the number of multiple processing states. The first ratio refers to the ratio of the number of target processing states to the number of the multiple processing states. For example, assuming that the first ratio is 50% and the number of multiple processing states is 32, then the first number of target processing states can be determined to be 16 (calculated by 32×50%).

[0076] Then, according to the first number, a second random path is determined from the connection graph of the processing state by a breadth-first search or a depth-first search with probability judgment. The second random path is one or more. The number of processing states included in all the second random paths is the first number. The second random path is then determined as the target backpropagation path for gradient update in this iteration.

[0077] It should be noted that the present disclosure does not limit the method for obtaining the second random path. In addition, when the target back propagation strategy is a dynamic random path strategy, the connectivity of the second random path does not need to be strictly satisfied, that is, it is also possible to be non-connected, as long as the following conditions are met: the starting node of the second random path is connected to the node for calculating the network loss, otherwise the back propagation error cannot be obtained, that is, it is required that there is no isolated back propagation path that is not connected to the node for calculating the network loss.

[0078] In an embodiment of the present disclosure, when the target back propagation strategy is a dynamic random path strategy, it is possible to determine a first number of target processing states based on a preset first ratio and the number of multiple processing states, and based on the first number, determine a second random path from a connection diagram of the processing states, and then determine the second random path as the target back propagation path for gradient update in this iteration, so that the target back propagation path can be determined based on the first ratio, so that the target back propagation path can meet the preset processing state coverage requirements.

[0079] In some possible implementations, the target back propagation strategy is a dynamic random path strategy, and determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: determining a second proportion corresponding to this iteration according to the variance of network loss within a preset number of iterations and a preset coefficient; determining a second number of target processing states according to the second proportion and the number of the multiple processing states; determining a third random path from a connection graph of the processing states according to the second number, wherein the connection graph of the processing states is obtained by expanding the multiple processing states along a time dimension and a space dimension, the time dimension being used to indicate the time order of the sample sequence, and the space dimension being used to indicate the connection order of the multiple network layers; determining the third random path as the target back propagation path for gradient update in this iteration.

[0080] When determining the target back propagation path, the variance of the network loss within a preset number of iterations can be calculated first. The preset number of iterations can be, for example, 5 times. The specific value of the preset number of iterations can be set according to actual conditions, and the present disclosure does not limit this. Then, based on the variance of the network loss within the preset number of iterations and the preset coefficient, the second ratio corresponding to this iteration is determined. The second ratio is proportional to the variance. That is, the larger the variance, the larger the second ratio, and the smaller the variance, the smaller the second ratio. Then, based on the second ratio and the number of multiple processing states, the second number of target processing states is determined. For example, assuming that the second ratio is 40% and the number of multiple processing states is 32, then the second number of target processing states can be determined to be 13 (obtained by 32×40% and rounded off).

[0081] Then, according to the second number, a third random path is determined from the connection graph of the processing state by a breadth-first search or a depth-first search with probability judgment. The third random path is one or more. The number of processing states included in all the third random paths is the second number. The third random path is then determined as the target backpropagation path for gradient update in this iteration.

[0082] It should be noted that the present disclosure does not limit the method for obtaining the third random path. In addition, when the target back propagation strategy is a dynamic random path strategy, the connectivity of the third random path does not need to be strictly satisfied, that is, it is also OK to be not connected, as long as the following conditions are met: the starting node of the third random path is connected to the node for calculating the network loss, otherwise the back propagation error cannot be obtained, that is, it is required that there is no isolated back propagation path that is not connected to the node for calculating the network loss.

[0083] In an embodiment of the present disclosure, when the target back-propagation strategy is a dynamic random path strategy, it is possible to determine a second proportion corresponding to this iteration based on the variance of the network loss within a preset number of iterations and a preset coefficient, and determine a second number of target processing states based on the second proportion and the number of multiple processing states; then, based on the second number, a third random path is determined from a connection diagram of the processing states, and the third random path is determined as the target back-propagation path for gradient update in this iteration, so that the proportion of the target processing state can be dynamically adjusted based on the variance of the network loss within a preset number of iterations and the preset coefficient, and then the target back-propagation path can be dynamically adjusted, thereby improving the accuracy and efficiency of gradient calculation during neural network training.

[0084] In some possible implementations, determining the second ratio corresponding to the current iteration based on the variance of the network loss within a preset number of iterations and a preset coefficient includes: performing normalization processing on the product of the variance of the network loss within a preset number of iterations and the preset coefficient to obtain the second ratio.

[0085] When determining the second ratio, the variance of the network loss within a preset number of iterations may be first calculated, and then the product of the variance and a preset coefficient may be calculated, and then the product may be normalized to obtain the second ratio.

[0086] The second ratio P can be calculated by the following formula (1):

[0087] P = sigmoid(a×S 2 ) (1)

[0088] Among them, a represents the preset coefficient, S 2 It represents the variance of network loss within the preset number of iterations, and sigmoid() represents the activation function, which is used to perform normalization.

[0089] In the embodiments of the present disclosure, the variance of the network loss within a preset number of iterations and the product of a preset coefficient can be normalized to obtain the second ratio, which is simple and fast and can improve the processing efficiency when calculating the second ratio.

[0090] In some possible implementations, determining the second ratio corresponding to the current iteration based on the variance of the network loss within a preset number of iterations and a preset coefficient includes: determining a derivative of the variance of the network loss within a preset number of iterations over time; and performing normalization processing on the product of the derivative and the preset coefficient to obtain the second ratio.

[0091] When determining the second ratio, the variance of the network loss within a preset number of iterations may be first calculated, and then the derivative of the variance over time may be calculated, and then the product of the derivative and the preset coefficient may be normalized to obtain the second ratio.

[0092] The second ratio P can be calculated by the following formula (2):

[0093] P = sigmoid(a×d) (2)

[0094] Among them, a represents the preset coefficient, d represents the derivative of the variance of the network loss within the preset number of iterations over time, and sigmoid() represents the activation function, which is used to perform normalization processing.

[0095] In the embodiments of the present disclosure, the derivative of the variance of the network loss within a preset number of iterations over time can be determined; and the product of the derivative and the preset coefficient is normalized to obtain a second ratio, so that when calculating the second ratio, the derivative of the variance of the network loss (i.e., the characteristic of the variance changing over time) can be considered, thereby improving the accuracy of the second ratio.

[0096] In some possible implementations, determining the second proportion corresponding to the current iteration based on the variance of the network loss within a preset number of iterations and a preset coefficient includes: performing normalization processing on the product of the variance of the network loss within the preset number of iterations and the preset coefficient to obtain a third proportion; determining a derivative of the variance of the network loss within the preset number of iterations over time; performing normalization processing on the product of the derivative and the preset coefficient to obtain a fourth proportion; determining the second proportion based on the third proportion, the fourth proportion, the first weight and the second weight, wherein the first weight is the weight coefficient of the third proportion, and the second weight is the weight coefficient of the fourth proportion.

[0097] When determining the second ratio, the variance of the network loss within a preset number of iterations can be first calculated, and the product of the variance and the preset coefficient can be normalized to obtain the third ratio; then the derivative of the variance of the network loss within the preset number of iterations over time is calculated, and the product of the derivative and the preset coefficient can be normalized to obtain the fourth ratio.

[0098] Then, the second ratio is determined according to the third ratio, the fourth ratio, the first weight, and the second weight. The first weight is a weight coefficient of the third ratio, the second weight is a weight coefficient of the fourth ratio, and the sum of the first weight and the second weight is 1.

[0099] The second ratio P can be calculated by the following formula (3):

[0100] P = w1×sigmoid(a×S 2 )+w2×sigmoid(a×d) (3)

[0101] In formula (3), sigmoid() represents an activation function used to perform normalization processing, w1 is the first weight, w2 is the second weight, a represents a preset coefficient, S 2 Represents the variance of the network loss within the preset number of iterations, sigmoid(a×S 2 ) represents the third ratio, d represents the derivative of the variance of the network loss within a preset number of iterations over time, and sigmoid(a×d) represents the fourth ratio.

[0102] In the embodiments of the present disclosure, the variance of the network loss within a preset number of iterations and the product of a preset coefficient can be normalized to obtain a third ratio; then the derivative of the variance of the network loss within the preset number of iterations over time is determined, and the product of the derivative and the preset coefficient is normalized to obtain a fourth ratio; then the second ratio is determined based on the third ratio, the fourth ratio, the first weight and the second weight, so that when calculating the second ratio, the variance of the network loss within the preset number of iterations, the derivative of the variance and the weights of the two can be considered, thereby improving the accuracy of the second ratio.

[0103] In some possible implementations, the target back propagation strategy is a hybrid strategy, and the step S12 may include: selecting a strategy from the preset shape path strategy, the fixed random path strategy, the dynamic random path strategy and all path strategies as the back propagation strategy for this iteration; and determining the target back propagation path for gradient update in this iteration according to the back propagation strategy for this iteration.

[0104] When the target back propagation strategy is a hybrid strategy, the back propagation strategy for this iteration needs to be dynamically determined before each iteration. A strategy can be selected from the preset shape path strategy, the fixed random path strategy, the dynamic random path strategy and all path strategies in a preset order or in a random manner or by other means as the back propagation strategy for this iteration. Then, according to the back propagation strategy for this iteration, the target back propagation path for gradient update in this iteration is determined. The method for determining the target back propagation path is similar to the above embodiment and will not be repeated here.

[0105] In the embodiments of the present disclosure, when the target back propagation strategy is a hybrid strategy, a strategy can be selected from a preset shape path strategy, a fixed random path strategy, a dynamic random path strategy and all path strategies as the back propagation strategy for this iteration, and based on the back propagation strategy for this iteration, the target back propagation path for gradient update in this iteration is determined, thereby improving the diversity of the target back propagation path and further improving the effectiveness of neural network training.

[0106] In some possible implementations, step S13 may include: inputting a sample sequence into the neural network, executing the processing of the multiple processing states in sequence, outputting a sequence processing result, and storing a state variable of the target network layer under the target processing state, the sample sequence including any one of speech, text, and video; determining the network loss of this iteration based on the sequence processing result; performing a gradient update on the parameters of the neural network based on the target back propagation path according to the network loss and the state variables of the target network layer under the target processing state to obtain the neural network of this iteration.

[0107] When the parameters of the neural network are updated by gradient through forward propagation and back propagation based on the target back propagation path, the sample sequence used to train the neural network can be first input into the neural network, and multiple processing states are processed in sequence, and the sequence processing results are output. During the processing, the state variables of the target network layer in the target processing state are stored. Among them, the sample sequence can be any one of speech, text, and video. The specific content of the sample sequence can be determined according to the data processing task to be performed by the neural network. For example, assuming that the neural network is used to perform a speech recognition task, the sample sequence used when training the neural network is speech. This process is the forward propagation in this iteration.

[0108] Then, the network loss of this iteration can be determined based on the sequence processing results. When performing unsupervised training on a neural network, the network loss of this iteration can be calculated based on the sequence processing results and the loss function. When performing supervised training on a neural network, the network loss of this iteration can be calculated based on the sequence processing results, the sample labels corresponding to the sample sequence, and the loss function.

[0109] After obtaining the network loss, we can calculate the gradient based on the target back propagation path according to the network loss and the state variables of the target network layer in the target state stored in the forward propagation process, and update the parameters of the neural network according to the calculated gradient to obtain the neural network of this iteration. This process is the back propagation in this iteration.

[0110] In the embodiments of the present disclosure, when the parameters of the neural network are gradient updated, the sample sequence can be input into the neural network, the processing of multiple processing states is performed in sequence, the sequence processing results are output, and the state variables of the target network layer in the target processing state are stored; then, the network loss of this iteration is determined according to the sequence processing results, and then the parameters of the neural network are gradient updated based on the back propagation path according to the network loss and the state variables of the target network layer in the target processing state, so as to obtain the neural network of this iteration. Since the back propagation path includes the target network layer in the target processing state, and the target processing state is a partial state among multiple processing states. Based on this, in the iterative training of the neural network, the state variables in the target processing state are stored and used, while the state variables of other processing states are not stored and used, thereby not only being able to significantly reduce the amount of cached data in the neural network training process, but also being able to reduce the amount of training calculations.

[0111] Figure 6 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 6 , the data processing method comprises:

[0112] Step S61, obtaining data to be processed;

[0113] Step S62, processing the data to be processed by a neural network to obtain a data processing result.

[0114] When performing data processing, in step S61, the data to be processed can be obtained, and the data to be processed is sequence data, which includes any one of voice, text, and video. Then in step S62, the data to be processed can be processed by a neural network to obtain a data processing result, wherein the neural network is a trained neural network, which is trained by the neural network training method described in any of the above embodiments.

[0115] In the embodiments of the present disclosure, when data processing is performed, the data to be processed can be obtained, and the data to be processed can be processed through a neural network to obtain a data processing result, thereby improving the efficiency of data processing.

[0116] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0117] In addition, the present disclosure also provides a neural network training device, a data processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any neural network training method or data processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0118] Figure 7 A block diagram of a neural network training device provided in an embodiment of the present disclosure.

[0119] Reference Figure 7 The present disclosure provides a neural network training device, the neural network training device comprising:

[0120] A strategy determination module 71, used to determine a target back propagation strategy from a plurality of preset back propagation strategies;

[0121] A path determination module 72, for any iteration in the neural network training, determines a target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes a portion of all back propagation paths;

[0122] The gradient update module 73 is used to perform gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

[0123] In some possible implementations, the neural network includes multiple network layers, each iteration in the neural network training includes multiple processing states, the target back-propagation path includes a target network layer under the target processing state, the target network layer is at least one of the multiple network layers, and the target processing state is at least one of the multiple processing states.

[0124] In some possible implementations, the parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration, including: inputting a sample sequence into the neural network, executing the processing of the multiple processing states in sequence, outputting a sequence processing result, and storing a state variable of the target network layer under the target processing state, the sample sequence including any one of speech, text, and video; determining the network loss of this iteration based on the sequence processing result; and gradient updating the parameters of the neural network based on the target back propagation path according to the network loss and the state variables of the target network layer under the target processing state to obtain the neural network of this iteration.

[0125] In some possible implementations, the target back propagation strategy is a dynamic random path strategy, which refers to a strategy for dynamically randomly generating a path in each iteration. The path determination module 72 includes:

[0126] A first random path determination submodule is used to determine a first random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the multiple processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the multiple network layers, wherein the first random path is randomly generated based on the connection graph before each iteration, and a starting node of the first random path is connected to a node for calculating a network loss;

[0127] The first reverse path determination submodule is used to determine the first random path as the target reverse propagation path for gradient update in this iteration.

[0128] In some possible implementations, the first random path determination submodule is used to: determine a first node based on a connection graph of the processing state, wherein the first node is a node for calculating network loss; starting from the first node, traverse the nodes connected to the first node with a preset probability, and determine the traversed nodes connected to the first node as second nodes; and determine the path containing the second node as the first random path.

[0129] In some possible implementations, the target back-propagation strategy is a dynamic random path strategy, and the path determination module 72 includes: a first quantity determination submodule, used to determine a first quantity of target processing states based on a preset first ratio and the number of the multiple processing states, wherein the first ratio refers to the ratio of the number of the target processing states to the number of the multiple processing states; a second random path determination submodule, used to determine a second random path from a connection graph of the processing states based on the first quantity, wherein the connection graph of the processing states is obtained by expanding the multiple processing states along the time dimension and the space dimension, wherein the time dimension is used to indicate the time order of the sample sequence, and the space dimension is used to indicate the connection order of the multiple network layers; a second reverse path determination submodule, used to determine the second random path as the target back-propagation path for gradient update in this iteration.

[0130] In some possible implementations, the target back-propagation strategy is a dynamic random path strategy, and the path determination module 72 includes: a proportion determination submodule, used to determine a second proportion corresponding to this iteration based on the variance of the network loss within a preset number of iterations and a preset coefficient; a second quantity determination submodule, used to determine the second number of target processing states based on the second proportion and the number of the multiple processing states; a third random path determination submodule, used to determine a third random path from a connection graph of the processing states based on the second number, the connection graph of the processing states being obtained by expanding the multiple processing states along the time dimension and the space dimension, the time dimension being used to indicate the time order of the sample sequence, and the space dimension being used to indicate the connection order of the multiple network layers; a third reverse path determination submodule, used to determine the third random path as the target back-propagation path for gradient update in this iteration.

[0131] In some possible implementations, the ratio determination submodule is used to: perform normalization processing on the product of the variance of the network loss within a preset number of iterations and a preset coefficient to obtain a second ratio.

[0132] In some possible implementations, the ratio determination submodule is used to: determine the derivative of the variance of the network loss within a preset number of iterations over time; and perform normalization processing on the product of the derivative and the preset coefficient to obtain a second ratio.

[0133] In some possible implementations, the proportion determination submodule is used to: perform normalization processing on the product of the variance of the network loss within a preset number of iterations and a preset coefficient to obtain a third proportion; determine the derivative of the variance of the network loss within a preset number of iterations over time; perform normalization processing on the product of the derivative and the preset coefficient to obtain a fourth proportion; determine the second proportion based on the third proportion, the fourth proportion, the first weight and the second weight, wherein the first weight is a weight coefficient of the third proportion, and the second weight is a weight coefficient of the fourth proportion.

[0134] In some possible implementations, the target back-propagation strategy is a preset shape path strategy, which refers to a strategy for determining a path according to a preset shape. The path determination module 72 includes: a target processing state selection submodule, which is used to select a target processing state from a connection diagram of processing states according to a preset shape, wherein the connection diagram of processing states is obtained by expanding the multiple processing states along the time dimension and the space dimension, wherein the time dimension is used to indicate the time order of the sample sequence, and the space dimension is used to indicate the connection order of the multiple network layers; a fourth reverse path determination submodule, which is used to determine the target back-propagation path for gradient update in this iteration according to the connection relationship of the target processing states in the connection diagram.

[0135] In some possible implementations, the target back-propagation strategy is a fixed random path strategy, which refers to a strategy in which a path is randomly generated and the path is fixed for each iteration. The path determination module 72 includes: a fourth random path determination submodule, used to determine the target random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the multiple processing states along the time dimension and the space dimension, wherein the time dimension is used to indicate the time order of the sample sequence, and the space dimension is used to indicate the connection order of the multiple network layers; a fifth reverse path determination submodule, used to determine the target random path as the target back-propagation path for each iteration of the neural network.

[0136] Figure 8 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0137] Reference Figure 8 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0138] A data acquisition module 81 is used to acquire data to be processed;

[0139] The data processing module 82 is used to process the data to be processed through a neural network to obtain a data processing result, wherein the neural network is trained by the neural network training method described in any of the above embodiments, and the data to be processed includes any one of voice, text, and video.

[0140] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0141] Reference Fig. 9 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502, and one or more I / O interfaces 503, connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501, so that the at least one processor 501 can execute the above-mentioned neural network training method or data processing method.

[0142] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned neural network training method or data processing method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0143] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned neural network training method or data processing method.

[0144] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0145] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0146] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0147] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0148] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0149] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0152] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0153] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A neural network training method, characterized in that: include: Determine a target back propagation strategy from among multiple preset back propagation strategies; For any iteration in the neural network training, determining a target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes a portion of all back propagation paths; The parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

2. The method according to claim 1, characterized in that The neural network includes multiple network layers, each iteration in the neural network training includes multiple processing states, the target back-propagation path includes a target network layer under the target processing state, the target network layer is at least one of the multiple network layers, and the target processing state is at least one of the multiple processing states.

3. The method according to claim 2, characterized in that The parameters of the neural network are gradient updated through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration, including: Inputting a sample sequence into the neural network, sequentially executing the processing of the multiple processing states, outputting a sequence processing result, and storing a state variable of the target network layer under the target processing state, wherein the sample sequence includes any one of speech, text, and video; Determine the network loss of this iteration according to the sequence processing result; According to the network loss and the state variables of the target network layer in the target processing state, based on the target back propagation path, the parameters of the neural network are gradient updated to obtain the neural network of this iteration.

4. The method according to claim 2, characterized in that: The target back propagation strategy is a dynamic random path strategy, which refers to a strategy for dynamically randomly generating a path in each iteration. Determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: Determine a first random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the plurality of processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the plurality of network layers, wherein the first random path is randomly generated based on the connection graph before each iteration, and a starting node of the first random path is connected to a node for calculating a network loss; The first random path is determined as the target back propagation path for gradient update in this iteration.

5. The method according to claim 4, characterized in that The step of determining a first random path from a connection graph of the processing state comprises: Determine a first node according to the connection graph of the processing state, wherein the first node is a node for calculating network loss; Starting from a first node, traversing nodes connected to the first node with a preset probability, and determining the traversed nodes connected to the first node as second nodes; A path including the second node is determined as a first random path.

6. The method according to claim 2, characterized in that The target back propagation strategy is a dynamic random path strategy, and determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: Determining a first number of target processing states according to a preset first ratio and the number of the plurality of processing states, wherein the first ratio refers to a ratio of the number of the target processing states to the number of the plurality of processing states; determining, according to the first number, a second random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the plurality of processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the plurality of network layers; The second random path is determined as the target back propagation path for gradient update in this iteration.

7. The method according to claim 2, characterized in that: The target back propagation strategy is a dynamic random path strategy, and determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: Determine a second ratio corresponding to this iteration according to the variance of the network loss within a preset number of iterations and a preset coefficient; determining a second number of target processing states according to the second ratio and the number of the plurality of processing states; determining, according to the second number, a third random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the plurality of processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the plurality of network layers; The third random path is determined as the target back propagation path for gradient update in this iteration.

8. The method according to claim 7, characterized in that The determining, according to the variance of the network loss within the preset number of iterations and the preset coefficient, the second ratio corresponding to the current iteration includes: A normalization process is performed on the product of the variance of the network loss within a preset number of iterations and a preset coefficient to obtain a second ratio.

9. The method according to claim 7, characterized in that: The determining, according to the variance of the network loss within the preset number of iterations and the preset coefficient, the second ratio corresponding to the current iteration includes: Determine the time-varying derivative of the variance of the network loss for a preset number of iterations; A normalization process is performed on the product of the derivative and the preset coefficient to obtain a second ratio.

10. The method according to claim 7, characterized in that The determining, according to the variance of the network loss within the preset number of iterations and the preset coefficient, the second ratio corresponding to the current iteration includes: Normalizing the product of the variance of the network loss within a preset number of iterations and a preset coefficient to obtain a third ratio; Determine the time-varying derivative of the variance of the network loss for a preset number of iterations; performing normalization processing on the product of the derivative and the preset coefficient to obtain a fourth ratio; The second ratio is determined according to the third ratio, the fourth ratio, a first weight and a second weight, wherein the first weight is a weight coefficient of the third ratio and the second weight is a weight coefficient of the fourth ratio.

11. The method according to claim 2, characterized in that The target back propagation strategy is a preset shape path strategy, which refers to a strategy for determining a path according to a preset shape. Determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: Selecting a target processing state from a connection graph of processing states according to a preset shape, wherein the connection graph of processing states is obtained by expanding the plurality of processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the plurality of network layers; According to the connection relationship of the target processing states in the connection graph, a target back propagation path for gradient update in this iteration is determined.

12. The method according to claim 2, characterized in that: The target back propagation strategy is a fixed random path strategy, which refers to a strategy in which a path is randomly generated and the path is fixed for each iteration. Determining the target back propagation path for gradient update in this iteration according to the target back propagation strategy includes: Determining a target random path from a connection graph of processing states, wherein the connection graph of processing states is obtained by expanding the plurality of processing states along a time dimension and a space dimension, wherein the time dimension is used to indicate a time order of a sample sequence, and the space dimension is used to indicate a connection order of the plurality of network layers; The target random path is determined as the target back propagation path at each iteration of the neural network.

13. A data processing method, characterized in that: include: Get the data to be processed; The data to be processed is processed by a neural network to obtain a data processing result. Wherein, the neural network is trained by the neural network training method described in any one of claims 1 to 12, and the data to be processed includes any one of voice, text, and video.

14. A neural network training device, characterized in that: The neural network includes a plurality of network layers, each iteration in the training of the neural network includes a plurality of processing states, and the apparatus includes: A strategy determination module, used to determine a target back propagation strategy from a plurality of preset back propagation strategies; A path determination module, for any iteration in the neural network training, determines a target back propagation path for gradient update in this iteration according to the target back propagation strategy, wherein the target back propagation path includes a portion of all back propagation paths; The gradient update module is used to perform gradient update on the parameters of the neural network through forward propagation and back propagation based on the target back propagation path to obtain the neural network of this iteration.

15. A data processing device, characterized in that: include: A data acquisition module, used to acquire data to be processed; A data processing module is used to process the data to be processed through a neural network to obtain a data processing result. Wherein, the neural network is trained by the neural network training method described in any one of claims 1 to 13, and the data to be processed includes any one of voice, text, and video.

16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the neural network training method according to any one of claims 1 to 12, or execute the data processing method according to claim 13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the neural network training method according to any one of claims 1 to 12, or the data processing method according to claim 13.

Images