A method and corresponding device for adjusting a neural network

Abstract

This application discloses a method for adjusting a neural network, applied in the training of AI models. This method obtains a first neural network containing multiple scale layers, which employs mixed-precision computation. Each scale layer has a scaling scale. In the backpropagation direction, the weight gradient of the operator at each scale layer is determined according to the corresponding scaling scale. For example, the scaling scale of the corresponding scale layer is adjusted based on the performance of the weight gradient in FP16 (e.g., whether there is INF or NAN). This effectively reduces the underflow rate of the FP16 gradient with minimal computation, allowing mixed-precision training to be well applied to neural network training. The technical solution provided in this application improves training efficiency while maintaining high training accuracy. Furthermore, the data using low-precision computation requires only a small amount of on-chip storage space, which also facilitates fast reading and writing of low-precision computation data by the chip.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method and apparatus for adjusting a neural network. Background Technology

[0002] Applying mixed precision to the training of neural networks refers to combining two or more operations of different precisions in the training process. For example, combining half-precision floating-point (FP16) and single-precision floating-point (FP32) to train a neural network can accelerate the training process by using FP16 while minimizing precision loss.

[0003] While FP16 can accelerate the training process, its narrow representation range, coupled with the numerous gradients generated during neural network training across a wide distribution, leads to a significant underflow rate. This underflow rate, especially as neural networks become more complex, generates even more smaller gradients, many of which exceed the lower bound of the FP16 representation range. This underflow rate negatively impacts the accuracy of neural network training. However, using only FP32 for training is problematic. While FP32 offers high precision, its data read and write speeds are significantly slower than FP16, impacting the chip's computing power. Furthermore, FP32 data requires substantial on-chip storage, hindering chip miniaturization.

[0004] Therefore, given the computing and storage capabilities of the chip, using mixed precision to train neural networks is a better choice. However, how to reduce the gradient underflow rate during mixed precision training has become a major challenge that needs to be overcome in the current mixed precision neural network training process. Summary of the Invention

[0005] This application provides a method for adjusting a neural network to reduce the underflow rate of gradients during training and improve the efficiency of neural network training. This application also provides corresponding devices, computer-readable storage media, computer program products, and chip systems.

[0006] The first aspect of this application provides a method for adjusting a neural network, comprising: obtaining a first neural network employing mixed-precision computation, the first neural network including multiple scale layers, wherein each scale layer has a scaling scale, the scaling scale of each scale layer refers to the scale used to amplify or reduce the gradient associated with each scale layer in the backpropagation direction when training the first neural network, the mixed-precision computation including first-precision computation; performing forward propagation processing on training samples input to the first neural network to obtain the value of a loss function; scaling the first gradient of the first operator in the target scale layer according to the scaling scale of the target scale layer in the backpropagation direction to obtain the second gradient of the first operator, the target scale layer being any one of the multiple scale layers, the first gradient of the first operator originating from the value of the loss function, the scaling operation being an amplification or reduction operation, the second gradient of the first operator being used to determine the weight gradient of each operator in the target scale layer; and adjusting the scaling scale of the target scale layer according to the performance of the weight gradient of each operator in the target scale layer within the expression range of the first-precision computation.

[0007] The neural network adjustment method provided in this application can be applied to the training process of neural networks. When training neural network models with different objectives, the adjustment method of this application can be used to adjust the scaling scale of each layer in the neural network. It should be noted that the solution provided in this application can be stored on the network (which may be in the cloud) in the form of a software package / plugin. Users can download and install it on their computer devices to execute the process of this application. Alternatively, the solution provided in this application can be provided to users in the form of cloud services, where users can upload the neural network to be trained to the cloud, and the cloud will use the solution provided in this application to train the neural network. Alternatively, the solution can be configured in a chip, and the computer device used for model training can install the chip to execute the process of this application. It can also be configured in a computer device, and the computer device can execute the process of this application when training the neural network. It should be understood that, apart from the above-mentioned methods of deploying the technical solution provided in this application, this application does not limit the specific form of using the technical solution provided in this application.

[0008] In this application, mixed-precision arithmetic refers to the combined application of two or more arithmetic operations of different precisions, such as in the training process of a neural network. The first precision operation can be a half-precision floating-point FP16 operation, and the second precision operation can be a single-precision floating-point FP32 operation. Of course, the first and second precision operations can also be other types of precision operations; for example, the first precision operation can be a single-precision floating-point FP32 operation, and the second precision operation can be a double-precision floating-point FP64 operation, as long as the precision of the second precision operation is higher than that of the first precision operation, meaning the range of expression for the second precision operation is greater than that for the first precision operation.

[0009] In this application, the first neural network refers to a neural network that has undergone scale layering, which can be obtained through automatic scale layering or manual scale layering.

[0010] In this application, a scale layer can be understood as a layer obtained by scale division. Each scale layer has a scaling scale, and the scaling scale of each scale layer is usually different. Of course, this application does not limit this, and the scaling scales of different scale layers can also be the same. The scaling scale refers to the scale by which the gradient associated with each scale layer in the backpropagation direction is amplified or reduced.

[0011] This application involves forward propagation and back propagation. Forward propagation refers to the process of processing the training samples input into the neural network until the value of the loss function (error loss) is obtained. Back propagation (BP) refers to using the loss function generated by forward propagation to update the parameters in the neural network. This process can involve determining the weight gradient of each layer operator through the value of the loss function, thereby updating the operator weights and bringing the error loss to converge. The back propagation algorithm is a back propagation process dominated by the loss value, aiming to obtain the optimal parameters of the neural network, such as the weight matrix.

[0012] In this application, the gradient associated with each scale layer includes the input gradient to the scale layer, the weight gradient used to update the weights, and the output gradient to be output to the scale layer.

[0013] In this application, the target scale layer may have one or more operators. The target scale layer includes at least the first operator mentioned above. If the target scale layer has multiple operators, then the multiple operators will have a logical order. The output of one operator may be used as the input of the next operator. The first operator refers to the operator that is first in the logical order among the multiple operators. Each operator in the target scale layer mentioned above includes all the operators in the target scale layer. If the target scale layer has only one operator, then each operator in the target scale layer mentioned above is the first operator mentioned above.

[0014] In this application, scaling and inverse scaling are two opposite operations. If scaling is an enlargement operation, then inverse scaling is a shrinking operation, and vice versa.

[0015] In this application, the first gradient can be understood as the input gradient, and the second gradient is the gradient after scaling according to the scaling scale of the target scale layer. The weight gradient of each operator in the target scale layer can be determined through the second gradient.

[0016] In this application, the first gradient of the first operator, derived from the value of the loss function, can be derived directly from the value of the loss function or indirectly from the value of the loss function. If the target scale layer is the first scale layer in the backpropagation direction, the first gradient of the first operator in the target scale layer can be obtained by differentiating the value of the loss function. In this case, the first gradient of the first operator in the target scale layer can be understood as directly derived from the value of the loss function. If the target scale layer is not the first scale layer in the backpropagation direction, the first gradient of the first operator in the target scale layer is obtained through the gradient output of the previous scale layer, and so on. The gradient output of the previous scale layer will be related to the first gradient of the first operator in the first scale layer in the backpropagation direction. In this case, the first gradient of the first operator in the target scale layer can be understood as indirectly derived from the value of the loss function.

[0017] In this application, using first precision arithmetic may include using a first precision type for calculation and / or storage, such as FP16 arithmetic which can be understood as using FP16 for calculation and / or storage, and using second precision arithmetic may include using a second precision type for calculation and / or storage, such as FP32 arithmetic which can be understood as using FP32 for calculation and / or storage.

[0018] In this application, the gradient underflow rate refers to the proportion of the number of gradients that go down beyond the expression range of a certain precision operation to the total number of gradients.

[0019] As can be seen from the first aspect above, during the training of a neural network, the first gradient of the first operator can be scaled according to the scaling scale of the scaling layer, thereby calculating the weight gradient of each operator in the scaling layer. Then, the performance of the weight gradient of each operator within the expression range of the first precision operation is observed, and the scaling scale of the corresponding scaling layer is adjusted. In this way, with a small amount of computation, the underflow rate of the gradient of the first precision operation can be effectively reduced. This allows mixed precision training to be well applied to the training of neural networks, improving training efficiency while maintaining high training precision. Moreover, the data using low precision operation only requires a small amount of storage space and running memory, which is also beneficial for the chip to quickly read and write low precision operation data, saving computational resources and costs when training neural networks.

[0020] The technical solution provided in this application can achieve the same accuracy as training a neural network using only high-precision hardware resources while maintaining comparable training efficiency, by combining low-precision and high-precision hardware resources. Alternatively, it can improve training efficiency by 1-3 times when training a neural network using the same mixed-precision hardware resources. In other words, training a neural network using the technical solution provided in this application can either save computational resources or shorten training time while ensuring training accuracy. The effect is particularly good when training low-level neural networks, heterogeneous neural networks, and deep neural networks.

[0021] In one possible implementation of the first aspect, the above steps: obtaining a first neural network employing mixed-precision operations include: receiving an initial neural network to be trained; marking a first type of operator in the initial neural network as employing first-precision operations to obtain a network employing mixed-precision operations, wherein a second type of operator in the network employing mixed-precision operations employs second-precision operations; and performing scale layering on the network employing mixed-precision operations to obtain the first neural network.

[0022] In this possible implementation, the initial neural network can be a neural network constructed using single-precision operations, such as ResNet50 or MobileNet. The initial neural network can be used to perform classification / recognition, compression / decompression, denoising, segmentation, enhancement, transformation, and feature extraction on various types of data, such as image / video data, audio data, and text data; this application does not limit this. The first type of operator can be a convolution (Conv) operator and / or a fully connected (FC) operator. The second type of operator can be all or some of the operators in the initial neural network except for the first type of operator, such as a normalization operator. Scale layering refers to configuring the layering of the initial neural network and the scaling scale of the corresponding layers manually or automatically. As can be seen from this approach, converting a single-precision neural network into a mixed-precision neural network and assigning different scaling scales to different network layers can greatly reduce the gradient underflow rate and stably train neural networks with a large dynamic range of gradient distribution in each network layer.

[0023] In one possible implementation of the first aspect, the above steps: scale-layering the network using mixed-precision computation, include: obtaining the initial scale of each network layer in the network using mixed-precision computation; merging network layers with the same initial scale to obtain the above first neural network.

[0024] In this possible implementation, the layers in the network before scale-based layering can be called network layers. Each network layer can have an initial scale, which can be obtained through training or configuration. Merging network layers with the same initial scale results in a merged network, which can then serve as the first neural network. This method of merging network layers with the same initial scale can further improve the efficiency of neural network training.

[0025] In one possible implementation of the first aspect, the above steps: performing scale layering on the network employing mixed-precision computation, include: obtaining the initial scale of each network layer in the network employing mixed-precision computation; merging network layers with the same initial scale to obtain a layer-merged network; merging the first scaling operation of the output interface of the first network layer and the second scaling operation of the input interface of the second network layer in the layer-merged network to obtain a first neural network, wherein the first network layer and the second network layer are adjacent, and the first network layer is the layer preceding the second network layer in the backpropagation direction.

[0026] Compared to the previous possible implementation, this approach further merges the scaling and inverse scaling operations of adjacent layers. For example, if the output of layer 1 needs to perform a scaling down operation of scale S, and the input of layer 2 needs to perform a scaling up operation of scale M, these two operations can be merged into a scaling operation of M / S. This method effectively reduces one scaling operation and one inverse scaling operation to a single scaling operation. This merged scaling operation can be performed at the output of the previous scale layer or at the input of the next scale layer. This implementation merges the scaling and inverse scaling operations of adjacent layers, reducing computational steps and further improving the efficiency of neural network training.

[0027] In one possible implementation of the first aspect, the above step of obtaining the initial scale of each network layer in the network employing mixed-precision computation includes: determining the initial scale of each network layer in the network employing mixed-precision computation according to a preset underflow rate, or receiving configuration information for configuring the initial scale of each network layer in the network employing mixed-precision computation, and determining the initial scale of each network layer in the network employing mixed-precision computation according to the configuration information.

[0028] In one possible implementation, the initial scale of each network layer can be automatically determined based on a preset underflow rate, which is an automatic scale-based layering approach. Alternatively, it can receive user-configured initial scale information for each network layer and then determine the initial scale of each network layer based on the configuration information, which is a manual scale-based layering approach. Therefore, this application provides diverse scale-based layering methods, improving the flexibility of scale-based layering.

[0029] In one possible implementation of the first aspect, the above steps, including determining the initial scale of each network layer in the network employing mixed-precision computation based on a preset underflow rate, include:

[0030] With the preset underflow rate as the target, different training samples are input to determine multiple sets of scale values ​​for each network layer, and the average of the multiple sets of scale values ​​for each network layer is the initial scale of each network layer.

[0031] In this possible implementation, when automatically determining the initial scale, a preset underflow rate can be used as the target. Multiple sets of scale values ​​for each network layer are calculated using different training samples, and then the average value is taken as the initial scale for each network layer. This can improve the accuracy of the initial scale.

[0032] In one possible implementation of the first aspect, the above steps include: adjusting the scaling scale of the target scale layer based on the performance of the weight gradient of each operator in the target scale layer within the expression range of the first precision operation, including: reducing the scaling scale of the target scale layer when the weight gradient of each operator in the target scale layer includes infinitely large values ​​or invalid numbers; and increasing the scaling scale of the target scale layer when the weight gradient of each operator in the target scale layer is within the expression range of the first precision operation.

[0033] In this possible implementation, when the weight gradient of each operator includes infinitely large values ​​or invalid numbers (invalid numbers can be fractions with a denominator of 0, etc.), it indicates that the scaling scale of the current target scale layer is too large and needs to be reduced. If the weight gradient of each operator is within the expression range of the first-precision operation, the scaling scale of the target scale layer can be further increased. This helps to find the optimal scaling scale for each scale layer, thereby improving the convergence speed of the neural network.

[0034] In one possible implementation of the first aspect, when the weight gradient of each operator is within the expression range of the first precision operation, the method further includes: performing an inverse scaling operation on the weight gradient of each operator in the target scale layer according to the scaling scale of the target scale layer, so as to obtain the weight gradient of each operator after the inverse scaling operation in the target scale layer; and updating the weight of each operator in the target scale layer according to the weight gradient of each operator after the inverse scaling operation in the target scale layer.

[0035] In this possible implementation, since the weight gradient of each operator in the target scale layer is calculated using the second gradient of the first operator obtained after scaling, when updating the weights, it is necessary to first perform an inverse scaling operation on the weight gradient of each operator, and then update the weight of each operator. This is more conducive to the convergence of the neural network.

[0036] In one possible implementation of the first aspect, the method further includes: determining the output gradient of the target scale layer based on the second gradient of the first operator; and performing an inverse scaling operation on the output gradient of the target scale layer according to the scaling scale of the target scale layer to obtain the output gradient of the target scale layer.

[0037] In this possible implementation, since the gradient to be output in the target scale layer is obtained by the second gradient of the first operator after scaling, the gradient to be output needs to be inversely scaled according to the corresponding scaling scale to obtain a suitable output gradient for the calculation of the next scale layer.

[0038] In one possible implementation of the first aspect, the method further includes: when the output gradient of the target scale layer is an infinitely large value or an invalid number, correcting the output gradient of the target scale layer to a valid value within the expression range of the first precision operation, and transmitting the corrected output gradient of the target scale layer to the adjacent scale layer of the target scale layer.

[0039] In this possible implementation, if the output gradient of the target scale layer is an infinite value or an invalid number, it means that the output gradient is not applicable to the weight update of the operator in each scale layer. In this case, the weight update step is skipped directly. However, in order not to affect the subsequent calculation process, the output gradient can be corrected to a valid value within the expression range of the first precision operation and then transmitted to the next scale layer in the backpropagation direction for calculation. This is beneficial to improving the scale update efficiency of the neural network.

[0040] In one possible implementation of the first aspect, the method further includes: during the forward propagation process, if the feature values ​​of the target scale layer include infinitely large values ​​or invalid numbers, then the update of the target scale layer is skipped.

[0041] In this possible implementation, the feature values ​​of the target scale layer are generated at each scale layer during the forward propagation process. During the forward propagation process, it is also possible to determine whether to update the operator weights in that scale layer based on the performance of the feature values ​​within the expression range of the first precision operation. If the forward features include infinite values ​​or invalid numbers, there is no need to update the operator weights. This is beneficial to improving the training efficiency of the neural network.

[0042] In one possible implementation of the first aspect, the method further includes: when the training of the first neural network reaches a preset condition, re-scaling the first neural network to obtain a second neural network.

[0043] In this possible implementation, the preset condition could be that the number of training iterations has reached a certain threshold, such as 300 training cycles, or that the neural network has been trained to a certain extent, such as when the difference in scaling scales between different scale layers is less than a preset value. In this case, the first neural network can be rescaled (the scaling method can be understood from the previous description), resulting in a new second neural network, which is then trained. This dynamic scaling layer update method can improve the training efficiency of the neural network.

[0044] A second aspect of this application provides a neural network adjustment device that has the function of implementing the method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, such as an acquisition unit, a first processing unit, a second processing unit, and a third processing unit. These units can be implemented by one or more processing units.

[0045] A third aspect of this application provides a computer device including at least one processor, a memory, an input / output (I / O) interface, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0046] The fourth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the one or more processors perform a method as described in the first aspect or any possible implementation thereof.

[0047] The fifth aspect of this application provides a computer program product storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by one or more processors, the one or more processors execute a method as described in the first aspect or any possible implementation thereof.

[0048] A sixth aspect of this application provides a chip system including at least one processor for supporting adjustment means of a neural network to implement the functions involved in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory containing program instructions and data necessary for the adjustment means of the neural network. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of a neural network processor provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of an adjusted architecture of a neural network provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the training process of the neural network provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of an embodiment of the neural network adjustment method provided in this application;

[0053] Figure 5 This is a structural diagram of FP16 and FP32;

[0054] Figure 6 This is a schematic diagram of another embodiment of the neural network adjustment method provided in this application;

[0055] Figure 7 This is a schematic diagram of an embodiment of scale layering provided in this application;

[0056] Figure 8 This is a schematic diagram of another embodiment of scale layering provided in this application;

[0057] Figure 9 This is a schematic diagram of another embodiment of scale layering provided in this application;

[0058] Figure 10 This is a schematic diagram of another embodiment of the neural network adjustment method provided in this application;

[0059] Figure 11 This is a schematic diagram of another embodiment of the neural network adjustment method provided in this application;

[0060] Figure 12 This is a schematic diagram of another embodiment of the neural network adjustment method provided in this application;

[0061] Figure 13 This is a schematic diagram of another embodiment of the neural network adjustment method provided in this application;

[0062] Figure 14 This is a schematic diagram of the structure of the neural network adjustment device provided in an embodiment of this application;

[0063] Figure 15 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0064] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will understand, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0066] This application provides a method for adjusting a neural network to reduce the underflow rate of gradients during training and improve the efficiency of neural network training. This application also provides corresponding devices, computer-readable storage media, computer program products, and chip systems. These will be described in detail below.

[0067] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0068] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0069] Intelligent manufacturing, intelligent transportation, smart home, smart healthcare, intelligent security, autonomous driving, smart city, intelligent terminal, etc.

[0070] Neural network models are typically trained on the model owner's computer device or platform (such as a server, virtual machine (VM), or container). The trained model is stored as a model file. When a user's device (such as a terminal device, server, edge device, VM, or container) needs to use the model, the user's device either actively loads the model file or the model owner's device actively sends the model file to the user's device for installation, thus enabling the model to be applied on the user's device and perform the corresponding function.

[0071] A server refers to a physical machine.

[0072] Terminal equipment (also known as user equipment, UE) is a device with wireless transceiver capabilities. It can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on water (such as on ships); and it can be deployed in the air (such as on airplanes, balloons, and satellites). The terminal can be a mobile phone, tablet, computer with wireless transceiver capabilities, virtual reality (VR) terminal, augmented reality (AR) terminal, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical care, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc.

[0073] Both VMs and containers are virtualized devices created by virtualizing the hardware resources of a physical machine.

[0074] A neural network model refers to a neural network built for one or more business objectives.

[0075] In this embodiment of the application, the neural network can be composed of neural units, and a neural unit can refer to a neural unit represented by x. s The operation unit takes the intercept b as input, and the output of this operation unit can be:

[0076]

[0077] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next layer of neural units. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0078] The neural network in this application embodiment can be a deep neural network (DNN), a convolutional neural network (CNN), or other types of neural networks. A brief introduction to deep neural networks and convolutional neural networks is given below.

[0079] 1. Deep Neural Networks

[0080] Deep neural networks (DNNs) can be understood as neural networks with many hidden layers. There's no specific metric for "many" layers; the commonly used terms "multi-layered neural network" and "deep neural network" are essentially the same. DNNs can be categorized into three layers based on their position: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs seem complex, the operation of each layer is actually not complicated; it can be simply described by the following linear relationship expression: ,in, It is the input vector. It is the output vector. It is an offset vector. It is the weight matrix (also called coefficients). It's an activation function. Each layer simply applies the input vector... The output vector is obtained after the above operations. Because DNNs have many layers, the coefficients... and offset vector The quantity will be even greater.

[0081] 2. Convolutional Neural Networks

[0082] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units on the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations in an image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0083] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0084] The training of neural networks can be achieved using AI chips, such as neural network processors (NPUs). Figure 1 As shown, the neural network processor 10 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor is the arithmetic circuit 103, which is controlled by the controller 104 to retrieve matrix data from the memory and perform multiplication operations.

[0085] In some implementations, the arithmetic circuit 103 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 103 is a two-dimensional pulsating array. The arithmetic circuit 103 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 103 is a general-purpose matrix processor.

[0086] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 102 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 101 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 108.

[0087] Unified memory 106 is used to store input and output data. Weight data is directly accessed via the Direct Memory Access Controller (DMAC) 105 and transferred to the weight memory 102. Input data is also transferred to unified memory 106 via DMAC.

[0088] The Bus Interface Unit (BIU) 110 is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer 109.

[0089] The bus interface unit 110 is used for the instruction fetch memory 109 to fetch instructions from external memory, and also for the memory access controller 105 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0090] The DMAC is mainly used to move input data from external memory DDR to unified memory 106, or to weight data to weight memory 102, or to input data to input memory 101.

[0091] The vector computation unit (107) comprises multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computation in non-convolutional / FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0092] In some implementations, the vector computation unit 107 can store the processed output vector into a unified buffer 106. For example, the vector computation unit 107 can apply a nonlinear function to the output of the arithmetic circuit 103, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 107 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 103, for example, for use in subsequent layers of a neural network.

[0093] The instruction fetch buffer 109 connected to the controller 104 is used to store the instructions used by the controller 104.

[0094] The unified memory 106, input memory 101, weight memory 102, and instruction fetch memory 109 are all on-chip memories. External memory is proprietary to the hardware architecture of this neural network processor.

[0095] The above introduced neural networks, deep neural networks, convolutional neural networks, and AI chips that may be used for neural network training. The following section will combine... Figure 2 This section introduces the forward propagation and backward propagation processes involved in neural network training.

[0096] like Figure 2 The neural network shown includes four layers (here, only four are shown). Figure 2 Listing a neural network (which can actually include many layers), specifically layer 1, layer 2, layer 3, and layer 4, the structure of each layer and the relationships between layers can be found in [reference needed]. Figure 2 To understand the content in the example below, of course... Figure 2 The example below is just one example and is not limited to this kind of hierarchical relationship.

[0097] The forward propagation process refers to processing the training samples input into the neural network layer by layer, such as first processing through layer 1, then layer 2, layer 3, and so on until layer 4 outputs the value of the loss function, which can also be called the error loss.

[0098] Back propagation (BP) refers to using the value of the loss function generated during forward propagation to update the parameters in a neural network. This process involves determining the weight gradient of each operator by using the value of the loss function, thereby updating the operator weights and ultimately converging the error loss. Figure 2 In this model, backpropagation proceeds from layer 4 to layer 3, then to layer 2, and finally to layer 1. The backpropagation algorithm is a loss function-driven backpropagation process aimed at obtaining the optimal parameters of the neural network, such as the weight matrix.

[0099] The neural network adjustment method provided in this application embodiment can be applied to, for example, Figure 3 During the training process of the neural network shown, as Figure 3As shown, computer equipment can train an initial neural network using training samples. Through multiple rounds of training, a target neural network can be obtained. This trained target neural network can then be applied to relevant terminal devices for business applications. During the training of the initial neural network, a first neural network can be obtained through scale layering, and then the scaling of the scale layers in the first neural network can be adjusted.

[0100] It should be noted that the solution provided in this application can be stored on the network (which may be in the cloud) in the form of a software package / plugin. Users can download and install it on their computer devices to execute the process of this application. Alternatively, the solution can be provided to users as a cloud service, allowing users to upload the neural network to be trained to the cloud, where the cloud will train the neural network using the solution provided in this application. Alternatively, the solution can be configured in a chip, and the computer device used for model training can execute the process of this application by installing the chip. Or, the solution can be configured in a computer device, and the computer device can execute the process of this application when training the neural network. It should be understood that, apart from the above-described methods of deploying the technical solution provided in this application, this application does not limit the specific form of using the technical solution provided in this application.

[0101] The following is combined Figure 4 This application introduces a method for adjusting a neural network based on embodiments.

[0102] like Figure 4 As shown, one embodiment of the neural network adjustment method provided in this application includes:

[0103] 201. A computer device acquires a first neural network employing mixed-precision computation, the first neural network comprising multiple scale layers, wherein each scale layer has a scaling scale.

[0104] The scaling scale for each scale layer refers to the scale by which the gradients associated with each scale layer in the backpropagation direction are amplified or reduced when training the first neural network.

[0105] Mixed-precision arithmetic refers to the combined application of two or more operations with different precisions, such as in the training of neural networks. Mixed-precision arithmetic includes first-precision arithmetic and second-precision arithmetic, with the second-precision arithmetic having a wider range of expression than the first-precision arithmetic.

[0106] First-precision arithmetic can be half-precision floating-point FP16 arithmetic, and second-precision arithmetic can be single-precision floating-point FP32 arithmetic. Of course, first-precision and second-precision arithmetic can also be other types of precision arithmetic. For example, first-precision arithmetic can be single-precision floating-point FP32 arithmetic, and second-precision arithmetic can be double-precision floating-point FP64 arithmetic, as long as the precision of the second-precision arithmetic is higher than that of the first-precision arithmetic, meaning the range of expression for the second-precision arithmetic is greater than that for the first-precision arithmetic. It should be understood that first-precision arithmetic can also be lower than FP16, such as FP8, INT8, etc.

[0107] Using first-precision arithmetic can include performing calculations and / or storing data using first-precision types, such as FP16 arithmetic, which can be understood as performing calculations and / or storing data using FP16. Using second-precision arithmetic can include performing calculations and / or storing data using second-precision types, such as FP32 arithmetic, which can be understood as performing calculations and / or storing data using FP32.

[0108] For information on FP16 and FP32, please refer to [link / reference]. Figure 5 To understand, such as Figure 5 As shown, FP16 is a half-precision floating-point number, using 1 bit to represent the sign, such as... Figure 5 In FP16, the corresponding bit is 15, and the exponent is represented by 5 bits, such as... Figure 5 In FP16, bits 10 to 14 are used, with 10 bits representing the decimal, such as... Figure 5 In FP16, the bits are 0 to 9; FP32 is a single-precision floating-point number, using 1 bit to represent the sign, such as... Figure 5 In FP32, bit 31 corresponds to the exponent, which is represented by 8 bits, such as... Figure 5 In FP32, bits 23 to 30 are used, with bit 23 representing the decimal part, such as... Figure 5 In FP32, the corresponding bits are 0 to 22. The data range of FP16 is (6 × 10^6)^22. -8 Up to 65504), the data range of FP32 is (1.4 × 10⁻⁶). -45 Up to 1.7×10 38 FP32 and FP16 represent different data ranges. In big data computing, FP16 has the risk of overflow.

[0109] The gradient underflow rate refers to the proportion of gradients that fall below the expression range of a certain precision operation to the total number of gradients.

[0110] The first neural network refers to a neural network that has undergone scale-based layering, which can be obtained through automatic or manual scale-based layering.

[0111] A scale layer can be understood as a layer obtained by dividing the data into scales. Each scale layer has a scaling scale, which is usually different for each scale layer. However, this application does not impose any restrictions on this, and different scale layers can also have the same scaling scale. The scaling scale refers to the scale by which the gradient associated with each scale layer in the backpropagation direction is amplified or reduced.

[0112] The gradients associated with each scale layer include the input gradients to the scale layer, the weight gradients used to update the weights, and the output gradients to be output to the scale layer.

[0113] 202. The computer device performs forward propagation processing on the training samples input to the first neural network to obtain the value of the loss function. Depending on the specific task performed by the first neural network, an appropriate loss function can be selected. For example, the loss function used by the first neural network may include, but is not limited to, the following types: 0-1 loss, hinge loss, softmax loss, logistic loss, cross entropy, softmax cross entropy, triplet loss, mean squared error (MSE), mean absolute error (MAE), smoothed L1 loss, L1 loss, L2 loss, center loss, etc.

[0114] 203. In the backpropagation direction, the computer device scales the first gradient of the first operator in the target scale layer according to the scaling scale of the target scale layer to obtain the second gradient of the first operator.

[0115] The target scale layer is any one of the multiple scale layers. The first gradient of the first operator comes from the value of the loss function. The scaling operation is either an enlargement operation or a shrinking operation. The second gradient of the first operator is used to determine the weight gradient of each operator in the target scale layer.

[0116] There may be one or more operators in the target scale layer. If there are multiple operators, then the multiple operators will have a logical order. The output of one operator may be used as the input of the next operator. The first operator refers to the operator that is logically first among the multiple operators.

[0117] The first gradient can be understood as the input gradient, while the second gradient is the gradient after scaling according to the target scale layer. The weight gradient of each operator in the target scale layer can be determined through the second gradient.

[0118] The first gradient of the first operator can be derived from the value of the loss function, either directly or indirectly. If the target scale layer is the first scale layer in the backpropagation direction, the first gradient of the first operator in that target scale layer can be obtained by differentiating the value of the loss function. In this case, the first gradient of the first operator in the target scale layer can be understood as directly derived from the value of the loss function. If the target scale layer is not the first scale layer in the backpropagation direction, the first gradient of the first operator in the target scale layer is obtained through the gradient output of the previous scale layer, and so on. The gradient output of the previous scale layer will be related to the first gradient of the first operator in the first scale layer in the backpropagation direction. In this case, the first gradient of the first operator in the target scale layer can be understood as indirectly derived from the value of the loss function.

[0119] 204. The computer device adjusts the scaling of the target scale layer based on the performance of the weight gradient of each operator in the target scale layer within the expression range of the first precision operation.

[0120] In this embodiment, during neural network training, the first gradient of the first operator can be scaled according to the scaling scale of the scaling layer, thereby calculating the weight gradient of each operator in the scaling layer. Then, the performance of the weight gradient of each operator within the expression range of the first precision operation is observed, and the scaling scale of the corresponding scaling layer is adjusted. In this way, the underflow rate of the gradient of the first precision operation can be effectively reduced with a small amount of computation. This allows mixed precision training to be well applied to the training of neural networks, improving training efficiency while maintaining high training precision. Moreover, the data using low precision operation only requires a small amount of storage space on the chip, which is also beneficial for the chip to quickly read and write the data of low precision operation.

[0121] The neural network adjustment scheme provided in this application embodiment can be referred to... Figure 6 To understand, such as Figure 6 As shown, the scheme includes converting the initial neural network model to mixed precision to obtain a network using mixed precision computation, and then scaling the mixed precision network to obtain a first neural network. Forward propagation is performed on the first neural network to obtain the loss function value; then, backpropagation is performed based on the loss function value. During backpropagation, intra-layer trial and error within the scaling layers and scaling adjustments are performed. Furthermore, scaling adjustments can be made to the scaling layers during both forward and backpropagation.

[0122] In other words, the above-mentioned method for adjusting the neural network can include the following parts: First, scale the initial neural network to obtain the first neural network; second, adjust the scaling of the scale layer through trial and error within the layer; and third, optimize the adjustment process through inter-layer correction.

[0123] First, scale-layer the initial neural network to obtain the first neural network.

[0124] In this embodiment of the application, after receiving the initial neural network to be trained, the computer device can mark the first type of operator in the initial neural network as using the first precision operation, and default the second type of operator in the initial neural network to use the second precision operation, so as to obtain a network using mixed precision operation; then the network using mixed precision operation is scaled to obtain the first neural network.

[0125] In this embodiment, the initial neural network can be a neural network constructed using single-precision operations, such as ResNet50, MobileNet, etc. The first type of operator can be a convolution (Conv) operator and / or a fully connected (FC) operator, and the second type of operator can be all or some of the operators in the initial neural network except for the first type of operator, such as the normalization operator.

[0126] The scale stratification schemes can include automatic scale stratification and manual scale stratification, which will be introduced below.

[0127] 1. Automatic scale stratification.

[0128] The process of automatic scale stratification can be found in [reference needed]. Figure 7 To understand, such as Figure 7 As shown,

[0129] 301. With a preset underflow rate as the target, input different training samples to determine multiple sets of scale values ​​for each network layer. The average of the multiple sets of scale values ​​for each network layer is the initial scale of each network layer.

[0130] Of course, step 301 is only one way to calculate the initial scale of a network layer, and the initial scale of a network layer can also be determined in other ways.

[0131] It should be noted that the training sample in this application embodiment refers to a batch of samples, not a single training sample. Figure 7 Training sample 1, training sample 2, ..., training sample n are all samples from different batches.

[0132] Each time a training sample is input, a set of scale values ​​is generated with a preset underflow rate as the target. This set of scale values ​​includes one scale value for each network layer. For example, inputting training sample 1 will generate set 1. If the network using mixed precision computation has m network layers, then set 1 will include scale value 11 for network layer 1, scale value 21 for network layer 2, ..., scale value m1 for network layer m. Similarly, inputting training sample 2 will generate set 2, which includes scale value 12 for network layer 1, scale value 22 for network layer 2, ..., scale value m2 for network layer m. And so on. Inputting training sample n will generate set n, which includes scale value 1n for network layer 1, scale value 2n for network layer 2, ..., scale value mn for network layer m. The initial scale of each network layer can be obtained by simply adding up the n scale values ​​of that network layer and then averaging them. For example: the initial scale of network layer 1 = (scale value 11 + scale value 12, ..., + scale value 1n) / n; the initial scale of network layer 2 = (scale value 21 + scale value 22, ..., + scale value 2n) / n; ...; the initial scale of network layer m = (scale value m1 + scale value m2, ..., + scale value mn) / n.

[0133] 302. Merge network layers with the same initial scale to obtain a layer-merged network.

[0134] In this embodiment of the application, the layer merging network can be directly used as the first neural network, or step 303 can be performed based on the layer merging network.

[0135] The process of layer merging can be found in [reference]. Figure 8 To understand, such as Figure 8 As shown, the network using mixed-precision computing comprises m network layers, namely network layer 1, network layer 2, network layer 3, ..., network layer m; where the initial scale of network layer 1 is a, the initial scale of network layer 2 is a, the initial scale of network layer 3 is b, ..., the initial scale of network layer m is f. Since the initial scale of network layer 1 and network layer 2 is the same, they can be merged. In this example, only the initial scales of network layer 1 and network layer 2 are the same; the initial scales of other network layers are different. Therefore, only network layer 1 and network layer 2 are merged. If other network layers have the same initial scale, they can be merged using the same merging scheme as network layer 1 and network layer 2. The merged network can be called a layer-merged network, which can be directly used as the first neural network. Figure 8 The first neural network on the right contains (m-1) scale layers. Among the (m-1) scale layers, the scaling scale of scale layer 1 is a, the scaling scale of scale layer 2 is b, ..., the scaling scale of scale layer (m-1) is f. Figure 8 In the first neural network on the right, dashed arrows represent backpropagation, and solid arrows represent forward propagation.

[0136] 303. The first scaling operation of the output interface of the first network layer and the second scaling operation of the input interface of the second network layer in the layer merging network are merged to obtain a first neural network, wherein the first network layer and the second network layer are adjacent, and the first network layer is the layer preceding the second network layer in the back propagation direction.

[0137] In this embodiment of the application, the process from the layer merging network to the first neural network can be referred to. Figure 9 To understand, such as Figure 9 As shown, based on Figure 8 The merging process on the left can be obtained Figure 9 The layer merging network on the left has layers 1-2 with an initial scale of 'a' and layer 3 with an initial scale of 'b'. If the input interface of layer 1-2 performs a scaling operation of 'a', then the first scaling operation is a scaling down operation of 'a'. If the input interface of layer 3 performs a scaling operation of 'b', then the second scaling operation is a scaling up operation of 'b'. The scaling down operation of layer 1-2 ('a') and the scaling up operation of layer 3 ('b') can be merged. This merging of the two operations is equivalent to performing a scaling up operation of 'b / a', thus obtaining... Figure 9 The first neural network on the right. In this first neural network, the scaling scale of scale layer 1 is a, and the scaling scale of scale layer 2 is still b. However, only one scaling operation of b / a needs to be performed at the output interface of scale layer 1 or the input interface of scale layer 2. Similarly, the scaling scale of scale layer (m-1) is still f, but only one scaling operation of f / e needs to be performed at the output interface of scale layer (m-2) or the input interface of scale layer (m-1), where e is the initial scale of the layer before network layer m. Figure 9 In the first neural network on the right, dashed arrows represent backpropagation, and solid arrows represent forward propagation.

[0138] This example demonstrates that when using scaling operations to merge networks as the first neural network, one scaling operation and one inverse scaling operation of the target scale layer are reduced to one scaling operation, reducing computational steps and further improving the efficiency of neural network training.

[0139] It should be noted that a single scaling operation after merging can be executed at the output interface of the previous scale layer or at the input interface of the next scale layer.

[0140] 2. Manual scale layering.

[0141] The initial scale of network layers in automatic scaling is determined based on a preset underflow rate for each layer in a network using mixed-precision computation. Manual scaling, on the other hand, receives configuration information specifying the initial scale for each layer in a network using mixed-precision computation, and then determines the initial scale based on this configuration information. Layer merging and scaling operations are described in the same way as in the automatic scaling section; please refer to that section for further understanding. They will not be repeated here.

[0142] In this embodiment, the neural network with single-precision operation is converted into a neural network with mixed-precision operation, and different scaling scales are assigned to different network layers. This can greatly reduce the underflow rate of gradients and enable stable training of neural networks with a large dynamic range of gradient distribution in each network layer.

[0143] 2. Adjust the scaling of the scale layer through trial and error within the layer.

[0144] In the backpropagation direction, the process of scaling the first neural network obtained in step 302 or step 303 is slightly different, and will be described separately below.

[0145] 1. The layer merging network of step 302 is used as the first neural network.

[0146] like Figure 10 As shown, in this first neural network, the scale layers in the backpropagation direction are scale layer 1, scale layer 2, ..., scale layer (m-1).

[0147] Scale layer 1 includes three operators: operator 1, operator 2, and operator 3. The logical relationship between the three operators is as follows: Figure 10 As shown, the output of operator 1 is the input of operator 2, and the output of operator 2 is the input of operator 3. Operator 1 is the first operator of scale layer 1. When the target scale layer is scale layer 1, the first gradient of operator 1 can be obtained by differentiating the value of the loss function. Then, a scaling operation is performed on the first gradient according to the scaling scale 'a' of scale layer 1. In this embodiment, taking the scaling operation as an amplification operation as an example, the first gradient is amplified by a factor of 'a' to obtain the second gradient of operator 1. Then, the weight gradient S1 and output gradient O1 of operator 1 are calculated using the second gradient. The output gradient O1 of operator 1 is output to operator 2. The weight gradient S2 of operator 2 is calculated using the output gradient O1, and the output gradient O2 of operator 2 is also calculated. The output gradient O2 of operator 2 is output to operator 3. The weight gradient S3 and output gradient O3 of operator 3 are calculated using the output gradient O2 of operator 2.

[0148] If the weight gradient of each operator in the target scale layer includes infinite (INF) or invalid (Not a number, NAN) values, then reduce the scaling scale of the target scale layer, for example, by 1 / a based on the scaling scale a. The adjusted scaling factor is b×1 / When the weight gradient of each operator is within the expression range of the first precision operation, the scaling scale of the target scale layer is increased, such as increasing it by 2 based on the scaling scale a. （1 / 1000） The adjusted scaling factor is b×2 （1 / 1000） .

[0149] It should be noted that the examples listed here are obtained by passing through b×1 / , and through b×2 （1 / 1000） Obtaining the adjusted scaling scale is just one method. Other methods can also be used to decrease or increase the scaling scale, such as subtracting a value from the current scaling scale or adding a value to it. Of course, other methods can also be used; anything that can decrease or increase the scaling scale is applicable to the scaling scale adjustment in this application.

[0150] In this embodiment, when the weight gradient of each operator includes infinitely large values ​​or invalid numbers (invalid numbers can be fractions with a denominator of 0, etc.), it indicates that the scaling scale of the current target scale layer is too large and needs to be reduced. When the weight gradient of each operator is within the expression range of the first precision operation, the scaling scale of the target scale layer can be further increased to attempt to expand it further. This helps to find the optimal scaling scale suitable for each scale layer, thereby improving the convergence speed of the neural network.

[0151] When the weight gradient of each operator in the target scale layer is within the expression range of the first precision operation, the weight gradient of each operator in the target scale layer is subjected to an inverse scaling operation according to the scaling scale of the target scale layer, so as to obtain the weight gradient of each operator in the target scale layer after the inverse scaling operation; the weight of each operator in the target scale layer is updated according to the weight gradient of each operator in the target scale layer after the inverse scaling operation.

[0152] Combination Figure 10In scale layer 1, when the weight gradients S1 of operator 1, S2 of operator 2, and S3 of operator 3 are all within the expression range of the first precision operation, the weight gradients S1, S2, and S3 are subjected to inverse scaling operation according to the scaling scale a of scale layer 1, that is, the operation of scaling down by a factor of a is performed, and the corresponding scaled weight gradients U1, U2, and U3 are obtained respectively. Furthermore, the weights of operator 1 can be updated according to the weight gradient U1, the weights of operator 2 can be updated according to the weight gradient U2, and the weights of operator 3 can be updated according to the weight gradient U3.

[0153] In addition, in this embodiment, the output gradient of the target scale layer can be determined based on the second gradient of the first operator; according to the scaling scale of the target scale layer, an inverse scaling operation is performed on the output gradient of the target scale layer to obtain the output gradient of the target scale layer. Combined with... Figure 10 The output gradient O3 of operator 3 can be used as the gradient to be output, and then the output gradient O3 can be reduced by a factor of a to obtain the output gradient to be output to scale layer 2, and then transmitted to scale layer 2.

[0154] The execution process of the scheme in the embodiment of this application in scale layer 2 is basically the same as that in scale layer 1. The difference is that the first gradient of the first operator in scale layer 2 is the output gradient of scale layer 1, which is received, instead of the first gradient being calculated directly by the value of the loss function in operator 1.

[0155] In the embodiments of this application, the process of adjusting the scaling scale in the scale layer can also be referred to... Figure 11 To understand.

[0156] like Figure 11 As shown, the process includes:

[0157] 401. Initialize the abnormal state bits in the first neural network to 0.

[0158] 402. Iterate through the operators in each scale layer and determine the weight gradient of each operator.

[0159] For example: Traverse the operators in the i-th layer, determine the weight gradient of each operator, and initially i=1, where i represents the layer number of the scale layer. If the scale layer number starts from 0, then the initial i=0.

[0160] The process of determining the weight gradient of each operator can be found in the previous section. Figure 10 To understand the part of the introduction.

[0161] 403. Whether the operator's weight gradient is traversed to NAN or INF.

[0162] 404. When the weight gradient of the operator is NAN or INF, the abnormal state bit is modified to 1, and the scaling factor is reduced by 1 / 3 based on the scaling factor of that scale layer. .

[0163] 405. If no NaN or INF is found after traversing the weight gradients of all operators at this scale layer, then the scaling factor is increased by 2 based on the scaling factor of this scale layer. （1 / 1000） .

[0164] 406. Execute i=i+1, traverse the next scale layer, and then repeat the above process from 402 to 406 until all scale layers of the first neural network have been traversed.

[0165] 407. Update the weights of operators when no abnormal states occur in all layers during traversal.

[0166] 2. The scaling operation in step 303 is used to merge the networks as the first neural network.

[0167] like Figure 12 As shown, in this first neural network, the scaling layers in the backpropagation direction are scaling layer 1, scaling layer 2, ..., scaling layer (m-1). The scaling scale of scaling layer 1 is a, the scaling scale of scaling layer 2 is b, ..., the scaling scale of scaling layer (m-1) is f. However, only one b / a scaling operation needs to be performed at the output interface of scaling layer 1 or the input interface of scaling layer 2, and so on. Only one f / e scaling operation needs to be performed at the output interface of scaling layer (m-2) or the input interface of scaling layer (m-1). In this embodiment, the example of performing one b / a scaling operation at the input interface of scaling layer 2 is used for illustration.

[0168] For the implementation process of the scheme in this application at scale layer 1, please refer to the previous sections. Figure 10 The difference lies in the fact that, instead of scaling down the output gradient by a factor of 'a', it directly amplifies it by a factor of b / a at the input interface of scale layer 2, unlike... Figure 10 In the process, the output gradient of scale layer 1 needs to be amplified by b times in scale layer 2.

[0169] The execution process in scale layer 2 is as follows: Figure 12 As shown, scale layer 2 includes three operators: operator 4, operator 5, and operator 6. The logical relationship between the three operators is as follows: Figure 12As shown, the output of operator 4 is the input of operator 5, and the output of operator 5 is the input of operator 6. Operator 4 is the first operator of scale layer 4. When the target scale layer is scale layer 2, the output gradient of scale layer 1 is the first gradient of operator 4. Then, a scaling operation is performed on the first gradient of operator 4 according to the scaling factor b / a of scale layer 2. In this embodiment, taking the scaling operation as an amplification operation as an example, the first gradient is amplified by a factor of b / a to obtain the second gradient of operator 4. Then, the weight gradient S4 and the output gradient O4 of operator 4 are calculated using the second gradient. The output gradient O4 of operator 4 is output to operator 5. The weight gradient S5 of operator 5 is calculated using the output gradient O4, and the output gradient O5 of operator 5 is also calculated. The output gradient O5 of operator 5 is output to operator 6. The weight gradient S6 and the output gradient O6 of operator 6 are calculated using the output gradient O5 of operator 5.

[0170] If the weight gradient of each operator in scale layer 2 includes INF or NAN, then the scaling scale of the target scale layer is reduced, such as reducing it by 1 / b based on the scaling scale b. The adjusted scaling factor is b×1 / When the weight gradient of each operator is within the expression range of the first precision operation, the scaling scale of the target scale layer is increased, such as increasing it by 2 based on the scaling scale b. （1 / 1000） The adjusted scaling factor is b×2 （1 / 1000） .

[0171] When the weight gradient of each operator is within the expression range of the first precision operation, the weight gradient of each operator is inversely scaled according to the scaling scale of the target scale layer to obtain the weight gradient of each operator after the inverse scaling operation; the weight of each operator is updated according to the weight gradient of each operator after the inverse scaling operation.

[0172] Combination Figure 12 When the weight gradients S4 of operator 4, S5 of operator 5, and S6 of operator 6 are all within the expression range of the first precision operation, an inverse scaling operation is performed on the weight gradients S4, S5, and S6 according to the scaling scale b of the scaling layer 2, that is, a scaling operation by a factor of b is performed, to obtain the corresponding scaled weight gradients U4, U5, and U6. Furthermore, the weights of operator 4 can be updated according to the weight gradient U4, the weights of operator 5 can be updated according to the weight gradient U5, and the weights of operator 6 can be updated according to the weight gradient U6.

[0173] Figure 12In this process, the gradient to be output in scale layer 2 is not processed and is directly output to the next scale layer. The processing in the next scale layer can be understood by referring to the processing in scale layer 2, except that the scaling scale of the next scale layer may be different.

[0174] Third, optimize the adjustment process through inter-layer correction.

[0175] In this embodiment, interlayer correction can be performed in both the back propagation direction and the forward propagation direction, which will be described below.

[0176] 1. Interlayer corrections during backpropagation.

[0177] When the output gradient of the target scale layer is an infinite value or an invalid number, the output gradient of the target scale layer is corrected to an effective value, and the corrected output gradient of the target scale layer is transmitted to the adjacent scale layers of the target scale layer.

[0178] like Figure 13 As shown, when the target scale layer is scale layer 1, the matrix of the output gradient of scale layer 1 is... If two inf values ​​appear, the output gradient needs to be corrected. This can be done by setting the inf values ​​in the matrix to 0, and then outputting the corrected gradient, as shown below: .

[0179] In this embodiment, if the output gradient of the target scale layer is an infinite value or an invalid number, it means that the output gradient is not applicable to the weight update of the operator in each scale layer. Therefore, the weight update step is skipped directly. However, in order not to affect the subsequent calculation process, the output gradient can be corrected to an effective value within the first precision range and then transmitted to the next layer for calculation, which is beneficial to improving the training efficiency of the neural network.

[0180] 2. Interlayer corrections during forward propagation.

[0181] During forward propagation, if the feature values ​​of the target scale layer include infinitely large values ​​or invalid numbers, the update of the target scale layer is skipped.

[0182] In this embodiment, the feature value of the target scale layer is the feature value generated by each scale layer during the forward propagation process. During the forward propagation process, it is also possible to determine whether to update the operator weights in the scale layer based on the performance of the feature value within the expression range of the first precision operation. If the forward features include infinite values ​​or invalid numbers, there is no need to update the operator weights. This is beneficial to improving the training efficiency of the neural network.

[0183] In addition, based on any of the above embodiments, the following process can also be performed in the embodiments of this application: when the training of the first neural network reaches the preset conditions, the first neural network is rescaled to obtain the second neural network.

[0184] In this embodiment, the preset condition can be that the number of training cycles has reached a certain threshold, such as 300 cycles, or that the neural network has been trained to a certain extent, such as when the difference in scaling scales between different scale layers is less than a preset value. In this case, the first neural network can be re-scaled, and the scaling method can be understood by referring to the previous description. Then, a new second neural network is obtained, and the second neural network is trained. This dynamic updating of scale layers can improve the training efficiency of the neural network.

[0185] The above describes the neural network adjustment method provided by the embodiments of this application. The following describes the neural network adjustment device provided by the embodiments of this application in conjunction with the accompanying drawings.

[0186] like Figure 14 As shown, one embodiment of the neural network adjustment device 50 provided in this application includes:

[0187] like Figure 14 As shown, the adjustment device 50 for the neural network includes:

[0188] The acquisition unit 501 is used to acquire a first neural network employing mixed-precision computation. The first neural network includes multiple scale layers, wherein each scale layer has a scaling scale. The scaling scale of each scale layer refers to the scale by which the gradient associated with each scale layer in the backpropagation direction is amplified or reduced during the training of the first neural network. The mixed-precision computation includes first-precision computation. The acquisition unit 501 can perform the above-described... Figure 4 Step 201 in the corresponding method embodiment.

[0189] The first processing unit 502 is used to perform forward propagation processing on the training samples of the first neural network input to the acquisition unit 501 to obtain the value of the loss function; the first processing unit 502 can perform the above-mentioned... Figure 4 Step 202 in the corresponding method embodiment.

[0190] The second processing unit 503 is configured to scale the first gradient of the first operator in the target scale layer according to the scaling scale of the target scale layer in the backpropagation direction to obtain the second gradient of the first operator. The target scale layer is any one of multiple scale layers. The first gradient of the first operator is derived from the value of the loss function obtained by the first processing unit 502. The scaling operation is either an enlargement or reduction operation. The second gradient of the first operator is used to determine the weight gradient of each operator in the target scale layer. The second processing unit 503 can perform the above... Figure 4 Step 203 in the corresponding method embodiment.

[0191] The third processing unit 504 is used to adjust the scaling scale of the target scale layer based on the weight gradient of each operator in the target scale layer obtained by the second processing unit 503 within the expression range of the first precision operation. This third processing unit 504 can perform the above-described... Figure 4 Step 204 in the corresponding method embodiment.

[0192] In this embodiment, during neural network training, the first gradient of the first operator can be scaled according to the scaling scale of the scaling layer, thereby calculating the weight gradient of each operator in the scaling layer. Then, the performance of the weight gradient of each operator within the expression range of the first precision operation is observed, and the scaling scale of the corresponding scaling layer is adjusted. In this way, the underflow rate of the gradient of the first precision operation can be effectively reduced with a small amount of computation. This allows mixed precision training to be well applied to the training of neural networks, improving training efficiency while maintaining high training precision. Moreover, the data using low precision operation only requires a small amount of storage space on the chip, which is also beneficial for the chip to quickly read and write the data of low precision operation.

[0193] Optionally, the mixed-precision operation further includes a second-precision operation, the expression range of which is greater than that of the first-precision operation. The acquisition unit 501 is used to: receive an initial neural network to be trained; mark the first type of operator in the initial neural network as using the first-precision operation to obtain a network using mixed-precision operation, the second type of operator in the network using mixed-precision operation uses the second-precision operation; and perform scale layering on the network using mixed-precision operation to obtain a first neural network.

[0194] Optionally, the acquisition unit 501 is used to: acquire the initial scale of each network layer in the network that uses mixed precision operation; and merge network layers with the same initial scale to obtain a first neural network.

[0195] Optionally, the acquisition unit 501 is used to: acquire the initial scale of each network layer in the network using mixed precision operation; merge network layers with the same initial scale to obtain a layer-merged network; merge the first scaling operation of the output interface of the first network layer and the second scaling operation of the input interface of the second network layer in the layer-merged network to obtain a first neural network, wherein the first network layer and the second network layer are adjacent, and the first network layer is the layer preceding the second network layer in the backpropagation direction.

[0196] Optionally, the acquisition unit 501 is used to determine the initial scale of each network layer in the network using mixed precision computation according to a preset underflow rate, or to receive configuration information that configures the initial scale of each network layer in the network using mixed precision computation, and determine the initial scale of each network layer in the network using mixed precision computation according to the configuration information.

[0197] Optionally, the third processing unit 504 is configured to: reduce the scaling scale of the target scale layer when the weight gradient of each operator in the target scale layer includes infinitely large values ​​or invalid numbers; and increase the scaling scale of the target scale layer when the weight gradient of each operator in the target scale layer is within the expression range of the first precision operation.

[0198] Optionally, the third processing unit 504 is further configured to, when the weight gradient of each operator is within the expression range of the first precision operation, perform an inverse scaling operation on the weight gradient of each operator in the target scale layer according to the scaling scale of the target scale layer, so as to obtain the weight gradient of each operator after the inverse scaling operation in the target scale layer; and update the weight of each operator in the target scale layer according to the weight gradient of each operator after the inverse scaling operation in the target scale layer.

[0199] Optionally, the third processing unit 504 is further configured to determine the output gradient of the target scale layer based on the second gradient of the first operator; and to perform an inverse scaling operation on the output gradient of the target scale layer according to the scaling scale of the target scale layer, so as to obtain the output gradient of the target scale layer.

[0200] Optionally, the third processing unit 504 is further configured to, when the output gradient of the target scale layer is an infinitely large value or an invalid number, correct the output gradient of the target scale layer to a valid value within the expression range of the first precision operation; and transmit the corrected output gradient of the target scale layer to the adjacent scale layer of the target scale layer.

[0201] Optionally, the third processing unit 504 is also configured to skip updating the target scale layer if the feature values ​​of the target scale layer include infinitely large values ​​or invalid numbers during the forward propagation process.

[0202] Optionally, the third processing unit 504 is further configured to rescale the first neural network to obtain a second neural network when the training of the first neural network reaches a preset condition.

[0203] It should be noted that the acquisition unit 501, the first processing unit 502, the second processing unit 503, and the third processing unit 504 can be implemented by one unit or module, or by multiple units or modules. This application embodiment does not impose any limitations on this, as long as the above method flow can be executed.

[0204] The neural network adjustment device provided in the embodiments of this application can be understood by referring to the corresponding content in the preceding section on neural network adjustment methods, and will not be repeated here.

[0205] Figure 15 The diagram shown illustrates a possible logical structure of a computer device 60 provided in an embodiment of this application. The computer device 60 may be a neural network adjustment device. The computer device 60 includes a processor 601, a communication interface 602, a memory 603, and a bus 604. The processor 601, communication interface 602, and memory 603 are interconnected via the bus 604. In an embodiment of this application, the processor 601 is used to control and manage the operation of the computer device 60; for example, the processor 601 is used to execute... Figures 2 to 13 In the method embodiment, the neural network adjustment process is described. Communication interface 602 is used to support communication between computer device 60 and the computer device 60. Memory 603 is used to store program code and data from the computer device 60.

[0206] The processor 601 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 601 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 604 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0207] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figures 3 to 8 The method for training the model, or performing the above... Figure 2-13 Methods for adjusting neural networks.

[0208] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 2-13 Adjustment methods for neural networks.

[0209] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 2-13 The adjustment method for the neural network. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for inter-process communication. This chip system can be composed of chips or may include chips and other discrete devices.

[0210] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0211] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0212] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0215] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for adjusting a neural network, characterized in that, include: A first neural network employing mixed-precision computation is obtained. The first neural network includes multiple scale layers, wherein each scale layer has a scaling scale, and the scaling scale of each scale layer refers to the scale by which the gradient associated with each scale layer in the backpropagation direction is amplified or reduced when training the first neural network. The mixed-precision computation includes first-precision computation. The training samples input into the first neural network are processed by forward propagation to obtain the value of the loss function; In the backpropagation direction, the first gradient of the first operator in the target scale layer is scaled according to the scaling scale of the target scale layer to obtain the second gradient of the first operator. The target scale layer is any one of the multiple scale layers. The first gradient of the first operator comes from the value of the loss function. The scaling operation is a magnification operation or a reduction operation. The second gradient of the first operator is used to determine the weight gradient of each operator in the target scale layer. The scaling scale of the target scale layer is adjusted based on the performance of the weight gradient of each operator in the target scale layer within the expression range of the first precision operation. The mixed precision operation further includes a second precision operation, wherein the expression range of the second precision operation is greater than that of the first precision operation; The process of obtaining the first neural network using mixed-precision computation includes: Receive an initial neural network to be trained, wherein the input data of the initial neural network includes at least one of image data, video data, audio data, or text data; The first type of operator in the initial neural network is marked as using the first precision operation to obtain a network using mixed precision operation, wherein the second type of operator in the network using mixed precision operation uses the second precision operation; The network employing mixed-precision computation is scaled to obtain the first neural network.

2. The adjustment method according to claim 1, characterized in that, The step of scaling the network employing mixed-precision computation to obtain the first neural network includes: Obtain the initial scale for each network layer in a network that uses mixed-precision computation; The network layers with the same initial scale are merged to obtain the first neural network.

3. The adjustment method according to claim 1, characterized in that, The step of scaling the network employing mixed-precision computation to obtain the first neural network includes: Obtain the initial scale for each network layer in a network that uses mixed-precision computation; By merging network layers with the same initial scale, a layer-merged network can be obtained. The first scaling operation of the output interface of the first network layer and the second scaling operation of the input interface of the second network layer in the layer merging network are merged to obtain the first neural network. The first network layer and the second network layer are adjacent, and the first network layer is the layer preceding the second network layer in the back propagation direction.

4. The adjustment method according to claim 2 or 3, characterized in that, The process of obtaining the initial scale of each network layer in a network employing mixed-precision computation includes: The initial scale of each network layer in the network employing mixed-precision computation is determined based on a preset underflow rate; or, Receive configuration information for configuring the initial scale of each network layer in the network employing mixed-precision computation, and determine the initial scale of each network layer in the network employing mixed-precision computation based on the configuration information.

5. The adjustment method according to any one of claims 1-3, characterized in that, The first type of operator includes convolution operators and / or fully connected operators.

6. The adjustment method according to any one of claims 1-3, characterized in that, The step of adjusting the scaling scale of the target scale layer based on the performance of the weight gradient of each operator in the target scale layer within the expression range of the first precision operation includes: If the weight gradient of each operator in the target scale layer contains an infinite value or an invalid number, then the scaling scale of the target scale layer is reduced. If the weight gradient of each operator in the target scale layer is within the expression range of the first precision operation, then the scaling scale of the target scale layer is increased.

7. The adjustment method according to any one of claims 1-3, characterized in that, The method further includes: According to the scaling scale of the target scale layer, perform the inverse scaling operation on the weight gradient of each operator in the target scale layer to obtain the weight gradient of each operator after the inverse scaling operation in the target scale layer. The weights of each operator in the target scale layer are updated based on the weight gradient of each operator after the inverse scaling operation in the target scale layer.

8. The adjustment method according to claim 2, characterized in that, The method further includes: The output gradient of the target scale layer is determined based on the second gradient of the first operator. According to the scaling scale of the target scale layer, the inverse scaling operation of the scaling operation is performed on the output gradient of the target scale layer to obtain the output gradient of the target scale layer.

9. The adjustment method according to claim 8, characterized in that, The method further includes: When the output gradient of the target scale layer is an infinite value or an invalid number, the output gradient of the target scale layer is corrected to a valid value within the expression range of the first precision operation. The corrected output gradient of the target scale layer is transmitted to the adjacent scale layers of the target scale layer.

10. The adjustment method according to any one of claims 1-3, characterized in that, The method further includes: During the forward propagation process, if the feature values ​​of the target scale layer include infinitely large values ​​or invalid numbers, the update of the target scale layer is skipped.

11. The adjustment method according to any one of claims 1-3, characterized in that, The method further includes: When the training of the first neural network reaches the preset conditions, the first neural network is rescaled to obtain the second neural network.

12. A neural network adjustment device, characterized in that, include: An acquisition unit is used to acquire a first neural network employing mixed-precision computation. The first neural network includes multiple scale layers, wherein each scale layer has a scaling scale. The scaling scale of each scale layer refers to the scale used to amplify or reduce the gradient associated with each scale layer in the backpropagation direction when training the first neural network. The mixed-precision computation includes first-precision computation. The first processing unit is used to perform forward propagation processing on the training samples of the first neural network obtained by the acquisition unit to obtain the value of the loss function; The second processing unit is configured to scale the first gradient of the first operator in the target scale layer according to the scaling scale of the target scale layer in the backpropagation direction to obtain the second gradient of the first operator. The target scale layer is any one of the plurality of scale layers. The first gradient of the first operator comes from the value of the loss function. The scaling operation is a magnification operation or a reduction operation. The second gradient of the first operator is used to determine the weight gradient of each operator in the target scale layer. The third processing unit is used to adjust the scaling scale of the target scale layer based on the performance of the weight gradient of each operator in the target scale layer within the expression range of the first precision operation obtained by the second processing unit. The mixed precision operation further includes a second precision operation, wherein the expression range of the second precision operation is greater than that of the first precision operation; The acquisition unit is used for: Receive an initial neural network to be trained, wherein the input data of the initial neural network includes at least one of image data, video data, audio data, or text data; The first type of operator in the initial neural network is marked as using the first precision operation to obtain a network using mixed precision operation, wherein the second type of operator in the network using mixed precision operation uses the second precision operation; The network employing mixed-precision computation is scaled to obtain the first neural network.

13. The adjusting device according to claim 12, characterized in that, The acquisition unit is used for: Obtain the initial scale for each network layer in a network that uses mixed-precision computation; The network layers with the same initial scale are merged to obtain the first neural network.

14. The adjusting device according to claim 12, characterized in that, The acquisition unit is used for: Obtain the initial scale for each network layer in a network that uses mixed-precision computation; By merging network layers with the same initial scale, a layer-merged network can be obtained. The first scaling operation of the output interface of the first network layer and the second scaling operation of the input interface of the second network layer in the layer merging network are merged to obtain the first neural network. The first network layer and the second network layer are adjacent, and the first network layer is the layer preceding the second network layer in the back propagation direction.

15. The adjusting device according to claim 13 or 14, characterized in that, The acquisition unit is used to determine the initial scale of each network layer in the network employing mixed-precision computation based on a preset underflow rate; or, Receive configuration information for configuring the initial scale of each network layer in the network employing mixed-precision computation, and determine the initial scale of each network layer in the network employing mixed-precision computation based on the configuration information.

16. The adjusting device according to any one of claims 12-14, characterized in that, The third processing unit is used for: If the weight gradient of each operator in the target scale layer contains an infinite value or an invalid number, then the scaling scale of the target scale layer is reduced. If the weight gradient of each operator in the target scale layer is within the expression range of the first precision operation, then the scaling scale of the target scale layer is increased.

17. The adjusting device according to any one of claims 12-14, characterized in that, The third processing unit is further configured to perform an inverse scaling operation on the weight gradient of each operator in the target scale layer according to the scaling scale of the target scale layer, so as to obtain the weight gradient of each operator in the target scale layer after the inverse scaling operation; and update the weight of each operator in the target scale layer according to the weight gradient of each operator in the target scale layer after the inverse scaling operation.

18. The adjusting device according to claim 13, characterized in that, The third processing unit is further configured to determine the output gradient of the target scale layer based on the second gradient of the first operator; and to perform an inverse scaling operation on the output gradient of the target scale layer according to the scaling scale of the target scale layer, so as to obtain the output gradient of the target scale layer.

19. The adjusting device according to claim 18, characterized in that, The third processing unit is further configured to correct the output gradient of the target scale layer to a valid value within the expression range of the first precision operation when the output gradient of the target scale layer is an infinitely large value or an invalid number. The corrected output gradient of the target scale layer is transmitted to the adjacent scale layers of the target scale layer.

20. The adjusting device according to any one of claims 12-14, characterized in that, The third processing unit is further configured to skip updating the target scale layer if, during the forward propagation process, the feature value of the target scale layer includes an infinitely large value or an invalid number.

21. The adjusting device according to any one of claims 12-14, characterized in that, The third processing unit is further configured to re-scale the first neural network to obtain a second neural network when the training of the first neural network reaches a preset condition.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1-11.

23. A computing device, characterized in that, It includes one or more processors and a storage medium storing computer programs; When the computer program is executed by the one or more processors, it implements the method as described in any one of claims 1-11.

24. A chip system, characterized in that, It includes one or more processors, which are invoked to perform the method as described in any one of claims 1-11.

25. A computer program product, characterized in that, Includes a computer program, which, when executed by one or more processors, is used to implement the method as described in any one of claims 1-11.

Images