Neural network training method and device, electronic equipment and storage medium

Abstract

The invention discloses a neural network training method, a neural network training device, electronic equipment and a computer readable storage medium, and the neural network training method is at least applied to a visual processing scene, and comprises the steps: inputting a training sample into a to-be-trained neural network, and obtaining an output result of the neural network under each current connection weight; the training sample at least comprises visual data; determining a first gradient of each connection weight based on the output result and the labeling information of the training sample; directed gradient noise is generated for each connection weight; for the target connection weight meeting the relation requirement of the first gradient and a set gradient threshold value, the directional gradient noise corresponding to the target connection weight and the first gradient are fused to obtain a fusion gradient, and updating of the target connection weight is achieved according to the fusion gradient; and the training is repeated until the obtained neural network meets the set requirement. The neural network training efficiency can be improved, the training time is shortened, and the training cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of neural network training, and in particular to a neural network training method, a neural network training device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence and the emergence of large models, the scale of neural networks has increased exponentially, and the learning data has grown rapidly. The huge amount of computing has led to a rapid increase in the training cost of neural networks. The current mainstream large model training costs are often as high as tens of millions or even hundreds of millions of dollars, resulting in huge economic costs. How to develop effective neural network training methods so that they can quickly and accurately implement functions, improve training efficiency, and reduce training costs is an urgent problem to be solved in the current artificial intelligence field.

[0003] The training method of neural networks is the key to determining the efficiency and performance of neural networks. However, unlike the rapidly developing scale of neural networks, the training methods of neural networks are developing relatively slowly. The current mainstream neural network training methods are still the traditional back propagation algorithm and gradient descent algorithm. The current neural network training method aims to reduce the loss function between the output of the neural network and the annotation of the training sample. Through the back propagation algorithm, the gradient of the loss function is calculated, and the gradient descent algorithm is used to iteratively search the parameters of the neural network in the opposite direction of the corresponding gradient, so that the neural network evolves to the local minimum of the loss function, realizing complex functions such as network prediction, classification, and generation.

[0004] However, the existing gradient descent algorithm can only search and update in a smaller effective parameter space, which makes the parameter update efficiency of the neural network low and is more prone to problems such as saddle points in the loss function, resulting in gradient vanishing or gradient explosion, making the neural network training inefficient, time-consuming and costly. Summary of the invention

[0005] In view of the above technical problems, the present application provides a neural network training method, a neural network training device, an electronic device and a computer-readable storage medium. The technical solutions are as follows: According to a first aspect of the present application, a neural network training method is provided, which is at least applied to a visual processing scenario, and the method comprises: Inputting a training sample into a neural network to be trained to obtain an output result of the neural network under each current connection weight; the training sample at least includes visual data; Determining a first gradient of each connection weight based on the output result and the labeling information of the training sample; Generating directional gradient noise for each of the connection weights; wherein the initial direction of the directional gradient noise is randomly generated and remains fixed during the training process; For a target connection weight that satisfies the relationship between the first gradient and the set gradient threshold, the directional gradient noise corresponding to the target connection weight and the first gradient are fused to obtain a fused gradient, and the target connection weight is updated according to the fused gradient; Repeat the training until the obtained neural network meets the set requirements.

[0006] According to a second aspect of the present application, a neural network training device is provided, which is at least applied to a visual processing scenario, and includes: An input unit, used to input a training sample into a neural network to be trained, and obtain an output result of the neural network under each current connection weight; the training sample at least includes visual data; A determination unit, configured to determine a first gradient of each of the connection weights based on the output result and the labeling information of the training sample; A generating unit, configured to generate directional gradient noise for each of the connection weights; wherein the initial direction of the directional gradient noise is randomly generated and remains fixed during the training process; An updating unit, configured to fuse the directional gradient noise corresponding to the target connection weight and the first gradient for a target connection weight that satisfies the relationship requirement between the first gradient and the set gradient threshold, obtain a fused gradient, and update the target connection weight according to the fused gradient; The training unit is used to repeat the training until the obtained neural network meets the set requirements.

[0007] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method as described in the first aspect.

[0008] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method described in the first aspect are implemented.

[0009] The technical solution provided by the present application takes the visual processing scenario as an example, takes the training sample and the annotation information of the training sample as the training sample set, and the training sample may include visual data; inputs the training sample into the neural network to be trained, and obtains the output result of the neural network under each current connection weight; based on the output result and the annotation information of the training sample input into the neural network, the first gradient of each connection weight of the neural network is determined according to the back propagation algorithm. Directed gradient noise is generated for each connection weight of the neural network; wherein the initial direction of any directed gradient noise is randomly generated, and remains fixed in any subsequent round of training and does not change; for the target connection weight that meets the relationship requirements between the first gradient and the set gradient threshold, the directed gradient noise corresponding to the target connection weight and the first gradient are fused to obtain a fused gradient, and the fused gradient is used as the new gradient of the target connection weight; and according to the fused gradient, the gradient descent algorithm is used to update the target connection weight, and the training sample and the annotation information of the training sample are used to repeat the training until the neural network meets the set requirements.

[0010] It can be seen that when the neural network is trained in a gradient descent manner, a directional gradient noise with fixed positive and negative signs (i.e., positive and negative directions) is generated for each connection weight of the neural network during the entire training process, and the directional gradient noise corresponding to the target connection weight that meets the relationship requirements between the first gradient and the set gradient threshold and the first gradient are fused to obtain a fused gradient, and the fused gradient is used as a new gradient to update the target connection weight. The directionality of the gradient noise, that is, the initial direction of the gradient noise is generated only once and reused in subsequent training, so that the training process can generate a more effective search for the parameter space, so as to perform continuous search in a larger effective parameter space, and at the same time, it can avoid the neural network from falling into a local minimum, thereby improving the training efficiency of the neural network, reducing the training time, and compressing the training cost.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 It is a schematic diagram of a neural network training scenario in related technology; Figure 2It is a flowchart of a neural network training method according to an embodiment of the present application; Figure 3 This is a schematic diagram of a neural network training scenario according to an embodiment of the present application; Figure 4 It is a structural schematic diagram of a neural network training device according to an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of this application.

[0015] With the development of artificial intelligence and the emergence of large models, the scale of neural networks has increased exponentially, and the learning data has grown rapidly. The huge amount of computing has led to a rapid increase in the training cost of neural networks. The current mainstream large model training costs are often as high as tens of millions or even hundreds of millions of dollars, resulting in huge economic costs. How to develop effective neural network training methods so that they can quickly and accurately implement functions, improve training efficiency, and reduce training costs is an urgent problem to be solved in the current artificial intelligence field.

[0016] In related technologies, the training method of neural networks is the key to determining the efficiency and performance of neural networks. However, unlike the rapidly developing scale of neural networks, the training methods of neural networks in related technologies have developed relatively slowly. The current mainstream neural network training methods are still the traditional back propagation algorithm and gradient descent algorithm, such as Figure 1 In the neural network training scenario shown, the current neural network training process can be roughly summarized as: initialize network parameters → input training data → forward propagation → calculate loss function → back propagation → (using gradient descent method) update network parameters → repeat training → training completed. It can be seen that the current neural network training method aims to reduce the loss function between the output of the neural network and the annotation of the training sample. Through the back propagation algorithm, the gradient of the loss function is calculated, and the gradient descent algorithm is used to iteratively search the parameters of the neural network in the opposite direction of the corresponding gradient, so that the neural network evolves to the local minimum of the loss function, realizing complex functions such as network prediction, classification, and generation.

[0017] However, the current gradient descent algorithm can only search and update in a smaller effective parameter space, which makes the parameter update efficiency of the neural network low and is more prone to problems such as saddle points in the loss function, resulting in gradient vanishing or gradient explosion, making the neural network training inefficient, time-consuming and costly.

[0018] It is worth noting that the above introduction to the neural network training scenarios in the related technologies is only an exemplary display. In practical applications, other neural network training scenarios are not excluded and no specific limitation is made to this.

[0019] In view of the above problems, the present application provides a neural network training method, which is at least applied to visual processing scenarios and can improve the efficiency of neural network training, reduce training time, and compress training costs. Figure 2 As shown, the method may include the following steps: S201, inputting training samples into the neural network to be trained, and obtaining output results of the neural network under each current connection weight.

[0020] The training samples at least include visual data.

[0021] S202: Determine a first gradient of each connection weight based on the output result and the labeling information of the training sample.

[0022] S203 , generating directional gradient noise for each connection weight.

[0023] The initial direction of the directional gradient noise is randomly generated and remains fixed during the training process.

[0024] S204. For a target connection weight that satisfies the relationship between the first gradient and the set gradient threshold, the directional gradient noise corresponding to the target connection weight and the first gradient are fused to obtain a fused gradient, and the target connection weight is updated according to the fused gradient.

[0025] S205, repeat the training until the obtained neural network meets the set requirements.

[0026] The technical solution provided by the embodiment of the present application, when training a neural network in a gradient descent manner, generates directional gradient noise with fixed positive and negative signs (i.e., positive and negative directions) throughout the training process for each connection weight of the neural network, and fuses the directional gradient noise corresponding to the target connection weight that meets the relationship requirements between the first gradient and the set gradient threshold and the first gradient to obtain a fused gradient, and uses the fused gradient as a new gradient to update the target connection weight. The directionality of the gradient noise, i.e., the initial direction of the gradient noise is generated only once and reused in subsequent training, so that the training process can generate a more effective search for the parameter space, so as to perform continuous search in a larger effective parameter space, and at the same time, it can prevent the neural network from falling into a local minimum, thereby improving the training efficiency of the neural network, reducing the training time, and compressing the training cost.

[0027] It is understandable that the order of S202 and S203 is not limited.

[0028] The execution subject of the training method of the above-mentioned neural network provided in the embodiment of the present application can have multiple specific implementations. As an example, the execution subject can be a server, or an electronic device such as a desktop computer, a laptop computer, a mobile terminal, etc. For the convenience of description, in any of the embodiments below, the training method of the neural network provided in the embodiment of the present application is described by taking the server as the execution subject as an example. Among them, the server can be a single device, or include multiple devices, for example, a distributed server, and the present application does not specifically limit this.

[0029] The training samples input into the neural network can be implemented in many specific ways. As an example, in the case of visual processing, the training samples can include visual data. As another example, the visual data can be image data or other data. In other scenarios, the training samples can also be other types of data, such as text data in a text processing scenario. Therefore, the specific implementation of the training samples is not limited.

[0030] There may be multiple specific implementations of the above-mentioned set requirements that the neural network needs to meet. As an example, the neural network meeting the set requirements may mean that the neural network can complete the specified task. As another example, the neural network meeting the set requirements may mean that the number of repeated training of the neural network reaches a threshold number of times. Therefore, there is no limitation on the specific implementation of the above-mentioned set requirements that the neural network needs to meet.

[0031] As an example, when the above-mentioned setting requirement refers to the ability of the neural network to complete the specified task, the server can obtain the training samples related to the requirements of the specified task that the neural network needs to complete and the annotation information of the training samples. Taking the type of the specified task as any machine learning task that applies a supervised learning or self-supervised learning training process as an example, the specified task can be any task such as visual scene processing, including classification tasks, recognition tasks, generation tasks, etc. Therefore, there is no specific limitation on the type of the specified task.

[0032] For ease of description, the following is an exemplary description of the above-mentioned designated task as an image classification task. As an example, when the above-mentioned designated task is an image classification task, the training sample input to the neural network to be trained can be an image to be classified. As another example, each image to be classified can be preprocessed, and the preprocessing method can include cropping, scaling and / or rotation. The image to be classified is converted into a one-dimensional vector form after preprocessing, and then input into the neural network to be trained. As another example, the annotation information of the training sample can be the category information of the image to be classified, wherein each image to be classified can correspond to a category. Under the training data set consisting of the image to be classified and the annotation information of the image to be classified, the server can realize the recognition and classification of the image after completing the training of the neural network. Taking a specific training scenario as an example, the server can classify the surrounding environment image of the unmanned equipment to determine whether there is an obstacle in the environment image. The corresponding training sample can be an environment image collected by the image acquisition device in the past. The annotation information of the training sample can be information on whether the environment image contains obstacles, that is, the annotation information is the two image categories of obstacles and obstacles.

[0033] As an example, before inputting training samples into the neural network to be trained, the neural network can be initialized first. Specifically, the connection weights of the neural network can be initially set. That is, before the training samples are input into the neural network, the initially set connection weights are the current connection weights of the neural network.

[0034] As an example, the method of initializing the neural network can be determined based on the architecture of the neural network, wherein the architecture of the neural network can be any network architecture that applies a supervised learning or self-supervised learning training process, for example, the network architecture can be a deep neural network, a convolutional neural network, or a transformer architecture, or other types of architectures. Therefore, there is no specific limitation on the type of neural network architecture.

[0035] For ease of description, the following uses the architecture of a neural network as a multi-layer feedforward neural network as an example to exemplify a method of initializing a neural network. As an example, the server can build a multi-layer feedforward neural network containing L layers of neurons. As another example, the first layer of the multi-layer feedforward neural network containing L layers of neurons can be the input layer, the Lth layer is the output layer, and the other layers can be hidden layers. The number of neurons in the input layer can be the input data dimension. , the number of neurons in the output layer can be the number of image categories , the number of neurons in the hidden layer, i.e. the width of the neural network, can be N, and the neurons between adjacent layers are fully connected. Specifically, the connection weight from layer l to layer l+1 is defined as ,in The numbers of neurons in the lth layer and the l+1th layer are respectively. It is worth noting that the server can set any connection weight initialization that meets the requirements of the network architecture, and the embodiment of the present application does not limit the initialization method of the neural network.

[0036] As an example, network trainability can be used as a reference standard to specifically introduce an initialization method based on a multi-layer feedforward neural network with adjustable trainability (adjustable trainability means that the trainability of the neural network can be adjusted through initialization settings). Of course, this method does not constitute an improper limitation on the embodiments of the present application. As another example, the server can construct a connection weight perturbation term to initialize the connection weights of the neural network to the sum of the unit matrix and the connection weight perturbation term to adjust the trainability of the neural network initialization. As another example, the server can select two forms, Bernoulli perturbation term and Gaussian perturbation term, as the connection weight perturbation term. The connection weights under the two connection weight perturbation terms can be expressed as: (1) (2) in, is the Bernoulli perturbation term, and its diagonal element , the off-diagonal elements follow a Bernoulli distribution with probability p , that is, the probability that the connection weight item takes 1 is p, and the probability that it takes 0 is 1-p, is a Gaussian perturbation term with a mean of 0 and a standard deviation of Gaussian distribution, , N is the width of the neural network. The server can adjust the perturbation intensity parameter, that is, the Bernoulli distribution probability p and the Gaussian distribution standard deviation parameter , adjusts the trainability of the neural network initialization. The larger the perturbation intensity parameter, the weaker the trainability of the neural network initialization.

[0037] As an example, the output result of the neural network under each current connection weight can be compared with the annotation information of the training sample input into the neural network, and the first gradient of each connection weight can be determined according to the back propagation algorithm. It can be understood that the first gradient of the connection weight can refer to the gradient of the connection weight without fusion of directional gradient noise in this round of training, and if the first gradient is fused with directional gradient noise, the fusion of the two gradients is the new gradient of the connection weight.

[0038] There are many specific ways to determine the above-mentioned first gradient. As an example, the server can convert the training samples with the above-mentioned annotation information input into the neural network into a target vector, and convert the output results of the neural network under each current connection weight into a vector form, and determine the loss function of the neural network based on the target vector and the output results converted into a vector form; based on the loss function, the above-mentioned first gradient is determined according to the back propagation algorithm. As another example, taking the above-mentioned designated task as an image classification task, the format of the target vector is a unique hot vector annotated to point to the image category, that is, the category item corresponding to the training image is 1, and the other category items are 0. As another example, the server can determine the loss function by comparing the target vector with the above-mentioned output results to describe the difference between the target vector and the output results. The server can flexibly set the form of the loss function, for example, a loss function containing high-order terms or other regularization terms, and there is no specific limitation on this.

[0039] As another example, the loss function can be expressed as follows: (3) in, Represents the parameters of the neural network. Specifically, It can be the connection weights and biases of the neural network, For parameters Under the loss function, the server can compare the parameters Next, the neural network takes input The output of With the target vector The loss function is calculated by the difference between

[0040] In order to complete the training task of the neural network, the server needs to minimize the loss function as much as possible. The general gradient descent algorithm can make the neural network evolve to the local minimum of the loss function by iteratively searching the parameters of the neural network in the opposite direction of the gradient of the corresponding loss function, so as to obtain better task results. As an example, the embodiment of the present application can provide optimization of the training method based on the general gradient descent algorithm, so it is also necessary to calculate the gradient of the loss function in the parameter space. As another example, since the loss function is a function of all parameters of the neural network, the server can determine the first gradient corresponding to each connection weight of the neural network and the absolute value of the first gradient according to the back propagation algorithm.

[0041] The following is an illustrative introduction using a multi-layer feedforward neural network as an example: (4) Among them, the weight term of the connection from the lth layer of neurons to the l+1th layer of neurons in the neural network is The corresponding gradient is the loss function The partial derivative of it. Among them, are the parameters of the neural network, which can be regarded as all A collection of .

[0042] As another example, in order to measure the training status of the neural network at multiple scales, when the neural network includes multiple layers, the server can also calculate the mean of the absolute values ​​of the first gradients of the connection weights in each layer of the neural network. , and the mean of the absolute values ​​of the first gradients of the connection weights in the neural network (i.e., all the connection weights of the neural network) : (5) (6) in, Represents the mean of the absolute values ​​of the first gradients corresponding to all connection weights from layer l to layer l+1 of the neural network, Represents the average of the absolute values ​​of the first gradients of each connection weight in the neural network (i.e., all connection weights of the neural network). The above three absolute values ​​related to the first gradient (4), (5), (6): The absolute value of the first gradient corresponding to each connection weight of the neural network , the mean of the absolute values ​​of the first gradients of the connection weights in each layer of the neural network , the mean of the absolute values ​​of the first gradients of all connection weights in the neural network (i.e., all connection weights of the entire neural network) , which can correspond to the three different gradient judgment and fusion methods provided in the embodiments of the present application.

[0043] In order to increase the search range of parameter space and avoid invalid search of traditional gradient descent algorithm in multi-parameter dimensions, as an example, the server can generate directional gradient noise with fixed positive and negative signs to correct the first gradient calculated by the back propagation algorithm in S202. The server can randomly generate directional gradient noise for each connection weight according to the specified probability. In the subsequent training, the sign (i.e., the initial direction) of the generated directional gradient noise is fixed, while the intensity can be flexibly adjusted. The initial direction of any directional gradient noise is randomly generated, that is, its positive or negative sign is randomly generated, and once the positive and negative signs are generated, they are fixed and used in the subsequent training process and will not change.

[0044] The following is an exemplary introduction to a specific process of generating directional gradient noise: As an example, to generate the initial direction of the directional gradient noise, the server will randomly generate the direction of the directional gradient noise for each connection weight with a specified probability.

[0045] The above-mentioned specified probability can be implemented in many specific ways. As an example, the specified probability can be implemented as follows: (7) (8) in, Represents the connection weight term The corresponding gradient noise, sgn( ) indicates its positive and negative signs (1, -1), that is, 50% probability is positive, 50% probability is negative. Of course, the specific value of the specified probability is not limited to this.

[0046] As another example, when generating the intensity of the directional gradient noise, the server may generate the intensity of the directional gradient noise assigned a positive value in a specified manner.

[0047] The above-mentioned designation method can be implemented in many specific ways. As an example, the designation method can be a random generation method. As another example, the random generation method can refer to random generation according to Gaussian distribution, and the mean of Gaussian distribution is a positive value, and its standard deviation is much smaller than its mean; it can also be randomly generated according to other types of distribution. As another example, the designation method can also be a method based on preset value generation. Therefore, the specific implementation of the above-mentioned designation method is not limited.

[0048] There are many specific implementations of the frequency of generating the intensity of the directional gradient noise. As an example, the intensity of the directional gradient noise can also be kept fixed during the training process after generation, just like the initial direction of the directional gradient noise, and no longer change. As another example, during multiple rounds of training, the intensity of the directional gradient noise can also be generated multiple times, for example, the intensity of the directional gradient noise is regenerated once in each round of training. Therefore, the specific implementation of the frequency of generating the intensity of the directional gradient noise is not limited.

[0049] In addition, the intensity of the generated directional gradient noise can be flexibly adjusted in a variety of ways during training. As an example, the intensity of the directional noise can be adjusted according to the training stage. As another example, the intensity of the directional noise can also be adjusted according to the size of the loss function value. Therefore, the specific implementation of the implementation method of adjusting the intensity of the generated directional gradient noise is not limited.

[0050] The following is an example of random generation according to Gaussian distribution using the above-mentioned specified method: Taking Gaussian distribution with a positive mean and a standard deviation much smaller than the mean as an example, the server can randomly generate the intensity of the directional gradient noise in the following way: (9) in, represents the intensity of directional gradient noise, , are the mean and standard deviation of the Gaussian distribution, specifically, , Need to meet and , to ensure that the intensity of the directional gradient noise generated by the server is positive. , the probability of the server generating a negative value is extremely low and can be ignored. Of course, the generation method of the intensity of the directional gradient noise is not limited to this.

[0051] As another example, the oriented gradient noise can be obtained by multiplying the initial direction of the oriented gradient noise by the intensity of the oriented gradient noise: (10) in, is the connection weight The corresponding directional gradient noise The initial direction of Represents the connection weight term The corresponding directional gradient noise strength.

[0052] It is worth noting that the above intensity of directional gradient noise The introduction of the generation method is only an example. In practical applications, other generation methods are not excluded. The server can generate the strength of the directional gradient noise with a positive value in any other way, or flexibly select the frequency of generating the strength of the directional gradient noise, or flexibly adjust the strength of the directional gradient noise according to the training process, such as the degree of training or the size of the loss function. Therefore, there is no specific limitation on the generation and update method of the strength of the directional gradient noise. However, the initial direction of the directional gradient noise It can only be generated once, and the same direction needs to be used repeatedly in subsequent training to ensure the effectiveness of parameter space search.

[0053] As an example, the server can fuse the first gradient calculated by back propagation in S202 and the directional gradient noise generated in S203. To prevent the gradient noise from affecting the effective gradient information transmission and hindering the gradient descent process, the server can set a gradient threshold. When the relevant value of the first gradient calculated by back propagation is less than or equal to the set gradient threshold, that is, when the calculated gradient information is weak, the directional gradient noise is fused to correct the gradient calculated by back propagation and specifically expand the parameter space search range.

[0054] As an example, the server may set three gradient thresholds, such as the weight gradient threshold , layer gradient threshold , network gradient threshold That is, the gradient threshold may include at least one of a weight gradient threshold, a layer gradient threshold, and a network gradient threshold. The embodiment of the present application does not limit the specific size of the gradient threshold. The server may set a reasonable gradient threshold according to the specific training scenario.

[0055] The above three set gradient thresholds can be used to determine whether the three types of values ​​related to the first gradient meet the requirements. Taking the multi-layer feedforward neural network as an example, when the above set gradient threshold includes the weight gradient threshold, the above relationship requires to include the absolute value of the first gradient. With weight gradient threshold The magnitude relationship is the first relationship requirement; when the neural network includes multiple layers, setting the gradient threshold may include the above-mentioned layer gradient threshold; the above-mentioned relationship requirement may include the mean of the absolute values ​​of the first gradients of the connection weights in the layer With layer gradient threshold The size relationship of is the second relationship requirement. At this time, the target connection weight includes all connection weight items in the target layer that meet the second relationship requirement. When the gradient threshold is set to include the network gradient threshold, the above relationship requirement can include the mean of the absolute values ​​of the first gradients of all connection weight items in the neural network. With the network gradient threshold The third relationship requirement is met, and the target connection weight includes all connection weight items of the neural network that meet the third relationship requirement.

[0056] There are multiple ways to fuse the directional gradient noise corresponding to the target connection weight and the first gradient. As an example, the server can sum the directional gradient noise corresponding to the target connection weight and the first gradient to obtain the target connection weight fusion gradient. According to the relationship between the value related to the first gradient and the set gradient threshold, the above three methods for determining the target connection weight can be provided (the first relationship requirement, the second relationship requirement and the third relationship requirement corresponding to the weight gradient threshold, the layer gradient threshold and the network gradient threshold mentioned above, respectively), corresponding to three optional noise fusion methods: As a first example, the absolute value of the first gradient determines gradient fusion. The server determines whether the absolute value of the first gradient of any connection weight is less than or equal to the weight gradient threshold, that is, whether it meets the first relationship requirement. If it does, the connection weight is the target connection weight, and the sum of the first gradient of the connection weight and the directional noise gradient (that is, the fused gradient) is used as the new gradient of the connection weight. The gradients corresponding to the remaining connection weight items remain unchanged (still the first gradient). For example, the expression of gradient fusion can be as follows: (11) As a second example, the mean of the absolute values ​​of the first gradients of each connection weight in the layer determines the gradient fusion. The server determines whether the mean of the absolute values ​​of the first gradients of each connection weight in the layer is less than or equal to the layer gradient threshold, that is, whether the second relationship requirement is met. If so, the layer is the target layer, and the sum of the first gradient of each connection weight in the target layer and the directional noise gradient (that is, the fused gradient) is used as the new gradient of each connection weight. The gradients corresponding to the connection weights in the remaining layers outside the target layer remain unchanged (still the first gradient). For example, the expression of gradient fusion can be as follows: (12) As a third example, the mean of the absolute values ​​of the first gradients of all connection weight items in the neural network determines the gradient fusion. The server determines whether the mean of the absolute values ​​of the first gradients of all connection weight items in the neural network is less than or equal to the network gradient threshold, that is, whether the third relationship requirement is met. If so, the sum of the first gradients of all connection weight items in the neural network and the directional noise gradient (i.e., the fused gradient) is used as the new gradient of each connection weight. For example, the expression of gradient fusion can be as follows: (13) in, and The connection weights are The corresponding first gradient and directional gradient noise, For Iverson brackets, if condition x is met, = 1, if the condition x is not satisfied, = 0.

[0057] The embodiment of the present application does not limit the above three gradient fusion methods. The server can flexibly select any one of the three fusion methods to perform gradient fusion, or flexibly combine the three gradient fusion methods in different training stages.

[0058] Considering that the gradient vanishing phenomenon often exists in the neural network to be trained, the gradient cannot be back-propagated from the output side to the input side. Specifically, the gradient size and information strength on the output side are often much greater than those on the input side. Therefore, the mean absolute value of the first gradient of all connection weight items in the neural network is often greater than the mean absolute value of the first gradient of each connection weight item in the layer. To solve this problem, as an example, for the training process of the same neural network, the network gradient threshold Need to be greater than the weight gradient threshold and layer gradient threshold , otherwise it is difficult to perform effective gradient fusion.

[0059] As an example, the server may use the fused gradient calculated in S204 to implement a network weight update for the target connection weight. The specific update method may refer to a general gradient descent algorithm: (14) in, is the learning rate, and as the weights are updated with gradient descent, the neural network can evolve in the reverse direction of the fused gradient with a constant or optimized step size.

[0060] It can be understood that in a certain round of training, if the target connection weight determined in the neural network is updated with a fused gradient, then in that round of training, other connection weights in the neural network except the target connection weight are updated with an unfused gradient, i.e., the first gradient.

[0061] When the training is repeated until the obtained neural network meets the set requirements, the set requirements can be implemented in multiple ways. As an example, the neural network meeting the set requirements can mean that the neural network can complete the specified task. As another example, the neural network meeting the set requirements can mean that the number of repeated training of the neural network reaches a threshold number. Therefore, there is no limitation on the specific implementation of the above set requirements that the neural network needs to meet. As an example, the repeated training in S205 can mean repeating the process of S201-S204 to train the neural network until the neural network meets the set requirements.

[0062] Considering that the above-mentioned gradient threshold is set as the layer gradient threshold or the network gradient threshold, and the relationship requirement is the second relationship requirement or the third relationship requirement, the average operation performed by the server on the absolute value of the first gradient will lose specific gradient intensity information, thereby generating a judgment error and losing part of the loss function gradient information. In response to this problem, if the set gradient threshold is the layer gradient threshold and the relationship requirement is the second relationship requirement, or the set gradient threshold is the network gradient threshold and the relationship requirement is the third relationship requirement, then the method of updating each connection weight of the neural network is alternately performed in each round of training in the first method and the second method; the first method includes: updating the target connection weight with the fused gradient, and updating the connection weights in the neural network except the target connection weight with the first gradient; the second method includes: updating all connection weight items in the neural network with the first gradient. By alternating the first method and the second method in multiple rounds of training to update each connection weight of the neural network, the effective transmission of the loss function information can be guaranteed. As another example, if the set gradient threshold is a weight gradient threshold, and the relationship requirement is the first relationship requirement, there is no restriction on alternating between different updating methods in multiple rounds of training, and when the server performs repeated training, each connection weight of the neural network can be updated in the first manner in each round of training.

[0063] It is worth noting that the main principle of the neural network training method described in any embodiment of the present application described above lies in the directionality of the gradient noise. Therefore, for the initial direction generation of the directional gradient noise, only the gradient noise with a fixed direction can produce the most effective search for the parameter space. Therefore, the server only generates the initial direction of the gradient noise once and reuses it in subsequent training. If the server repeatedly generates and uses different gradient noise directions during the training process, it is impossible to complete the effective search of the parameter space. In addition, since there are many local minima in the over-parameterized neural network, when the server generates the initial direction of the directional gradient noise, the specific selection of the initial direction may not be restricted.

[0064] Figure 3 A schematic diagram of a specific training scenario of a neural network provided in an embodiment of the present application: Among them, after the training data is input into the network to be trained, the server can obtain the network output of the neural network under this connection weight. By comparing the network output with the training sample annotation, the server can calculate the gradient according to the back propagation algorithm, determine the absolute value of the gradient corresponding to each connection of the network, the absolute value mean of the gradient of each layer and the absolute value mean of the gradient of the entire network. This process can be regarded as the calculation part of the gradient descent training process based on the fusion of directed gradient noise; then, the server can generate a directional gradient noise with a fixed sign for each connection weight, and confirm the relationship between the absolute value of each gradient and the gradient threshold by setting a gradient threshold, fuse the directional gradient noise with the back propagation calculated gradient (i.e., the first gradient), and finally use the fused gradient as the new gradient value to update each connection weight of the network. This process can be regarded as the training part of the gradient descent training process based on the fusion of directed gradient noise. The server will repeat the calculation part and the training part until the network can meet the set requirements, such as correctly identifying the image category of the sample to be trained in the visual processing scenario.

[0065] As an example, the neural network training method described in any of the above embodiments may be applicable to a machine learning task in which supervised learning or self-supervised learning is applied during the training process, wherein the machine learning task includes a classification task, a recognition task, or a generation task, and / or, the neural network training method described in any of the above embodiments may be applicable to a network architecture in which supervised learning or self-supervised learning is applied during the training process, wherein the network architecture includes a deep neural network, a convolutional neural network, and a Transformer architecture, and its initialization mode may depend on the network architecture.

[0066] Corresponding to the above method embodiment, the present application embodiment also provides a neural network training device, which is at least applied to visual processing scenarios, see Figure 4 As shown, the device may include: An input unit 401 is used to input a training sample into a neural network to be trained, and obtain an output result of the neural network under each current connection weight; the training sample at least includes visual data; A determination unit 402, configured to determine a first gradient of each of the connection weights based on the output result and the labeling information of the training sample; A generating unit 403, configured to generate directional gradient noise for each of the connection weights; wherein the initial direction of the directional gradient noise is randomly generated and remains fixed during the training process; An updating unit 404 is used to fuse the directional gradient noise corresponding to the target connection weight and the first gradient for the target connection weight that meets the relationship requirement between the first gradient and the set gradient threshold to obtain a fused gradient, and update the target connection weight according to the fused gradient; The training unit 405 is used to repeat the training until the obtained neural network meets the set requirements.

[0067] As an example, the determination unit 402 is specifically used to convert the training sample with the labeled information into a target vector, and convert the output result into a vector form, determine the loss function of the neural network based on the target vector and the output result converted into a vector form; and determine the first gradient based on the loss function.

[0068] As an example, the oriented gradient noise is obtained by multiplying the initial direction of the oriented gradient noise by the intensity of the oriented gradient noise; wherein the initial direction of the oriented gradient noise is randomly generated according to a specified probability; and the intensity of the oriented random gradient noise is generated in a specified manner and is assigned a positive value.

[0069] As an example, the set gradient threshold includes at least one of a weight gradient threshold, a layer gradient threshold and a network gradient threshold; wherein the network gradient threshold is greater than the weight gradient threshold and the layer gradient threshold; when the set gradient threshold includes the weight gradient threshold, the relationship requirement includes a first relationship requirement between the absolute value of the first gradient and the weight gradient threshold; when the neural network includes multiple layers, the set gradient threshold includes the layer gradient threshold; the relationship requirement includes a second relationship requirement between the mean of the absolute values ​​of the first gradients of each connection weight in the layer and the layer gradient threshold, and the target connection weight includes all connection weight items in the target layer that meet the second relationship requirement; when the set gradient threshold includes the network gradient threshold, the relationship requirement includes a third relationship requirement between the mean of the absolute values ​​of the first gradients of all connection weight items in the neural network and the network gradient threshold, and the target connection weight includes all connection weight items of the neural network that meet the third relationship requirement.

[0070] As an example, satisfying the first relationship requirement includes: the absolute value of the first gradient is less than or equal to the weight gradient threshold; satisfying the second relationship requirement includes: the average of the absolute values ​​of the first gradients of each connection weight in the layer is less than or equal to the layer gradient threshold; satisfying the third relationship requirement includes: the average of the absolute values ​​of the first gradients of all connection weight items in the neural network is less than or equal to the network gradient threshold.

[0071] As an example, the updating unit 404 is specifically configured to sum the directional gradient noise corresponding to the target connection weight and the first gradient to obtain the fused gradient.

[0072] As an example, the updating unit 404 is also used to update each connection weight of the neural network in a first manner and a second manner alternately in each round of training if the set gradient threshold is the layer gradient threshold and the relationship requirement is the second relationship requirement, or the set gradient threshold is the network gradient threshold and the relationship requirement is the third relationship requirement; the first manner includes: updating the target connection weight with the fused gradient, and updating the connection weights in the neural network other than the target connection weight with the first gradient; the second manner includes: updating all connection weight items in the neural network with the first gradient.

[0073] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the neural network training method described in any of the above embodiments.

[0074] The present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes: Processor 501; Memory 502 for storing processor executable instructions; The processor 501 is configured to implement the neural network training method described in any one of the above embodiments.

[0075] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the training method of the neural network described in any of the above embodiments is implemented.

[0076] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0077] The above embodiments can be applied to one or more computer devices, where the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0078] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0079] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0080] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0081] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this application.

[0082] The description of "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0083] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0084] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0085] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for training a neural network, at least for visual processing scenarios, characterized in that: include: Inputting the training samples into the neural network to be trained, and obtaining the output result of the neural network under each current connection weight; The training samples at least include visual data; Determining a first gradient of each connection weight based on the output result and the labeling information of the training sample; Generating directional gradient noise for each of the connection weights; wherein the initial direction of the directional gradient noise is randomly generated and remains fixed during the training process; For a target connection weight that satisfies the relationship between the first gradient and the set gradient threshold, the directional gradient noise corresponding to the target connection weight and the first gradient are fused to obtain a fused gradient, and the target connection weight is updated according to the fused gradient; Repeat the training until the obtained neural network meets the set requirements.

2. The method according to claim 1, characterized in that The determining, based on the output result and the labeling information of the training sample, a first gradient of each connection weight includes: Converting the training sample with the labeled information into a target vector, converting the output result into a vector form, and determining the loss function of the neural network based on the target vector and the output result converted into a vector form; Based on the loss function, the first gradient is determined.

3. The method according to claim 1, characterized in that The oriented gradient noise is obtained by multiplying the initial direction of the oriented gradient noise by the intensity of the oriented gradient noise; Wherein, the initial direction of the directional gradient noise is randomly generated according to a specified probability; The intensity of the directional random gradient noise is generated in a specified manner and is assigned a positive value.

4. The method according to claim 1, characterized in that The set gradient threshold includes at least one of a weight gradient threshold, a layer gradient threshold and a network gradient threshold; wherein the network gradient threshold is greater than the weight gradient threshold and the layer gradient threshold; When the set gradient threshold includes the weight gradient threshold, the relationship requirement includes a first relationship requirement between an absolute value of a first gradient and the weight gradient threshold; When the neural network includes multiple layers, the set gradient threshold includes the layer gradient threshold; the relationship requirement includes a second relationship requirement between the mean of the absolute values ​​of the first gradients of each connection weight in the layer and the layer gradient threshold, and the target connection weight includes all connection weight items in the target layer that meet the second relationship requirement; When the set gradient threshold includes the network gradient threshold, the relationship requirement includes a third relationship requirement between the mean of the absolute values ​​of the first gradients of all connection weight items in the neural network and the network gradient threshold, and the target connection weight includes all connection weight items of the neural network that meet the third relationship requirement.

5. The method according to claim 4, characterized in that Meeting the first relationship requirement includes: the absolute value of the first gradient is less than or equal to the weight gradient threshold; Meeting the second relationship requirement includes: the average of the absolute values ​​of the first gradients of the connection weights in the layer is less than or equal to the layer gradient threshold; Satisfying the third relationship requirement includes: the average of the absolute values ​​of the first gradients of all connection weight items in the neural network is less than or equal to the network gradient threshold.

6. The method according to claim 1, characterized in that The fusing the directional gradient noise corresponding to the target connection weight and the first gradient includes: The oriented gradient noise corresponding to the target connection weight and the first gradient are summed to obtain the fused gradient.

7. The method according to claim 4, characterized in that The method further comprises: If the set gradient threshold is the layer gradient threshold and the relationship requirement is the second relationship requirement, or, the set gradient threshold is the network gradient threshold and the relationship requirement is the third relationship requirement, then the method of updating each connection weight of the neural network is performed alternately in each round of training in the first method and the second method; The first method includes: updating the target connection weight with the fused gradient, and updating the connection weights other than the target connection weight in the neural network with the first gradient; The second method includes: All connection weight items in the neural network are updated with the first gradient.

8. A neural network training device, at least for visual processing scenarios, characterized in that: include: An input unit, used to input the training sample into the neural network to be trained, and obtain the output result of the neural network under each current connection weight; The training samples at least include visual data; A determination unit, configured to determine a first gradient of each of the connection weights based on the output result and the labeling information of the training sample; A generating unit, configured to generate directional gradient noise for each of the connection weights; wherein the initial direction of the directional gradient noise is randomly generated and remains fixed during the training process; An updating unit, configured to fuse the directional gradient noise corresponding to the target connection weight and the first gradient for a target connection weight that satisfies the relationship requirement between the first gradient and the set gradient threshold, obtain a fused gradient, and update the target connection weight according to the fused gradient; The training unit is used to repeat the training until the obtained neural network meets the set requirements.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images