Neural network model training method, image processing method and device

Abstract

The present application discloses a training method, image processing method and device for a neural network model in the field of artificial intelligence. The training method includes: inputting training data into the neural network model for feature extraction, obtaining the first weight gradient of the neural network model based on the extracted features; obtaining candidate weight parameters, the partial derivative of the function value of the target loss function with respect to the candidate weight parameters is 0, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label, and the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the first weight gradient; updating the parameters of the neural network model based on the candidate weight parameters. The method of the present application can accelerate the training process and improve the training efficiency of the neural network model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a training method for a neural network model, an image processing method, and an apparatus. Background Art

[0002] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0003] With the rapid development of artificial intelligence (AI) technology, the performance of neural networks (e.g., convolutional neural networks) has continued to improve, and neural networks have achieved significant success in processing and analyzing various media signals, including images, video, and speech. To improve the training speed and inference accuracy of neural network models, a batch normalization (BN) layer is often added to neural network models. Incorporating this layer into deep neural networks can effectively reduce fluctuations during training and accelerate convergence. It also mitigates the deep neural network's sensitivity to noise, improving its robustness.

[0004] However, the operations of the BN layer are mainly non-matrix operations, and the computational complexity depends on the number of input data (features). Usually, during the training process of the neural network model, the number of input data (features) is more than 100,000, which leads to a high computational complexity of the BN layer and affects the processing efficiency. Especially when the main computing core of the current mainstream neural network processing chip is matrix operation, BN cannot effectively utilize the matrix operation capabilities of the existing chips, and requires a large overhead, which affects the model training speed.

[0005] Therefore, how to improve the training efficiency of neural network models has become a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The present application provides a training method, an image processing method and a device for a neural network model, which can accelerate the training process and improve the training efficiency of the neural network model.

[0007] In a first aspect, a training method for a neural network model is provided, the method comprising: obtaining training data; inputting the training data into the neural network model for feature extraction, and obtaining a first prediction label based on the extracted features; determining a function value of a first loss function corresponding to the first prediction label based on the first prediction label and the true label of the training data, the function value of the first loss function corresponding to the first prediction label being used to indicate the difference between the first prediction label and the true label of the training data; calculating a first weight gradient of the neural network model based on the function value of the first loss function corresponding to the first prediction label; obtaining candidate weight parameters, the partial derivative of the function value of the target loss function with respect to the candidate weight parameters being 0, wherein the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label, the function value of the second loss function corresponding to the first prediction label being used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the first weight gradient, the weight change being the difference between the candidate weight parameters and the weight parameters of the neural network model; and updating the parameters of the neural network model based on the candidate weight parameters.

[0008] In the scheme of the embodiment of the present application, the difference between the candidate weight parameters and the weight parameters of the neural network model is controlled by the second loss function, thereby controlling the parameter fluctuation range during the training process, ensuring the controllability of the gradient fluctuation, avoiding the gradient fluctuation leading to training divergence and poor generalization, improving the training stability, and accelerating the training process; moreover, the first loss function is used to continuously improve the reasoning accuracy of the neural network model during the training process, and the second loss function can achieve a convergence trend consistent with the first loss function, ensuring the reasoning accuracy of the neural network model. For example, the scheme of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be understood as the original loss function of the neural network model. The original loss function is used to make the predicted labels output by the trained neural network model as similar as possible to the true labels of the training data. The scheme of the embodiment of the present application can improve the training stability and accelerate the training process while ensuring the realization of the original prediction task.

[0009] In addition, the training overhead of the neural network model using the first loss function is almost the same as that of the neural network model using the target loss function. That is to say, compared with the neural network model using the first loss function, the solution of the embodiment of the present application can improve the training speed and reasoning accuracy of the neural network model without increasing the overhead. On the one hand, the solution of the embodiment of the present application greatly saves training resources and training costs. On the other hand, the same training resources can support the training of more neural network models.

[0010] In addition, the solution of the embodiment of the present application can be completed on the basis of the existing neural network, that is, a new optimization goal is set based on the existing neural network model, which can be completed without the user's perception, that is, without the user making any modifications, thereby improving the user experience.

[0011] The type of training data depends on the task of the neural network model. For example, if the neural network model is used for image processing, the training data can be images.

[0012] The type of label depends on the task being performed by the neural network model. For example, if the neural network model is used for image classification, the label can be a classification label, representing the image's category. The true label of the training data is the true category to which the image belongs. The predicted label can be the predicted category of the image or the predicted probability of the category to which the image belongs.

[0013] The specific type of the first loss function is related to the task of the neural network model. For example, if the neural network model is used for image processing, the first loss function may be a cross entropy loss function. The embodiment of the present application does not limit the type of the first loss function.

[0014] Exemplarily, updating the neural network model according to the candidate weight parameters may include: using the candidate weight parameters as weight parameters in the updated neural network model.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: obtaining an original neural network model; deleting the BN layer in the original neural network model to obtain the neural network model.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: obtaining at least one second prediction label based on features output by at least one layer in the neural network model; determining a function value of a first loss function corresponding to at least one second prediction label based on the at least one second prediction label and the true label of the training data, the function value of the first loss function corresponding to the at least one second prediction label is used to indicate the difference between the at least one second prediction label and the true label of the training data; calculating at least one second weight gradient of the neural network model based on the function value of the first loss function corresponding to the at least one second prediction label; and the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to the at least one second prediction label, the function value of the second loss function corresponding to the at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient.

[0017] According to the solution of the embodiment of the present application, a second prediction label is obtained based on the features output by at least one layer in the neural network model, and then a second weight gradient is calculated based on the second prediction label as an auxiliary reference for updating the parameters of the neural network model, which can reduce gradient fluctuations and improve the stability of training.

[0018] In combination with the first aspect, in some implementations of the first aspect, the method further includes: obtaining at least one second prediction label based on features output by at least one downsampling layer in the neural network model; determining a function value of a first loss function corresponding to at least one second prediction label based on the at least one second prediction label and the true label of the training data, and the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between the at least one second prediction label and the true label of the training data; calculating at least one second weight gradient of the neural network model based on the function value of the first loss function corresponding to the at least one second prediction label; and the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, and the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient.

[0019] According to the solution of the embodiment of the present application, a second prediction label is obtained based on the features output by the downsampling layer, and then a second weight gradient is calculated based on the second prediction label as an auxiliary reference for updating the parameters of the neural network model, which can further reduce gradient fluctuations and improve the stability of training.

[0020] In addition, only the second prediction label obtained by the features output by the downsampling layer is used for auxiliary training, which increases the amount of computation less and does not occupy a large amount of training resources. In other words, with almost no increase in overhead, the gradient fluctuation is further reduced and the stability of training is improved.

[0021] A downsampling layer is a layer that makes the resolution of the output feature map smaller than the resolution of the input feature map. If a neural network model includes N downsampling layers, then the neural network model includes N locations where the resolution is reduced.

[0022] M second prediction labels can be obtained according to the features output by M downsampling layers in the N downsampling layers.

[0023] Obtaining the M second prediction labels according to the features output by the M downsampling layers may be as follows: inputting the features output by the M downsampling layers into the M output layers respectively to obtain the M second prediction labels.

[0024] The M output layers have the same structure as the output layer of the neural network model. The structure of the output layer is related to the task of the neural network model. For example, when the neural network model is used for image classification, the output layer can also be called a classification layer. Specifically, the classification layer can be a softmax layer, which can output the predicted classification label of the image.

[0025] For example, the function value of the objective loss function can satisfy the following formula.

[0026] L all =L new_n +L new ;

[0027] Among them, L all Represents the function value of the target loss function, L new_n Represents the function value of the second loss function corresponding to the second predicted label. If the second predicted label is obtained based on the features output by the downsampling layer, L new_n It can also be understood as the function value of the second loss function corresponding to the nth downsampling layer.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, including: the weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

[0029] According to the solution of the embodiment of the present application, the proportion of the function value of the second loss function corresponding to the second prediction label in the function value of the target loss function decreases with the number of updates of the neural network model. This can increase the stability of the neural network model in the early stage of training. In the later stage of training, when its proportion is reduced to 0, the neural network model completes training only based on the second loss function corresponding to the first prediction label, further reducing the amount of calculation and reducing training overhead.

[0030] For example, the function value L of the target loss function all The following formula can be satisfied.

[0031] L all =a n *L new_n +L new ;

[0032] a n Indicates L new_n The corresponding weight, a n It is negatively correlated with the number of updates of the neural network model.

[0033] In combination with the first aspect, in certain implementations of the first aspect, the neural network model includes a first feature normalization layer, which is located before the output layer of the neural network model, and the first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0034] In the solution of the embodiment of the present application, feature normalization is performed to control the forward fluctuation within a certain range, making the entire training process more stable and further accelerating the training process.

[0035] Specifically, the first FN layer performs feature normalization processing on the features input into the first FN layer, and inputs the processed results into the output layer of the neural network model, and the output layer outputs the first prediction label.

[0036] For example, feature normalization processing can be: dividing the input features by the square root of all input features. Specifically, the result of feature normalization processing can satisfy the following formula:

[0037]

[0038] Among them, x i represents the i-th input feature, m represents the number of input features, Represents the result after feature normalization of the i-th input feature.

[0039] In combination with the first aspect, in certain implementations of the first aspect, the neural network model includes at least one second feature normalization layer, the at least one second feature normalization layer is respectively located after at least one downsampling layer, the at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer, and obtain at least one second prediction label based on the features output by the at least one downsampling layer in the neural network model, including: inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and obtaining at least one second prediction label based on the processed results.

[0040] In the solution of the embodiment of the present application, the forward fluctuation is controlled within a certain range through feature normalization processing, which makes the entire training process more stable and can further accelerate the training process. In particular, compared with the neural network model using modules such as BN, the neural network model using modules such as BN usually requires a large number of BN modules, which takes up a large amount of training overhead. The solution of the embodiment of the present application can achieve the same convergence speed and reasoning accuracy as the neural network model using modules such as BN, and the solution of the embodiment of the present application only sets the FN layer before the downsampling layer and the last output layer of the neural network model. The required FN layer is relatively small, and the training overhead is much less than the training overhead required for the neural network model using modules such as BN.

[0041] In combination with the first aspect, in certain implementations of the first aspect, the function value of the second loss function corresponding to the first prediction label is used to indicate the sum of the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the first weight gradient.

[0042] In combination with the first aspect, in some implementations of the first aspect, the second loss function satisfies the following formula:

[0043]

[0044] Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function indicating the difference between the weight change amount and the weight gradient, d2 represents a function indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

[0045] That is to say, the candidate weight parameter w can satisfy the following formula:

[0046]

[0047] For example, d1 can be a distance metric function, that is, the weight change ww k and The distance between them represents the difference between them. For example, d1 can be a cosine similarity function. d2 can be a distance metric function, that is, by using w and w k The distance between them represents the difference between the two. For example, d2 can be the cosine similarity function

[0048] Specifically, by making L new The partial derivative of w is 0, and the candidate weight parameter is obtained. The candidate weight parameter is the one that makes L new The weight parameter with the smallest function value.

[0049] In combination with the first aspect, in certain implementations of the first aspect, updating the parameters of the neural network model according to the candidate weight parameters includes: updating the parameters of the neural network model according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

[0050] In the embodiment of the present application, by setting the learning rate, the parameter update process can be smoothed, the parameter fluctuation range can be further controlled, the stability of the training can be further improved, the convergence process can be accelerated, and the training efficiency can be improved.

[0051] In conjunction with the first aspect, in certain implementations of the first aspect, the weight parameters of the updated neural network model satisfy the following formula:

[0052] w k+1 =(ww k )*lr+w k ;

[0053] Among them, w k+1 Represents the weight parameters of the updated neural network model, w k Represents the weight parameters of the neural network model, w represents the candidate weight parameters, and lr represents the learning rate.

[0054] In a second aspect, an image processing method is provided, the method comprising: obtaining an image to be processed; performing feature extraction on the image to be processed using a target neural network model, and obtaining a processing result of the image to be processed based on the features extracted by the target neural network model; wherein the target neural network model is obtained by inputting training data into the neural network model for feature extraction, obtaining a first predicted label based on the features extracted by the neural network model, determining a function value of a first loss function corresponding to the first predicted label based on the first predicted label and the true label of the training data, the function value of the first loss function corresponding to the first predicted label is used to indicate the difference between the first predicted label and the true label of the training data, calculating a first weight gradient of the neural network model based on the function value of the first loss function corresponding to the first predicted label, obtaining a candidate weight parameter, the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0, and updating the parameters of the neural network model based on the candidate weight parameter, wherein the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first predicted label, the function value of the second loss function corresponding to the first predicted label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

[0055] The neural network model updating method in the first aspect can be used to obtain a target neural network model. The target neural network model can be a trained neural network model, and the trained neural network model can be used to classify the image to be processed.

[0056] In this application, since the target neural network model is obtained by adopting the method of the first aspect mentioned above, it is more in line with or close to the application requirements of the neural network model. Using such a neural network model for image classification can achieve better image classification effects (for example, the classification results are more accurate, etc.).

[0057] During the training process, the size of the weight change of the second loss function is used to control the parameter fluctuation range during the training process, thereby ensuring the controllability of the gradient fluctuation, avoiding the gradient fluctuation from leading to training divergence and poor generalization, improving the training stability, and accelerating the training process; moreover, the first loss function is used to continuously improve the reasoning accuracy of the neural network model during the training process, and the second loss function can achieve a convergence trend consistent with the first loss function, ensuring the reasoning accuracy of the neural network model. For example, the scheme of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be understood as the original loss function of the neural network model. The original loss function is used to make the predicted labels output by the trained neural network model as similar as possible to the true labels of the training data. The scheme of the embodiment of the present application can improve the training stability and accelerate the training process while ensuring the realization of the original prediction task.

[0058] In combination with the second aspect, in certain implementations of the second aspect, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient, at least one second prediction label is obtained based on the features output by at least one layer in the neural network model, the function value of the second loss function corresponding to at least one second prediction label is determined based on at least one second prediction label and the true label of the training data, the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between at least one second prediction label and the true label of the training data, and at least one second weight gradient is calculated based on the function value of the first loss function corresponding to at least one second prediction label.

[0059] In combination with the second aspect, in certain implementations of the second aspect, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient, at least one second prediction label is obtained based on the features output by at least one downsampling layer in the neural network model, the function value of the second loss function corresponding to at least one second prediction label is determined based on at least one second prediction label and the true label of the training data, the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between at least one second prediction label and the true label of the training data, and at least one second weight gradient is calculated based on the function value of the first loss function corresponding to at least one second prediction label.

[0060] In combination with the second aspect, in certain implementations of the second aspect, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, including: the weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

[0061] In combination with the second aspect, in certain implementations of the second aspect, the target neural network model includes a first feature normalization layer, which is located before the output layer of the target neural network model, and the first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0062] In combination with the second aspect, in certain implementations of the second aspect, the neural network model includes at least one second feature normalization layer, and the at least one second feature normalization layer is located after at least one downsampling layer. The at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer. The at least one second prediction label is obtained by inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and the processed result is obtained.

[0063] In combination with the second aspect, in certain implementations of the second aspect, the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the sum of the differences between the weight change and the first weight gradient.

[0064] In conjunction with the second aspect, in some implementations of the second aspect, the second loss function satisfies the following formula:

[0065]

[0066] Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function indicating the difference between the weight change amount and the weight gradient, d2 represents a function indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

[0067] In combination with the second aspect, in certain implementations of the second aspect, updating the parameters of the neural network model according to the candidate weight parameters includes: updating the parameters of the neural network model according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

[0068] In conjunction with the second aspect, in certain implementations of the second aspect, the weight parameters of the updated neural network model satisfy the following formula:

[0069] w k+1 =(ww k )*lr+w k ;

[0070] Among them, w k+1 Represents the weight parameters of the updated neural network model, w k Represents the weight parameters of the neural network model, w represents the candidate weight parameters, and lr represents the learning rate.

[0071] In a third aspect, a training device for a neural network model is provided, the device comprising a module or unit for executing the method in the above-mentioned first aspect and any one of the implementations of the first aspect.

[0072] In a fourth aspect, an image processing device is provided, comprising a module or unit for executing the method in the second aspect and any one of the implementations of the second aspect.

[0073] It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect, third aspect and fourth aspect.

[0074] In a fifth aspect, a training device for a neural network model is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0075] The processor in the fifth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). Among them, the TPU is a dedicated integrated circuit for artificial intelligence accelerators fully customized by Google for machine learning.

[0076] In a sixth aspect, an image processing device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in the second aspect and any one of the implementation methods of the second aspect.

[0077] The processor mentioned in the sixth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processor (NPU), a tensor processing unit (TPU), and so on. The Tensor Processing Unit (TPU) is a custom AI accelerator application-specific integrated circuit (ASIC) developed by Google for machine learning.

[0078] In a seventh aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first aspect or the second aspect.

[0079] In an eighth aspect, a computer program product comprising instructions is provided, which, when run on a computer, enables the computer to execute the method in any one of the implementations of the first or second aspects above.

[0080] In a ninth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementations of the first or second aspect.

[0081] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods of the first aspect or the second aspect.

[0082] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided by an embodiment of the present application;

[0084] Figure 2 A schematic diagram of the structure of a system architecture provided in an embodiment of the present application;

[0085] Figure 3 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0086] Figure 4 A schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0087] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0088] Figure 6 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0089] Figure 7 A schematic block diagram of a training device for a neural network model provided in an embodiment of the present application;

[0090] Figure 8 A schematic flow chart of a training method for a neural network model provided in an embodiment of the present application;

[0091] Figure 9 A schematic flow chart of feature normalization processing provided in an embodiment of the present application;

[0092] Figure 10 is a schematic block diagram of a neural network model provided in an embodiment of the present application;

[0093] Figure 11 is a schematic block diagram of another neural network model provided in an embodiment of the present application;

[0094] Figure 12is a schematic flow chart of the image processing method provided in an embodiment of the present application;

[0095] Figure 13 is a schematic block diagram of a training device for a neural network model provided in an embodiment of the present application;

[0096] Figure 14 is a schematic block diagram of an image processing device provided in an embodiment of the present application;

[0097] Figure 15 is a schematic block diagram of a training device for a neural network model provided in an embodiment of the present application;

[0098] Figure 16 It is a schematic block diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0100] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0101] The following is a detailed explanation of the above artificial intelligence thematic framework from the two dimensions of "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

[0102] The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom."

[0103] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0104] (1) Infrastructure:

[0105] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and provides support through a basic platform.

[0106] The infrastructure can communicate with the outside world through sensors, and the computing power of the infrastructure can be provided by smart chips.

[0107] The smart chips here can be hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), and field programmable gate array (FPGA).

[0108] The basic platform of the infrastructure can include related platform guarantees and support such as distributed computing framework and network, and can include cloud storage and computing, interconnected networks, etc.

[0109] For example, for infrastructure, data can be obtained through sensors and external communication, and then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0110] (2) Data:

[0111] The data layer above the infrastructure represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0112] (3) Data processing:

[0113] The above data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.

[0114] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0115] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0116] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0117] (4) General abilities:

[0118] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0119] (5) Smart products and industry applications:

[0120] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0121] The embodiments of the present application can be applied in many fields of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities and other fields.

[0122] Specifically, the embodiments of the present application can be applied in fields that require the use of (deep) neural networks, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution and natural language processing.

[0123] The following is a brief introduction to the two application scenarios of album picture classification and safe city.

[0124] Album picture categories:

[0125] When a user stores a large number of pictures on a terminal device (e.g., a mobile phone) or a cloud disk, identifying images in the album can facilitate the user or the system to categorize and manage the album, thereby improving the user experience.

[0126] By using the training method of the neural network model of the embodiment of the present application, a neural network suitable for album classification can be obtained or optimized. This neural network can then be used to classify images, thereby labeling images of different categories for easier viewing and searching by users. Furthermore, these image classification labels can also be provided to the album management system for classification management, saving users' management time, improving album management efficiency, and enhancing the user experience.

[0127] Attribute recognition in the safe city scenario:

[0128] In the safe city scenario, multiple attribute recognition is required, such as pedestrian attribute recognition and cycling attribute recognition. Deep neural networks play an important role in multiple attribute recognition with their powerful capabilities. By adopting the training method of the neural network model of the embodiment of the present application, a neural network suitable for attribute recognition in the safe city scenario can be obtained or optimized. Next, the neural network can be used to process the input road image to identify different attribute information in the road image.

[0129] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0130] (1) Neural Network

[0131] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be:

[0132]

[0133] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0134] (2) Deep Neural Networks

[0135] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0136] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0137] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0138] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0139] (3) Convolutional Neural Networks

[0140] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as a position-independent method for extracting image information. Convolution kernels can be formalized as matrices of random size, and during CNN training, the kernels can be learned to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0141] (4) Residual Network

[0142] The residual network (ResNet) is a convolutional neural network structure formed by cascading multiple residual units, among which the residual unit is its most important feature.

[0143] Simply put, the residual network can be: in addition to the layer-by-layer connection between multiple hidden layers in the deep neural network, for example, the first hidden layer is connected to the second hidden layer, the second hidden layer is connected to the third hidden layer, and the third hidden layer is connected to the fourth hidden layer (this is a data operation path of the neural network, which can also be called neural network transmission). The residual network also has an additional direct branch, which is directly connected from the first hidden layer to the fourth hidden layer, that is, skipping the processing of the second and third hidden layers, and transmitting the data of the first hidden layer directly to the fourth hidden layer for calculation, which constitutes a residual unit.

[0144] Based on the direct gradient conduction characteristics of the residual unit, the depth of the ResNet network structure can reach hundreds or even thousands of layers, such as the typical ResNet50 and ResNet101.

[0145] (5) Loss function

[0146] During deep neural network training, the goal is to ensure that the output of a deep neural network is as close as possible to the desired target value. This is done by comparing the network's predictions with the desired target values ​​and then updating the weight vectors of each layer based on the difference between the two. (Of course, this initial update typically involves a process of pre-configuring parameters for each layer of the deep neural network.) For example, if the network's prediction is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the deep neural network can predict the desired target value or a value very close to it. Therefore, it is necessary to predefine how to compare the difference between the predicted and target values. This is known as the loss function or objective function, which is a key equation used to measure the difference between the predicted and target values. For example, a higher loss indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss. Generally speaking, lower loss indicates higher quality training, while higher loss indicates lower quality training. Similarly, lower loss fluctuations indicate more stable training, while higher loss fluctuations indicate more unstable training.

[0147] (6) Backpropagation algorithm

[0148] Neural networks can use the back propagation (BP) algorithm to modify the parameters of the neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters of the neural network model, thereby converging the error loss. The BP algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0149] For example, the loss value generated during each training phase of a neural network model is passed through the model layer by layer, from back to front. As it passes to each layer, the update to that layer's parameters is calculated (partial derivative operation). This update is related to the gradient. The gradient value is linearly positively correlated with the loss value.

[0150] After calculating the gradient, the parameters of each layer of the neural network need to be updated. The direction of the update is determined by the gradient, and the size of the update is determined by the learning rate (lr). The learning rate usually needs to be manually adjusted by the algorithm developer during the training process.

[0151] (7) Batch Normalization

[0152] Batch normalization (BN) is a data (or feature) processing method used in deep neural networks. Specifically, batch normalization is performed by subtracting the mean of the input data (features), dividing it by its variance, and then linearly adjusting the result. By adding this layer to a deep neural network, it can effectively reduce fluctuations during training and accelerate the convergence of the training.

[0153] like Figure 2 As shown, the embodiment of the present application provides a system architecture 100. Figure 2 In the embodiment of the present invention, the data acquisition device 160 is used to acquire training data. For the image processing method of the present invention, the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.

[0154] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0155] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output value with the target value until the difference between the value output by the training device 120 and the target value is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0156] The above-mentioned target model / rule 101 can be used to implement the image processing method of the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network model. For example, a convolutional neural network or a residual network, etc. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0157] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 2The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) or virtual reality (VR), a vehicle terminal, etc. It can also be a server or a cloud. Figure 2 In the embodiment of the present application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: an image to be processed input by the client device.

[0158] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0159] Finally, the I / O interface 112 returns the processing result, such as the image classification result obtained above, to the client device 140 for providing to the user.

[0160] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0161] exist Figure 2 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0162] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0163] like Figure 2 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be the neural network in the embodiment of the present application. Specifically, the neural network constructed in the embodiment of the present application can be CNN, deep convolutional neural networks (DCNN), residual network, etc.

[0164] Since CNN is a very common neural network, the following Figure 3 This section focuses on a detailed introduction to the structure of CNNs. As mentioned above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves performing multiple levels of learning at different levels of abstraction through algorithms that update neural network models. As a deep learning architecture, a CNN is a feed-forward artificial neural network, in which individual neurons respond to input images.

[0165] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 3 As shown. Figure 3 In the example, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and pass the obtained image to be processed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, thereby obtaining an image processing result. Figure 3 The internal layer structure of CNN200 is introduced in detail.

[0166] Convolutional layer / pooling layer 220:

[0167] Convolutional layer:

[0168] like Figure 3The convolutional layer / pooling layer 220 shown may include layers 221-226. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0169] The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0170] Convolution layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from an input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), thereby extracting specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used. Instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix is ​​used to extract edge information, another weight matrix is ​​used to extract specific colors from an image, and yet another weight matrix is ​​used to blur unwanted noise in an image. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by these weight matrices of the same size are also of the same size. The extracted convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0171] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0172] When the convolutional neural network 200 has multiple convolutional layers, the convolutional layers (for example, 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0173] Pooling layer:

[0174] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 3 Each layer 221-226 illustrated in Figure 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the image processing process, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0175] Neural Network Layer 230:

[0176] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 3 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type, for example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0177] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 3 The propagation from 210 to 240 is forward propagation) and the reverse propagation (such as Figure 3 The propagation from 240 to 210 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will be updated to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0178] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 4 As shown. Figure 4 In FIG, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. Figure 3 compared to, Figure 4 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 220 are operated in parallel, and the features extracted from each layer are input to the neural network layer 230 for processing.

[0179] It should be noted that Figure 3 and Figure 4 The convolutional neural network shown is only an example of two possible convolutional neural networks for the image processing method of an embodiment of the present application. In specific applications, the convolutional neural network adopted by the image processing method of an embodiment of the present application may also exist in the form of other network models.

[0180] In addition, the structure of the convolutional neural network obtained by the search method of the neural network structure of the embodiment of the present application can be as follows: Figure 3 and Figure 4 The convolutional neural network structure is shown in .

[0181] Figure 5 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor 50. The chip can be set as follows Figure 2 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 3 and Figure 4 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 5 is implemented in the chip shown.

[0182] The neural network processor NPU 50 is mounted as a coprocessor on the host central processing unit (CPU), and the host CPU assigns tasks. The core of the NPU is the arithmetic circuit 503. The controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.

[0183] In some implementations, the arithmetic circuit 503 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0184] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 502 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 501 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 508.

[0185] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization (BN), local response normalization, etc.

[0186] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0187] The unified memory 506 is used to store input data and output data.

[0188] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0189] The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0190] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0191] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0192] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0193] in, Figure 3 and Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0194] Introduced above Figure 2 The execution device 110 in the embodiment of the present application can execute each step of the image processing method. Figure 3 and Figure 4 The CNN model shown and Figure 5 The chip shown can also be used to execute the various steps of the image processing method of the embodiment of the present application.

[0195] The following is a detailed introduction to the method for constructing a neural network and the image processing method in accordance with the embodiment of the present application in conjunction with the accompanying drawings.

[0196] like Figure 6As shown, an embodiment of the present application provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local device 301 and the local device 302 are connected to the execution device 310 via a communication network.

[0197] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and the like. The execution device 310 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the neural network model update method of the embodiment of the present application.

[0198] Specifically, in one implementation, the execution device 110 may perform the following process:

[0199] Get training data;

[0200] The training data is input into the neural network model for feature extraction, and the first prediction label is obtained based on the extracted features;

[0201] Determine a function value of a first loss function corresponding to the first predicted label according to the first predicted label and the true label of the training data, where the function value of the first loss function corresponding to the first predicted label is used to indicate a difference between the first predicted label and the true label of the training data;

[0202] Calculating a first weight gradient of the neural network model according to a function value of a first loss function corresponding to the first predicted label;

[0203] Obtaining a candidate weight parameter, wherein the partial derivative of a function value of a target loss function with respect to the candidate weight parameter is 0, wherein the function value of the target loss function is determined according to the function value of a second loss function corresponding to the first prediction label, and the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, where the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model;

[0204] Update the parameters of the neural network model based on the candidate weight parameters.

[0205] By executing the above process, the device 110 can obtain a target neural network model, which can be used for image classification or image processing, etc.

[0206] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 310. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0207] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0208] In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the target neural network from the execution device 310, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network to perform image classification or image processing, etc.

[0209] In another implementation, the target neural network can be directly deployed on the execution device 310. The execution device 310 obtains the image to be processed from the local device 301 and the local device 302, and uses the target neural network model to classify or perform other types of image processing on the image to be processed.

[0210] The above-mentioned execution device 310 can also be a cloud device. In this case, the execution device 310 can be deployed in the cloud; or, the above-mentioned execution device 310 can also be a terminal device. In this case, the execution device 310 can be deployed on the user terminal side. The embodiment of the present application does not limit this.

[0211] In order to increase the training speed and inference accuracy of the neural network model, a BN layer is usually added to the neural network model.

[0212] Batch Normalization (BN) is a data (feature) processing method used in deep neural networks. Adding a BN layer to a deep neural network can effectively reduce fluctuations during training and accelerate convergence. It also mitigates the deep neural network's sensitivity to noise, improving its robustness.

[0213] Specifically, BN is performed by subtracting the mean of the input batch data (features), dividing by its variance, and then linearly adjusting the result. The operations in this process are mainly non-matrix operations, and the computational complexity depends on the number of input batch data (features). Typically, the number of input batch data (features) is over 100,000, resulting in high BN computational complexity.

[0214] The main computing core of the current mainstream neural network processing chips is matrix operations. BN cannot effectively utilize the matrix operation capabilities of existing chips, and its overhead accounts for a large proportion in the entire deep neural network. For example, in ResNet50, BN's overhead accounts for more than 25%, affecting the speed of model iterative training. This has caused BN to become a performance bottleneck in deep neural network training.

[0215] Therefore, how to improve the training efficiency of neural network models has become an urgent problem to be solved.

[0216] The embodiment of the present application proposes a training method for a neural network model, which can accelerate the training process and improve the training efficiency of the neural network model.

[0217] Figure 7 A schematic diagram of a training device 700 for a neural network model according to an embodiment of the present application is shown. Figure 7 A brief description of the functions of each module in the

[0218] The training device 700 includes a neural network model 710 , an optimization target module 720 , a dynamic stabilization module 730 and an adjustment module 740 .

[0219] The solution of the embodiment of the present application can be used to train neural network models of various structures. For example, the neural network model 710 can be a residual network or a convolutional neural network.

[0220] Optionally, the neural network model 710 may include a first feature normalization layer, which is located before the output layer of the neural network model and is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0221] For example, the neural network model 710 may be located at Figure 5 In the NPU 50, the NPU 50 performs the operations in the neural network model 710. For example, the CPU assigns a task to retrieve training data, such as image data, from the external memory to the memory 501 or the unified memory 506 for loading by the NPU 50 to perform the operations in the neural network model 710.

[0222] The first optimization target module 720 is used to train the neural network model 710. The first optimization target module 720 provides an optimization target for the neural network model 710 and updates the parameters of the neural network model 710 based on the optimization target until the training of the neural network model 710 is completed. The optimization target can be a loss function or an objective function. The optimization target module 720 can also be referred to as an optimization target layer.

[0223] Specifically, the first optimization target module 720 calculates the first weight gradient of the neural network model based on the first prediction label output by the neural network model, and then determines the candidate weight parameters that minimize the function value of the target loss function, and updates the parameters of the neural network model 710 based on the candidate weight parameters until the training of the neural network model 710 is completed.

[0224] In one implementation, a function value of the target loss function is determined based on a function value of a second loss function corresponding to the first predicted label. The function value of the second loss function corresponding to the first predicted label is used to indicate a difference between the candidate weight parameter and the weight parameter of the neural network model, and a difference between a weight change and a first weight gradient, where the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

[0225] For example, the optimization target module 720 may be located at Figure 5 In the NPU50, the calculations in the optimization process of the neural network model are performed by NPU50.

[0226] The dynamic stable module (DSM) 730 includes a second optimization target module 731 . The second optimization target module 731 is used to train the neural network model 710 in collaboration with the first optimization target module 720 .

[0227] The dynamic stabilization module 730 is located after any layer in the neural network model 710 .

[0228] For example, the neural network model 710 may include a downsampling layer, and the dynamic stabilization module 730 may be located after the downsampling layer in the neural network model 710 .

[0229] The number of dynamic stabilization modules 730 can be one or more. For example, the neural network model 710 includes N downsampling layers. N is a positive integer. The number of dynamic stabilization modules 730 is M, and the M dynamic stabilization modules 730 are respectively located after M of the N downsampling layers. M is a positive integer less than or equal to N.

[0230] It should be noted that the above is merely illustrative, and the number and position of the dynamic stabilization modules 730 can be set as needed, and the embodiments of the present application do not limit this.

[0231] Specifically, the second optimization target module 731 obtains a second predicted label based on the features output by any layer in the neural network model and calculates a second weight gradient of the neural network model based on the second predicted label. The second optimization target module 731 works in conjunction with the first optimization target module 720 to determine a candidate weight parameter that minimizes the function value of the target loss function and updates the parameters of the neural network model 710 based on the candidate weight parameter until the training of the neural network model 710 is completed.

[0232] The synergistic effect of the second optimization target module 731 and the first optimization target module 720 means that the function value of the target loss function is jointly determined by these two modules. Specifically, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to the second prediction label, and the function value of the second loss function corresponding to the second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model, as well as the difference between the weight change and the second weight gradient.

[0233] Furthermore, the dynamic stabilization module 730 may further include a second feature normalization (FN) module 732 .

[0234] The second FN module 732 is used to perform feature normalization processing on the features input into the second FN module 732 .

[0235] Exemplarily, the second FN module 732 is located after the downsampling layer of the neural network model. That is, the second FN module 732 can perform feature normalization on the features output by the downsampling layer. In this case, the second optimization target module 731 obtains the second predicted label based on the features output by the downsampling layer in the neural network model. Alternatively, the second optimization target module 731 can obtain the second predicted label based on the normalized result.

[0236] It should be noted that there may be one or more dynamic stabilization modules 730, one or more second optimization target modules 731, and one or more second FN modules 732. Therefore, the function value of the second loss function corresponding to the second predicted label may be one or more. For example, the number of dynamic stabilization modules 730 may be equal to the number of downsampling layers in the neural network model. In other words, there may be a one-to-one correspondence between dynamic stabilization modules 730 and downsampling layers.

[0237] For example, the dynamic stabilization module 730 may be located at Figure 5 In the NPU50, the NPU50 executes the operations required by the dynamic stabilization module 730.

[0238] The adjustment module 740 is used to control the dynamic stabilization module 730. Specifically, the adjustment module 740 can enable part or all of the dynamic stabilization module 730 or disable part or all of the dynamic stabilization module 730.

[0239] The adjustment module 740 may send an enable signal or a disable signal to the dynamic stabilization module 730 according to the training status of the neural network model.

[0240] In one implementation, the adjustment module 740 can adjust the weight of the function value of the second loss function corresponding to the second predicted label in the function value of the target loss function based on the number of updates. For example, at the beginning of training, the adjustment module 740 can enable part or all of the dynamic stabilization module 730. As the number of updates increases, the weight is continuously reduced. When the weight is reduced to 0, it can be understood that the adjustment module 740 has disabled part or all of the dynamic stabilization module 730.

[0241] In the embodiment of the present application, the number of updates can also be understood as the number of iterations.

[0242] For example, the adjustment module 740 may be located at Figure 5 In the main CPU, the main CPU executes the instructions that the adjustment module 730 needs to execute, for example, sending an enable signal or a disable signal to the dynamic stabilization module 730.

[0243] In the inference stage, the trained neural network model 710 is deployed as needed, and the processed data is processed by the trained neural network model to obtain the processing result.

[0244] For example, the neural network model 710 can be used for image processing, and the inference stage is the image processing stage.

[0245] In the image processing stage, it is only necessary to deploy the trained neural network model 710, input the image to be processed into the trained neural network model 710, and process the image through the trained neural network model 710 to obtain the image processing result.

[0246] For a detailed description of the training phase, please refer to method 800 below. For a detailed description of the reasoning phase, please refer to method 900 below.

[0247] According to the method of the embodiment of the present application, by controlling the parameter fluctuation range during the training process, the controllability of the gradient fluctuation is guaranteed, the gradient fluctuation is avoided to cause training divergence and poor generalization, the training stability is improved, and the training process is accelerated; moreover, the first loss function is used to continuously improve the reasoning accuracy of the neural network model during the training process, and the second loss function can achieve a convergence trend consistent with the first loss function, thereby ensuring the reasoning accuracy of the neural network model. For example, the scheme of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be understood as the original loss function of the neural network model. The original loss function is used to make the predicted label output by the trained neural network model as similar as possible to the real label of the training data. The scheme of the embodiment of the present application can ensure the realization of the original prediction task, or in other words, while ensuring the original reasoning accuracy, improve the training stability and accelerate the training process.

[0248] The following combination Figures 8 to 11 The training method of the neural network model in the embodiment of the present application is described in detail.

[0249] Figure 8 A training method 800 for a neural network model provided in an embodiment of the present application is shown. Figure 8 The method shown can be executed by a training device for a neural network model, which can be a cloud service device or a terminal device, for example, a computer, a server, or other device with sufficient computing power to execute the training method for a neural network model, or a system composed of a cloud service device and a terminal device. Figure 2 Training equipment 120, Figure 5 Neural network processor 50 or Figure 6 The execution device 310 or local device executes.

[0250] For example, the method 800 may be specifically performed as follows: Figure 2 The training device 120 shown in FIG. 8 is executed, and the training data in the method 800 can be as follows: Figure 2 The training data maintained in the database 130 is shown.

[0251] The method 800 includes steps S810 to S860, which are described in detail below.

[0252] S810: Obtain training data.

[0253] The type of training data is related to the task of the neural network model. For example, if the neural network model is used for image processing, the training data can be images. Specifically, the neural network model can be used for tasks such as image classification, image detection, image segmentation, or image generation. For another example, if the neural network model is used for text recognition, the training data can be text. The embodiments of this application do not limit the type of training data.

[0254] For example, the training data can be stored in Figure 5 The external memory shown is accessed by the main CPU and then processed in the NPU 50 .

[0255] S820, input the training data into the neural network model for feature extraction, and obtain a first predicted label based on the extracted features. The type of label is related to the processing task of the neural network model. For example, if the neural network model is used for image classification, the label can be used to represent the category of the image. The true label of the training data is the true category to which the image belongs. The predicted label can be the predicted category of the image or the predicted probability of the category to which the image belongs. The predicted label is the output of the neural network model, and the true label of the training data is the target output of the neural network model. Exemplarily, the true label of the training data can be the result of manual pre-labeling.

[0256] Exemplarily, the neural network model may be the neural network model 710 in the device 700 .

[0257] For example, the neural network model can be Figure 3 or Figure 4 The convolutional neural network shown. For another example, the neural network model can be a residual network. The embodiment of the present application does not limit the type of neural network model.

[0258] Alternatively, the original neural network model includes a BN layer. In this case, method 800 further includes: deleting the BN layer in the original neural network model to obtain the neural network model. In other words, the neural network model trained in the embodiment of the present application is the neural network model after deleting the BN layer. It should be understood that in this case, the neural network model obtained after training does not include the deleted BN layer.

[0259] Figure 10 A schematic diagram of a neural network model in an embodiment of the present application is shown. Method 800 can be based on Figure 10 The neural network model shown is executed. conv represents the convolutional layer. Figure 10 The first FN layer in may be the first FN module of the device 700 . Figure 10 The first optimization target module in may be the first optimization target module 720 in the device 700 .

[0260] For example, the neural network model may be input by a user.

[0261] For example, user input such as Figure 10 The neural network model shown is used to obtain the trained neural network model.

[0262] For another example, the user inputs an original neural network model that includes a BN layer. In this case, method 800 further includes: deleting the BN layer in the original neural network model. After training is completed, the trained neural network model fed back to the user does not include the deleted BN layer.

[0263] Specifically, the training data is input into the neural network model for forward propagation to obtain the first predicted label.

[0264] For example, if the training data is an image and the neural network model is used for image classification, the output layer of the neural network model is a classification layer, such as a softmax layer, and the output layer outputs a first prediction label. The first prediction label can be used to indicate the predicted probability of the category to which the image belongs.

[0265] Optionally, the neural network model includes a first feature normalization layer. The first feature normalization layer is used to perform feature normalization processing on features input to the first feature normalization layer. The first feature normalization layer is located before the output layer of the neural network model.

[0266] The first FN layer performs feature normalization on the features input to the first FN layer, and inputs the processed results into the output layer of the neural network model, which outputs the first prediction label, for example, Figure 10 shown.

[0267] For example, the feature normalization process may be: dividing the input feature by the square root of all input features. Specifically, the result of the feature normalization process may satisfy the following formula:

[0268]

[0269] Among them, x i represents the i-th input feature, m represents the number of input features, Represents the result after feature normalization of the i-th input feature.

[0270] Figure 9 FIG1 shows a schematic diagram of a feature normalization process provided by an embodiment of the present application. Figure 9 As shown, the input features are a, b, c, and d, which are four eigenvalues. After feature normalization, the output results are

[0271] It should be understood that the above is merely illustrative, and other feature normalization processing methods may also be applied to method 800 . The embodiments of the present application do not limit the specific implementation of feature normalization processing.

[0272] S830: Determine a function value of a first loss function corresponding to the first predicted label according to the first predicted label and the true label of the training data.

[0273] The function value of the first loss function is used to indicate the difference between the predicted labels of the neural network model and the true labels of the training data.

[0274] The specific type of the first loss function is related to the task of the neural network model. For example, if the neural network model is used for image processing, the first loss function may be a cross entropy loss function. The embodiment of the present application does not limit the type of the first loss function.

[0275] Accordingly, the function value of the first loss function corresponding to the first predicted label is used to indicate the difference between the first predicted label and the true label of the training data.

[0276] For example, step S830 may be performed by the first target optimization module 720 in the apparatus 700. Alternatively, Figure 10 The first optimization target module in execution.

[0277] S840: Calculate a first weight gradient of the neural network model according to the function value of the first loss function corresponding to the first prediction label.

[0278] Specifically, the first weight gradient of the neural network model is calculated by back-propagating the function value of the first loss function corresponding to the first prediction label.

[0279] For example, step S840 may be performed by the first target optimization module 720 in the apparatus 700. Alternatively, Figure 10 The first optimization target module in execution.

[0280] S850: Obtain a candidate weight parameter, where the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0. That is, the candidate weight parameter is a weight parameter that minimizes the function value of the target loss function.

[0281] The function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label. In this case, the candidate weight parameter is a weight parameter that minimizes the function value of the second loss function corresponding to the first predicted label.

[0282] The function value of the second loss function is used to indicate the difference between the candidate weight parameters and the weight parameters in the current neural network model, and the difference between the weight change and the weight gradient value. The weight change refers to the difference between the candidate weight parameters and the weight parameters in the current neural network model.

[0283] Accordingly, the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters in the current neural network model and the difference between the weight change and the first weight gradient value.

[0284] Specifically, the function value of the second loss function is used to indicate the sum of the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change amount and the weight gradient.

[0285] Accordingly, the function value of the second loss function corresponding to the first prediction label is used to indicate the sum of the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient.

[0286] For example, the second loss function L new The following formula can be satisfied:

[0287]

[0288] That is to say, the candidate weight parameter w can satisfy the following formula:

[0289]

[0290] Among them, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model obtained after the kth iteration, that is, the weight parameters of the current neural network model. Represents the weight gradient value of the neural network model. For example, when calculating the function value of the second loss function corresponding to the first predicted label, is the first weight gradient. d1 represents a function for indicating the difference between the weight change and the weight gradient. The difference between the weight change and the weight gradient can be used to measure the direction of the weight gradient. Therefore, d1 can also be said to be a function for measuring the direction of the weight gradient. For example, d1 can be a distance metric function, that is, by the weight change ww k and The distance between them represents the difference between them. For example, this distance function can be used to calculate ww k and The square or absolute value of the difference between k and The square or absolute value of the difference between ww kand For another example, d1 can be a cosine similarity function. d2 represents a function for indicating the difference between the candidate weight parameters and the weight parameters in the current neural network model. The difference between the candidate weight parameters and the weight parameters in the current neural network model can be used to measure the magnitude of the weight gradient. Therefore, it can also be said that d2 is a function for measuring the magnitude of the weight gradient. Exemplarily, d2 can be a distance metric function, that is, by using w and w k The distance between them represents the difference between them. For example, this distance function can be used to calculate w and w k The square or absolute value of the difference between w and w k The square or absolute value of the difference between w and w k The distance between them. For another example, d2 can be a cosine similarity function. λ1 represents the weight corresponding to d1, and λ2 represents the weight corresponding to d2. The weights can be set as needed, for example, λ1 = 0.4, λ2 = 0.6.

[0291] It should be noted that the above d1 is only for illustration, and d1 can also be other metric functions, as long as the metric function can indicate the difference between the weight change and the weight gradient. The above d2 is only for illustration, and d2 can also be other metric functions, as long as the metric function can indicate the difference between the candidate weight parameters and the weight parameters in the current neural network model. The embodiments of the present application do not limit the specific methods used for d1 and d2.

[0292] Specifically, by making L new The partial derivative of w is 0, and the candidate weight parameter is obtained. The candidate weight parameter is the one that makes L new The weight parameter with the smallest function value.

[0293] For example, to calculate the above L new The weight parameter that makes the partial derivative of w equal to 0 is the candidate weight parameter that minimizes the function value of the target loss function.

[0294] The first loss function is used to make the first predicted label output by the trained neural network model as close as possible to the true label of the training data. Function d1 can make the update direction of the weight parameters in the neural network model (given by ww k The update direction indicated by ) is the same as the update direction indicated by the first loss function (indicated by The update direction indicated by the second loss function is as consistent as possible to ensure that the convergence during training is consistent with the first loss function and gradually converges, that is, the neural network model can be trained in the direction of reducing the function value of the first loss function, thereby ensuring the reasoning accuracy of the trained neural network model. In other words, the function d1 in the second loss function is compatible with the first loss function. The function d2 in the second loss function can control the jump amplitude between the updated weight parameters and the current weight parameters, and then control the fluctuation range of the weight parameters, avoiding large jumps in the updated weight parameters, improving the stability of training, accelerating the convergence process, and improving training efficiency.

[0295] For example, step S850 may be performed by the target optimization module 720 in the apparatus 700. Alternatively, Figure 10 The first optimization target module in execution.

[0296] S860: Update the parameters of the neural network model according to the candidate weight parameters.

[0297] Updating the parameters in the neural network model is training the neural network model.

[0298] Exemplarily, step S860 may be: using the candidate weight parameters as weight parameters in the updated neural network model.

[0299] Optionally, step S860 may be: updating the parameters of the neural network model according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

[0300] For example, the weight parameters in the updated neural network model can satisfy the following formula:

[0301] w k+1 =(ww k )*lr+w k ;

[0302] Among them, w k+1 It represents the weight parameter of the neural network model obtained after the k+1th iteration, that is, the weight parameter of the neural network model updated after this iteration, and lr represents the learning rate.

[0303] By setting the learning rate, the parameter update process can be smoothed, the parameter fluctuation range can be further controlled, the training stability can be further improved, the convergence process can be accelerated, and the training efficiency can be improved.

[0304] For example, step S860 may be performed by the target optimization module 720 in the apparatus 700. Alternatively, Figure 10 The first optimization target module in execution.

[0305] Repeat steps S810 to S860 until the training of the neural network model is completed.

[0306] Specifically, step S810 to step S860 are repeated until the training of the neural network model meets the training termination condition, that is, the training is completed, and a trained neural network model is obtained.

[0307] The training termination condition can be set as needed.

[0308] Exemplarily, the training of the neural network model satisfies the training termination condition, including: the reasoning accuracy of the neural network model satisfies a preset condition.

[0309] For example, the inference accuracy of the neural network model is greater than or equal to the first threshold. That is, when the neural network model tends to converge, the training is terminated.

[0310] For another example, if the change in the inference accuracy of the neural network model after P consecutive updates is less than or equal to the second threshold, the training is terminated when the inference accuracy stabilizes or no longer increases significantly.

[0311] It should be understood that the above training termination conditions of the neural network model are only illustrative and the present application embodiment does not limit the training termination conditions. For example, the training termination conditions of the neural network model may also include: the number of iterations is greater than or equal to a preset number.

[0312] In one implementation, the solution of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be the original loss function of the neural network model. As mentioned above, the first target optimization module 720 can be understood as the target optimization layer after the output layer of the neural network module. During the training process of the original neural network model, the original target optimization layer updates the weight parameters of the neural network model according to the function value of the first loss function. During the training process of method 800, the first target optimization module 720 updates the weight parameters of the neural network model according to the function value of the target loss function. In other words, the target optimization layer in the embodiment of the present application can be used to replace the target optimization layer in the training process of the original neural network model without changing the structure of the neural network model, and the solution of the embodiment of the present application can be used for training. If the original neural network model includes a BN layer, the BN layer can be deleted, and then the target optimization layer in the embodiment of the present application can be used to replace the target optimization layer in the training process of the original neural network model, and the solution of the embodiment of the present application can be used for training. The neural network model obtained after training does not include the deleted BN layer.

[0313] In an embodiment of the present application, the difference between the candidate weight parameters and the weight parameters of the neural network model is controlled by the second loss function, thereby controlling the parameter fluctuation range during the training process, ensuring the controllability of the gradient fluctuation, avoiding the gradient fluctuation leading to training divergence and poor generalization, improving the training stability, accelerating the training process, and improving the training efficiency; moreover, the first loss function is used to continuously improve the reasoning accuracy of the neural network model during the training process, and the second loss function can achieve a convergence trend consistent with the first loss function, thereby improving the reasoning accuracy of the neural network model. For example, the scheme of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be understood as the original loss function of the neural network model. The original loss function is used to make the predicted label output by the trained neural network model as similar as possible to the true label of the training data. The scheme of the embodiment of the present application can ensure that the original prediction task is achieved, and by increasing the constraints on the gradient direction and size, the training stability is improved, the training process is accelerated, and it is universal for a variety of deep neural network models.

[0314] In addition, the training overhead of the neural network model using the first loss function is almost the same as that of the neural network model using the target loss function, that is, compared with the neural network model using the first loss function, the solution of the embodiment of the present application can improve the training speed and reasoning accuracy of the neural network model without increasing the overhead. Existing neural network models usually use the first loss function. The solution of the embodiment of the present application only sets a new optimization target, namely the target loss function, on the basis of the existing neural network model, which is equivalent to only adding the calculation process of the candidate weight parameters, and almost no increase in overhead. Further, compared with the neural network model using modules such as BN, the solution of the embodiment of the present application can achieve the same convergence speed and reasoning accuracy as the neural network model using modules such as BN, and the solution of the embodiment of the present application can effectively utilize the computing power of the processor, and the required training overhead is much less than the training overhead required for the neural network model using modules such as BN. On the one hand, the solution of the embodiment of the present application greatly saves training resources and saves training costs. On the other hand, the same training resources can support the training of more neural network models.

[0315] In addition, the solution of the embodiment of the present application can be completed on the basis of the existing neural network, that is, a new optimization target is set based on the existing neural network model, and the original optimization target can be replaced without the user's perception, that is, without the user making any modifications, thereby automatically accelerating the training of the neural network model, improving training efficiency, and improving user experience.

[0316] Optionally, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to at least one second predicted label. The function value of the second loss function corresponding to the at least one second predicted label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient.

[0317] In this case, Figure 11 A schematic diagram of a neural network model in an embodiment of the present application is shown. Method 800 can be based on Figure 11 The neural network model shown is executed. Figure 11 The DSM in may be the DSM730 in the device 700 . Figure 11 The DSM in is located after the downsampling layer in the neural network model. It should be noted that Figure 11 This is for illustration only, and the DSM may also be located at other positions in the neural network model, which is not limited in the embodiments of the present application.

[0318] Specifically, the second weight gradient can be obtained through steps S1 to S3.

[0319] For example, steps S1 to S3 can be performed by Figure 11 The dynamic stabilization module in the embodiment is executed. The dynamic stabilization module is described in detail with reference to the dynamic stabilization module 730 in the aforementioned apparatus 700.

[0320] It should be understood that Figure 11 The number of DSMs in is only for illustration, and more or fewer dynamic stabilization modules can be set during the training process.

[0321] Steps S1 to S3 are described below.

[0322] S1. Obtain at least one second prediction label based on features output by at least one layer in the neural network model.

[0323] The at least one layer may be any one or more layers in the neural network model.

[0324] Exemplarily, the neural network model may include at least one downsampling layer, and the at least one layer may be at least one downsampling layer. Step S1 may include: obtaining at least one second predicted label based on features output by the at least one downsampling layer in the neural network model.

[0325] A downsampling layer is a layer that makes the resolution of the output feature map smaller than the resolution of the input feature map.

[0326] If the neural network model includes N downsampling layers, then the neural network model includes N locations at which the resolution is reduced. In step S1, M second predicted labels can be obtained based on the features output by M of the N downsampling layers. N is a positive integer. M is a positive integer less than or equal to N.

[0327] For example, Figure 11 The number of dynamic stabilization modules in the DSM may be M, and each DSM is placed after a downsampling layer. The second optimization target module in the DSM may be used to execute step S1.

[0328] Specifically, obtaining at least one second prediction label based on features output by at least one layer in the neural network model can be as follows: inputting the features output by the at least one layer into a corresponding output layer to obtain at least one second prediction label.

[0329] For example, obtaining M second prediction labels according to the features output by M downsampling layers may be as follows: inputting the features output by the M downsampling layers into M output layers respectively to obtain M second prediction labels.

[0330] The output layer described above has the same structure as the original output layer of the neural network model. The structure of the output layer is related to the task of the neural network model. For example, when the neural network model is used for image classification, the output layer can also be called the classification layer. Specifically, the classification layer can be a softmax layer, which can output the predicted classification label of the image.

[0331] Furthermore, M second prediction labels are obtained according to the features output by the M downsampling layers. This can be done by inputting the features of the M downsampling outputs into M global pooling layers, M fully connected layers, and M output layers in sequence to obtain M second prediction labels.

[0332] Optionally, the neural network model includes at least one second feature normalization layer. The at least one second feature normalization layer is respectively located after the at least one downsampling layer, and the at least one second feature normalization layer is used to perform feature normalization processing on features input into the at least one second feature normalization layer.

[0333] In this case, step S1 may include: inputting the features output by at least one downsampling layer in the neural network model into at least one second FN layer for feature normalization processing, and obtaining at least one second predicted label according to the processed results. For example, the second FN layer can be regarded as Figure 11 The second FN layer in the DSM.

[0334] Specifically, M second prediction labels are obtained according to the features output by the M downsampling layers: the features of the M downsampling outputs are respectively input into the M second FN layers for feature normalization processing, and the processed results are respectively input into the M output layers to obtain M second prediction labels.

[0335] The feature normalization process used by the second FN layer may be the same as that used by the first FN layer.

[0336] For example, the feature normalization process may be: dividing the input feature by the square root of all input features. Specifically, the result of the feature normalization process may satisfy the following formula:

[0337]

[0338] Among them, x i represents the i-th input feature, m represents the number of input features, Represents the result after feature normalization of the i-th input feature.

[0339] Furthermore, M second prediction labels are obtained according to the features output by the M downsampling layers. The features of the M downsampling outputs are respectively input into M global pooling layers, M fully connected layers, M second FN layers and M output layers in sequence to obtain M second prediction labels.

[0340] S2. Determine a function of a first loss function corresponding to the at least one second predicted label according to the at least one second predicted label and the true label of the training data.

[0341] The function value of the first loss function corresponding to the at least one second predicted label is used to indicate the difference between the at least one second predicted label and the true label of the training data.

[0342] For example, the first loss function may be a cross entropy loss function. A detailed description of the first loss function is provided in step S830 and will not be repeated here.

[0343] For example, the function values ​​of the first loss function corresponding to the M second prediction labels can be determined based on the M second prediction labels.

[0344] For example, the number of dynamic stabilization modules is M, such as Figure 11 As shown, each DSM is placed after a downsampling layer, and the second optimization target module in the DSM can be used to perform step S2.

[0345] S3. Calculate at least one second weight gradient of the neural network model according to the function value of the first loss function corresponding to the at least one second prediction label.

[0346] Specifically, at least one second weight gradient is calculated by back-propagating a function value of a first loss function corresponding to at least one second prediction label.

[0347] For example, M second weight gradients may be calculated based on the function values ​​of the first loss function corresponding to the M second prediction labels.

[0348] For example, the number of dynamic stabilization modules is M, such as Figure 11 As shown, each DSM is placed after a downsampling layer, and the second optimization target module in the DSM can be used to perform step S3.

[0349] As described above, the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the second weight gradient.

[0350] The specific description of the second loss function can be found in the aforementioned step S850, which will not be repeated here.

[0351] The function value of the target loss function can be determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label. Figure 11 As shown, the second optimization target module and the first optimization target module jointly train the neural network model.

[0352] Exemplarily, the function value of the target loss function is the sum of the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label.

[0353] For example, the function value of the objective loss function can satisfy the following formula.

[0354] L all =L new_n +L new ;

[0355] Among them, L all Represents the function value of the target loss function, L new_n Represents the function value of the second loss function corresponding to the second predicted label. If the second predicted label is obtained based on the features output by the downsampling layer, L new_n It can also be understood as the function value of the second loss function corresponding to the nth downsampling layer.

[0356] For example, the function value L of the objective loss function is all The following formula can be satisfied.

[0357] L all =a n *L new_n+L new ;

[0358] Among them, a n Indicates L new_n If the second predicted label is obtained based on the features output by the downsampling layer, a n It can also be understood as the weight of the function value of the second loss function corresponding to the nth downsampling layer. For example, a n It can be a pre-set fixed value. Alternatively, a n It can be a variable value, for example, it changes with the number of iterations. For detailed description, please refer to Method 1, Method 2, Method 3, Method 4 and Method 5 below.

[0359] When training with the initial weights of a neural network model, the model is prone to instability in the early stages of training. According to the solution of the embodiments of the present application, a second predicted label is obtained based on the features output by at least one layer of the neural network, and a second weight gradient is calculated based on the second predicted label as an auxiliary reference for updating the parameters of the neural network model. This can reduce gradient fluctuations and improve training stability.

[0360] Furthermore, due to the reduced resolution of the feature maps after the downsampling layer, significant information is lost. Analysis revealed that the gradients of the downsampling layer fluctuate significantly, leading to unstable training. According to the solution of the embodiment of the present application, a second predicted label is obtained based on the features output by the downsampling layer, and a second weight gradient is calculated based on the second predicted label as an auxiliary reference for updating the parameters of the neural network model. This can further reduce gradient fluctuations and improve training stability.

[0361] In addition, only the second prediction label obtained by the features output by the downsampling layer is used for auxiliary training, which increases the amount of computation less and does not occupy a large amount of training resources. In other words, with almost no increase in overhead, the gradient fluctuation is further reduced and the stability of training is improved.

[0362] In addition, through feature normalization processing, the forward fluctuation is controlled within a certain range, making the entire training process more stable and further accelerating the training process. In particular, compared with the neural network model using modules such as BN, the neural network model using modules such as BN usually requires a large number of BN modules, which takes up a lot of training overhead. The solution of the embodiment of the present application can achieve the same convergence speed and reasoning accuracy as the neural network model using modules such as BN, and the solution of the embodiment of the present application only sets the FN layer before the downsampling layer and the last output layer of the neural network model. The required FN layer is relatively small, and the training overhead is much less than the training overhead required for the neural network model using modules such as BN.

[0363] Optionally, at the beginning of training, the loss function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to at least one second predicted label.

[0364] In the later stage of training, the loss function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label.

[0365] That is, in the early stage of training, steps S810 to S860 and steps S1 to S3 are executed, and the second prediction label is used to assist the training. In the later stage of training, only steps S810 to S860 are executed, and the training is completed only by the first prediction label.

[0366] For example, in the early stages of training, Figure 11 The neural network model shown in the figure is trained. In the later stage of training, the Figure 10 The neural network model shown completes training.

[0367] In this case, in the formula of the above objective loss function, at the initial stage of training a n Greater than 0, in the later stages of training, a n is 0.

[0368] In the embodiment of the present application, the training method can be varied. For example, the method 800 is executed by the apparatus 700. At the beginning of the training, the adjustment module 740 sends an enable signal to enable the DSM 730, for example, based on Figure 11 The neural network model is trained; in the later stage of training, the adjustment module 740 sends a disable signal to disable the DSM730, that is, based on Figure 10 The neural network model shown is trained.

[0369] The training methods at different stages can be set as needed. The following five methods (method 1, method 2, method 3, method 4 and method 5) are used as examples to illustrate the training methods at different stages.

[0370] Mode 1: The weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates.

[0371] That is to say, the proportion of the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function continues to decrease as the training process progresses.

[0372] For example, the weight value may vary with the number of iterations, i.e., the weight value may be reduced after completing an iteration. For another example, the weight value may vary with the number of epochs, i.e., the weight value may be reduced after completing an epoch. The embodiments of the present application do not limit the frequency of weight changes, as long as the weight is negatively correlated with the number of updates.

[0373] As shown above, the function value of the objective loss function L all The following formula can be satisfied.

[0374] L all =a n *L new_n +L new ;

[0375] In this case, a n It decreases as the number of updates increases. n is reduced to 0, then the function value of the target loss function is determined only by the function value of the second loss function corresponding to the first predicted label. In other words, the initial stage of training refers to a n In the stage where the value is not 0, steps S1 to S3 are executed.

[0376] Method 2: When the number of updates is less than or equal to the third threshold, the loss function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label.

[0377] Furthermore, when the number of updates is greater than a third threshold, the loss function value of the target loss function is determined according to the function value of the second loss function corresponding to the first prediction label.

[0378] That is, the initial stage of training refers to a stage in which the number of updates is less than or equal to the third threshold, and steps S1 to S3 are performed in this stage.

[0379] In the embodiment of the present application, the number of updates can also be understood as the number of iterations.

[0380] In the formula of the above objective loss function, when the number of iterations is greater than the third threshold, a n =0, when the number of iterations is less than or equal to the third threshold, a n Greater than 0.

[0381] Exemplarily, when the number of iterations is less than or equal to the third threshold, a n is a fixed value, or, n Negatively correlated with the number of updates.

[0382] That is, the DSM module in the apparatus 700 participates in training only when the number of iterations is less than or equal to the third threshold.

[0383] Method 3: When the number of epochs is less than or equal to T, the loss function value of the target loss function is determined based on the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to at least one second predicted label. T is a positive integer.

[0384] Furthermore, when the number of epochs is greater than T, the loss function value of the target loss function is determined according to the function value of the second loss function corresponding to the first prediction label.

[0385] 1 epoch means training all training samples in the training set once. In the formula of the above objective loss function, when the number of epochs is less than or equal to T, a n Greater than 0, when the number of epochs is greater than T, a n =0.

[0386] For example, when the number of epochs is less than or equal to T, a n is a fixed value, or, n Negatively correlated with the number of updates.

[0387] Typically, the entire training cycle lasts for dozens of epochs, with the second predicted label assisting the training process only in the first T epochs. T is usually a small value, such as 5-10 epochs. This can further reduce the amount of computation and lower the overhead.

[0388] That is, all DSMs in the apparatus 700 participate in training only in the first T epochs (eg, 5-10 epochs).

[0389] For example, there are N downsampling layers in a neural network model, and a DSM can be connected after the N downsampling layers.

[0390] like Figure 11 As shown, from the start of training to the Tth epoch, the N DSMs and the first optimization target module synchronously train the neural network model; Figure 10 As shown, from the Tth epoch to the end of training, the neural network model is trained by the first optimization target module.

[0391] Mode 4: When the change in the function value of the target loss function obtained after Q consecutive iterations is less than the fourth threshold, the function value of the target loss function is determined according to the second loss function value corresponding to the first predicted label. Q is a positive integer.

[0392] Further, when the change value of the function value of the target loss function obtained after Q consecutive iterations is greater than or equal to the fourth threshold, the loss function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label.

[0393] That is, when the change in the function value of the target loss function obtained after Q consecutive iterations is greater than or equal to the fourth threshold, steps S1 to S3 are executed, and the training is assisted by the second prediction label. When the change in the function value of the target loss function obtained after Q consecutive iterations is less than the fourth threshold, the neural network model is trained only according to the function value of the second loss function corresponding to the first prediction label.

[0394] In the formula of the above target loss function, when the change value of the target loss function obtained after Q consecutive iterations is less than the fourth threshold, a n = 0, when the change in the function value of the target loss function obtained after Q consecutive iterations is greater than or equal to the fourth threshold, a n Greater than 0.

[0395] Exemplarily, when the change in the function value of the target loss function obtained after Q consecutive iterations is greater than or equal to the fourth threshold, a n is a fixed value, or, n Negatively correlated with the number of updates.

[0396] Exemplarily, the DSM module in the apparatus 700 participates in training only when the change in the function value of the target loss function obtained after Q consecutive iterations is greater than or equal to the fourth threshold.

[0397] Method 5: When the change value of the function value of the second loss function corresponding to the first prediction label obtained after Z consecutive iterations is less than the fifth threshold, the loss function value of the target loss function is determined according to the function value of the second loss function corresponding to the first prediction label.

[0398] Furthermore, when the change value of the function value of the second loss function corresponding to the first prediction label obtained after Z consecutive iterations is greater than or equal to the fifth threshold, the loss function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label.

[0399] That is, when the change in the function value of the second loss function corresponding to the first prediction label obtained after Z consecutive iterations is greater than or equal to the fifth threshold, steps S1 to S3 are executed, and the training is assisted by the second prediction label. When the change in the function value of the second loss function corresponding to the first prediction label obtained after Z consecutive iterations is less than the fifth threshold, the neural network model is trained only based on the function value of the second loss function corresponding to the first prediction label.

[0400] In the formula of the above target loss function, when the change value of the function value of the second loss function corresponding to the first predicted label obtained after Z consecutive iterations is less than the fifth threshold, a n = 0, when the change in the function value of the second loss function corresponding to the first predicted label obtained after Z consecutive iterations is greater than or equal to the fifth threshold, a n Greater than 0.

[0401] Exemplarily, when the change value of the function value of the second loss function corresponding to the first prediction label obtained after Z consecutive iterations is greater than or equal to the fifth threshold, a n is a fixed value, or, a n Negatively correlated with the number of updates.

[0402] Exemplarily, the DSM module in the apparatus 700 participates in training only when the change in the function value of the second loss function corresponding to the first predicted label obtained after Z consecutive iterations is greater than or equal to the fifth threshold.

[0403] Method 4 and Method 5 can be understood as that when the fluctuation value of the function value of the loss function is small, it can be considered that the training has entered the late stage, and the training is completed only by the function value of the second loss function corresponding to the first prediction label.

[0404] It should be understood that the above division method is only for illustration. In actual application, the early training stage and the late training stage can be divided as needed, or in other words, the DSM can be enabled or disabled according to the training process.

[0405] Furthermore, the above example only uses enabling or disabling all DSMs as an example. In actual applications, you can enable or disable some DSMs as needed. In other words, you can use the features output by some downsampling layers to obtain the second predicted label to assist in training. For example, as the number of iterations increases, the number of DSMs disabled gradually increases until all DSMs are disabled.

[0406] In the early stages of training, the function value of the loss function usually fluctuates greatly, that is, the function value of the second loss function corresponding to the first prediction label varies greatly. In this case, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, that is, the training process is assisted by the second prediction label to improve the stability of the neural network model in the early stages of training. In the later stages of training, the function value of the loss function tends to be stable, that is, the function value of the second loss function corresponding to the first prediction label varies less, and the neural network model can be trained only based on the function value of the second loss function corresponding to the first prediction label.

[0407] According to the solution of the embodiment of the present application, the training process is assisted by the second prediction label only in the early stage of training, which can increase the stability of the neural network model in the early stage of training, while further reducing the amount of calculation and reducing the training overhead. For example, when the function value of the target loss function fluctuates greatly, the DSM 730 in the device 700 is enabled. When the function value of the target loss function fluctuates less, or when the function value of the target loss function is relatively stable, the DSM 730 in the device 700 is disabled. The DSM is only used in the early stage of training and is disconnected in the later stage of training. In this way, while ensuring the stability of training, the training overhead is almost not increased.

[0408] In addition, according to the solution of the embodiment of the present application, the training method is adaptively adjusted. For example, in the aforementioned methods 3 and 4, the training method is adjusted according to the changes in the function value of the loss function during the training process, and the computing power of the processor is fully utilized. This can improve the training stability of the neural network model with almost no increase in implementation difficulty and training overhead.

[0409] In one implementation, the solution of the embodiment of the present application can be used to optimize the training process of the original neural network model, and the first loss function can be the original loss function of the neural network model. As mentioned above, the target optimization module 720 can be understood as the target optimization layer after the output layer of the neural network module. During the training process of the original neural network model, the target optimization layer updates the weight parameters of the neural network model according to the function value of the first loss function. During the training process of method 800, the target optimization layer updates the weight parameters of the neural network model according to the function value of the target loss function. If the original neural network model includes a BN layer, the BN layer can be deleted. Furthermore, a DSM module is added after the downsampling layer of the original neural network model to assist in the training of the neural network model. That is to say, without changing the structure of the neural network model, the target optimization layer in the embodiment of the present application can be used to replace the target optimization layer in the training process of the original neural network model, and DSM can be added, and DSM can be enabled or disabled during the training process to adjust the training method, and the solution of the embodiment of the present application can be used for training.

[0410] Figure 12 The schematic flow chart of the image processing method 1200 provided in an embodiment of the present application is shown. The method can be executed by an apparatus or device capable of image processing. For example, the apparatus can be a cloud service device or a terminal device, such as a computer or server with sufficient computing power to execute the image processing method, or a system composed of a cloud service device and a terminal device. For example, the method 1200 can be executed by Figure 2 The execution device 110 in Figure 5 Neural network processor 50 or Figure 6 The execution device 310 or local device executes.

[0411] For example, the method 1200 may be specifically performed as follows: Figure 2 The execution device 110 shown in FIG. 1 is executed, and the image to be processed in the method 1200 may be as follows: Figure 2 Input data is provided by client device 140 as shown.

[0412] Figure 12 The target neural network model used in the image processing method 1200 can be the above Figure 8 Method 1200 includes steps S1210 to S1220. The specific implementation of method 1200 can refer to the aforementioned method 800. To avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 1200.

[0413] S1210: Obtain an image to be processed.

[0414] Among them, the image to be processed can be an image captured by a terminal device (or other devices or equipment such as a computer or server) through a camera, or the image to be processed can also be an image obtained from inside the terminal device (or other devices or equipment such as a computer or server) (for example, an image stored in the photo album of the terminal device, or an image obtained by the terminal device from the cloud), and the embodiments of the present application are not limited to this.

[0415] S1220, using the target neural network model to extract features of the image to be processed, and obtaining a processing result of the image to be processed based on the features extracted by the target neural network model.

[0416] Among them, the target neural network model is obtained by inputting the training data into the neural network model for feature extraction, obtaining the first prediction label according to the features extracted by the neural network model, determining the function value of the first loss function corresponding to the first prediction label according to the first prediction label and the true label of the training data, the function value of the first loss function corresponding to the first prediction label is used to indicate the difference between the first prediction label and the true label of the training data, calculating the first weight gradient of the neural network model according to the function value of the first loss function corresponding to the first prediction label, obtaining the candidate weight parameter, the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0, and updating the parameters of the neural network model according to the candidate weight parameter, wherein the function value of the target loss function is determined according to the function value of the second loss function corresponding to the first prediction label, the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

[0417] Optionally, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient, the at least one second prediction label is obtained based on the features output by at least one downsampling layer in the neural network model, the function value of the second loss function corresponding to at least one second prediction label is determined based on at least one second prediction label and the true label of the training data, the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between at least one second prediction label and the true label of the training data, and the at least one second weight gradient is calculated based on the function value of the first loss function corresponding to at least one second prediction label.

[0418] Optionally, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, including: the weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

[0419] Optionally, the target neural network model includes a first feature normalization layer, which is located before the output layer of the target neural network model, and is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0420] Optionally, the neural network model includes at least one second feature normalization layer, and the at least one second feature normalization layer is located after at least one downsampling layer. The at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer. The at least one second prediction label is obtained by inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and the processed result is obtained.

[0421] Optionally, the second loss function satisfies the following formula:

[0422]

[0423] Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function indicating the difference between the weight change amount and the weight gradient, d2 represents a function indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

[0424] Optionally, updating the parameters of the neural network model according to the candidate weight parameters includes: updating the parameters of the neural network model according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

[0425] Optionally, the weight parameters of the updated neural network model satisfy the following formula:

[0426] w k+1 =(ww k )*lr+w k ;

[0427] Among them, w k+1 Represents the weight parameters of the updated neural network model, wk Represents the weight parameters of the neural network model, w represents the candidate weight parameters, and lr represents the learning rate.

[0428] The following combination Figures 13 to 16 The apparatus of the embodiment of the present application is described. It should be understood that the apparatus described below can execute the method of the embodiment of the present application, and in order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the apparatus of the embodiment of the present application.

[0429] Figure 13 It is a schematic block diagram of a training device for a recommendation model according to an embodiment of the present application. Figure 13 The training device 3000 for the recommendation model shown includes an acquisition unit 3010 and a processing unit 3020 .

[0430] The acquisition unit 3010 and the processing unit 3020 can be used to execute the training method of the recommendation model of the embodiment of the present application, specifically, can be used to execute method 800.

[0431] The acquisition unit 3010 is used to acquire training data.

[0432] The processing unit 3020 is used to input the training data into the neural network model for feature extraction, and obtain a first prediction label based on the extracted features; determine the function value of the first loss function corresponding to the first prediction label based on the first prediction label and the true label of the training data, and the function value of the first loss function corresponding to the first prediction label is used to indicate the difference between the first prediction label and the true label of the training data; calculate the first weight gradient of the neural network model based on the function value of the first loss function corresponding to the first prediction label; obtain candidate weight parameters, and the partial derivative of the function value of the target loss function with respect to the candidate weight parameters is 0, wherein the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label, and the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameters and the weight parameters of the neural network model; update the parameters of the neural network model based on the candidate weight parameters.

[0433] Optionally, as an embodiment, the processing unit 3020 is further used to: obtain at least one second prediction label based on the features output by at least one downsampling layer in the neural network model; determine the function value of the first loss function corresponding to at least one second prediction label based on the at least one second prediction label and the true label of the training data, and the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between the at least one second prediction label and the true label of the training data; calculate at least one second weight gradient of the neural network model based on the function value of the first loss function corresponding to the at least one second prediction label; and the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, and the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient.

[0434] Optionally, as an embodiment, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, including: the weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

[0435] Optionally, as an embodiment, the neural network model includes a first feature normalization layer, which is located before the output layer of the neural network model, and is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0436] Optionally, as an embodiment, the neural network model includes at least one second feature normalization layer, the at least one second feature normalization layer is respectively located after at least one downsampling layer, the at least one second feature normalization layer is used to perform feature normalization processing on the features input into at least one second feature normalization layer, and the processing unit 3020 is specifically used to: input the features output by at least one downsampling layer in the neural network model into at least one second feature normalization layer for feature normalization processing, and obtain at least one second prediction label based on the processed results.

[0437] Optionally, as an embodiment, the second loss function satisfies the following formula:

[0438]

[0439] Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function indicating the difference between the weight change amount and the weight gradient, d2 represents a function indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

[0440] Optionally, as an embodiment, the processing unit 3020 is specifically used to: update the parameters of the neural network model according to the candidate weight parameters and the learning rate, wherein the learning rate is used to indicate the magnitude of the change of the weight parameters of the neural network model.

[0441] Optionally, as an embodiment, the weight parameters of the updated neural network model satisfy the following formula:

[0442] w k+1 =(ww k )*lr+w k ;

[0443] Among them, w k+1 Represents the weight parameters of the updated neural network model, w k Represents the weight parameters of the neural network model, w represents the candidate weight parameters, and lr represents the learning rate.

[0444] Figure 14 It is a schematic block diagram of the image processing device 4000 provided in an embodiment of the present application. Figure 14 The image processing device 4000 shown includes an acquisition unit 4010 and a processing unit 4020 .

[0445] The acquisition unit 4010 and the processing unit 4020 can be used to execute the image processing method of the embodiment of the present application, for example, can be used to execute method 1200.

[0446] The acquisition unit 4010 is used to acquire the image to be processed.

[0447] The processing unit 4020 is used to extract features of the image to be processed using the target neural network model, and obtain a processing result of the image to be processed based on the features extracted by the target neural network model.

[0448] Among them, the target neural network model is obtained by inputting the training data into the neural network model for feature extraction, obtaining the first prediction label according to the features extracted by the neural network model, determining the function value of the first loss function corresponding to the first prediction label according to the first prediction label and the true label of the training data, the function value of the first loss function corresponding to the first prediction label is used to indicate the difference between the first prediction label and the true label of the training data, calculating the first weight gradient of the neural network model according to the function value of the first loss function corresponding to the first prediction label, obtaining the candidate weight parameter, the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0, and updating the parameters of the neural network model according to the candidate weight parameter, wherein the function value of the target loss function is determined according to the function value of the second loss function corresponding to the first prediction label, the function value of the second loss function corresponding to the first prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

[0449] Optionally, as an embodiment, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, the function value of the second loss function corresponding to at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient, at least one second prediction label is obtained based on the features output by at least one downsampling layer in the neural network model, the function value of the second loss function corresponding to at least one second prediction label is determined based on at least one second prediction label and the true label of the training data, the function value of the first loss function corresponding to at least one second prediction label is used to indicate the difference between at least one second prediction label and the true label of the training data, and at least one second weight gradient is calculated based on the function value of the first loss function corresponding to at least one second prediction label.

[0450] Optionally, as an embodiment, the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label, including: the weight corresponding to the function value of the second loss function corresponding to at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

[0451] Optionally, as an embodiment, the target neural network model includes a first feature normalization layer, which is located before the output layer of the target neural network model, and the first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

[0452] Optionally, as an embodiment, the neural network model includes at least one second feature normalization layer, and the at least one second feature normalization layer is located after at least one downsampling layer. The at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer. The at least one second prediction label is obtained by inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and the processed result is obtained.

[0453] Optionally, as an embodiment, the second loss function satisfies the following formula:

[0454]

[0455] Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function indicating the difference between the weight change amount and the weight gradient, d2 represents a function indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

[0456] Optionally, as an embodiment, updating the parameters of the neural network model according to the candidate weight parameters includes: updating the parameters of the neural network model according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

[0457] Optionally, as an embodiment, the weight parameters of the updated neural network model satisfy the following formula:

[0458] w k+1 =(ww k )*lr+w k ;

[0459] Among them, w k+1 Represents the weight parameters of the updated neural network model, w k Represents the weight parameters of the neural network model, w represents the candidate weight parameters, and lr represents the learning rate.

[0460] It should be noted that the training apparatus 3000 and the apparatus 4000 are implemented in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0461] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0462] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0463] Figure 15 This is a schematic diagram of the hardware structure of the training device for the neural network model provided in an embodiment of the present application. Figure 15 The neural network model training apparatus 5000 shown (the apparatus 5000 may be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, the processor 5002, and the communication interface 5003 are connected to each other via the bus 5004.

[0464] The memory 5001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 may store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is used to execute the various steps of the training method of the neural network model of the embodiment of the present application. Specifically, the processor 5002 may execute the above Figure 8 Steps S810 to S860 in the method shown.

[0465] The processor 5002 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the training method of the neural network model of the method embodiment of the present application.

[0466] The processor 5002 may also be an integrated circuit chip having signal processing capabilities, for example, Figure 5 During implementation, each step of the neural network model training method of the present application can be completed by hardware integrated logic circuits in the processor 5002 or software instructions.

[0467] The above-mentioned processor 5002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 5001, and the processor 5002 reads the information in the memory 5001 and completes it in combination with its hardware. Figure 13 The functions required to be performed by the units included in the training device shown, or the functions required to perform the method embodiments of the present application Figure 8 The training method of the neural network model shown.

[0468] The communication interface 5003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 5000 and other devices or a communication network. For example, training data can be obtained through the communication interface 5003.

[0469] The bus 5004 may include a path for transmitting information between various components of the device 5000 (eg, the memory 5001 , the processor 5002 , and the communication interface 5003 ).

[0470] Figure 16 Schematic diagram of the hardware structure of the image processing device according to an embodiment of the present application. Figure 16 The image processing apparatus 6000 shown includes a memory 6001 , a processor 6002 , a communication interface 6003 , and a bus 6004 . The memory 6001 , the processor 6002 , and the communication interface 6003 are connected to each other via the bus 6004 .

[0471] The memory 6001 may be a ROM, a static storage device, or a RAM. The memory 6001 may store a program. When the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute the various steps of the image processing method of the embodiment of the present application. Specifically, the processor 6002 may execute the above Figure 12 Steps S1210 to S1220 in the method shown

[0472] The processor 6002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

[0473] The processor 6002 may also be an integrated circuit chip having signal processing capabilities, for example, Figure 5 During implementation, each step of the image processing method of the embodiment of the present application can be completed by hardware integrated logic circuits in the processor 6002 or software instructions.

[0474] The processor 6002 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor, or it may be any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module may be located in a storage medium well-established in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 6001. The processor 6002 reads the information in the memory 6001 and, in conjunction with its hardware, performs the functions required to be performed by the units included in the image processing apparatus of the embodiments of this application, or executes the image processing method of the method embodiments of this application.

[0475] The communication interface 6003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 6000 and other devices or a communication network. For example, the image to be processed can be obtained through the communication interface 6003.

[0476] The bus 6004 may include a path for transmitting information between various components of the device 6000 (eg, the memory 6001 , the processor 6002 , and the communication interface 6003 ).

[0477] It should be noted that although the above-mentioned apparatus 5000 and apparatus 6000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the apparatus 5000 and apparatus 6000 may also include only the devices necessary to implement the embodiments of the present application, and do not necessarily include Figure 15 and Figure 16 All devices shown in .

[0478] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0479] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0480] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0481] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0482] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0483] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0484] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0485] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0486] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0487] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0488] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0489] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0490] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A training method for a neural network model, characterized in that: include: Acquire training data, where the training data is an image or text; Inputting the training data into a neural network model for feature extraction, and obtaining a first prediction label based on the extracted features, wherein the neural network model is applied to the field of image processing or the field of text processing; Determining a function value of a first loss function corresponding to the first predicted label according to the first predicted label and the true label of the training data, wherein the function value of the first loss function corresponding to the first predicted label is used to indicate a difference between the first predicted label and the true label of the training data; Calculating a first weight gradient of the neural network model according to a function value of a first loss function corresponding to the first predicted label; Obtaining a candidate weight parameter, wherein the partial derivative of a function value of a target loss function with respect to the candidate weight parameter is 0, wherein the function value of the target loss function is determined based on a function value of a second loss function corresponding to the first prediction label, and the function value of the second loss function corresponding to the first prediction label is used to indicate a difference between the candidate weight parameter and the weight parameter of the neural network model and a difference between a weight change and the first weight gradient, wherein the weight change is a difference between the candidate weight parameter and the weight parameter of the neural network model; Update the parameters of the neural network model according to the candidate weight parameters.

2. The method according to claim 1, characterized in that The method further comprises: Obtain at least one second prediction label according to features output by at least one downsampling layer in the neural network model; Determining a function value of a first loss function corresponding to the at least one second predicted label according to the at least one second predicted label and the true label of the training data, wherein the function value of the first loss function corresponding to the at least one second predicted label is used to indicate a difference between the at least one second predicted label and the true label of the training data; Calculating at least one second weight gradient of the neural network model according to the function value of the first loss function corresponding to the at least one second predicted label; and The function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to the at least one second prediction label, and the function value of the second loss function corresponding to the at least one second prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the second weight gradient.

3. The method according to claim 1 or 2, characterized in that The function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to the at least one second predicted label, including: The weight corresponding to the function value of the second loss function corresponding to the at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

4. The method according to claim 1 or 2, characterized in that The neural network model includes a first feature normalization layer, which is located before the output layer of the neural network model. The first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

5. The method according to claim 2, characterized in that The neural network model includes at least one second feature normalization layer, each of which is located after the at least one downsampling layer, and the at least one second feature normalization layer is used to perform feature normalization processing on features input into the at least one second feature normalization layer, and Obtaining at least one second prediction label according to features output by at least one downsampling layer in the neural network model includes: The features output by at least one downsampling layer in the neural network model are respectively input into the at least one second feature normalization layer for feature normalization processing, and at least one second prediction label is obtained according to the processed results.

6. The method according to claim 1 or 2, characterized in that The second loss function satisfies the following formula: Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function for indicating the difference between the weight change and the weight gradient, d2 represents a function for indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

7. The method according to claim 1 or 2, characterized in that The updating of the parameters of the neural network model according to the candidate weight parameters includes: The parameters of the neural network model are updated according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

8. The method according to claim 7, characterized in that The weight parameters of the updated neural network model satisfy the following formula: w k+1 =(w-w k )*lr+w k ; Among them, w k+1 Represents the weight parameters of the updated neural network model, w k represents the weight parameter of the neural network model, w represents the candidate weight parameter, and lr represents the learning rate.

9. An image processing method, characterized in that: include: Get the image to be processed; Using a target neural network model to extract features of the image to be processed, and obtaining a processing result of the image to be processed according to the features extracted by the target neural network model; The target neural network model is obtained by inputting training data into the neural network model for feature extraction, obtaining a first predicted label based on the features extracted by the neural network model, determining the function value of the first loss function corresponding to the first predicted label based on the first predicted label and the true label of the training data, the function value of the first loss function corresponding to the first predicted label is used to indicate the difference between the first predicted label and the true label of the training data, calculating the first weight gradient of the neural network model based on the function value of the first loss function corresponding to the first predicted label, obtaining the candidate weight parameter, the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0, and updating the parameters of the neural network model based on the candidate weight parameter, wherein the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first predicted label, the function value of the second loss function corresponding to the first predicted label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

10. The method according to claim 9, characterized in that The function value of the objective loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label. The function value of the second loss function corresponding to the at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient. The at least one second prediction label is obtained based on the features output by at least one downsampling layer in the neural network model. The function value of the second loss function corresponding to the at least one second prediction label is determined based on the at least one second prediction label and the true label of the training data. The function value of the first loss function corresponding to the at least one second prediction label is used to indicate the difference between the at least one second prediction label and the true label of the training data. The at least one second weight gradient is calculated based on the function value of the first loss function corresponding to the at least one second prediction label.

11. The method according to claim 9 or 10, characterized in that The function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to the at least one second predicted label, including: The weight corresponding to the function value of the second loss function corresponding to the at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

12. The method according to claim 9 or 10, characterized in that The target neural network model includes a first feature normalization layer, which is located before the output layer of the target neural network model. The first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

13. The method according to claim 10, characterized in that The neural network model includes at least one second feature normalization layer, which is located after the at least one downsampling layer. The at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer. The at least one second prediction label is obtained by inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and the processing result is obtained.

14. The method according to claim 9 or 10, characterized in that The second loss function satisfies the following formula: Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function for indicating the difference between the weight change and the weight gradient, d2 represents a function for indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

15. The method according to claim 9 or 10, characterized in that The updating of the parameters of the neural network model according to the candidate weight parameters includes: The parameters of the neural network model are updated according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

16. The method according to claim 15, characterized in that The weight parameters of the updated neural network model satisfy the following formula: w k+1 =(w-w k )*lr+w k ; Among them, w k+1 Represents the weight parameters of the updated neural network model, w k represents the weight parameter of the neural network model, w represents the candidate weight parameter, and lr represents the learning rate.

17. A training device for a neural network model, characterized in that: include: An acquisition unit, configured to acquire training data, wherein the training data is an image or text; A processing unit for: Inputting the training data into a neural network model for feature extraction, and obtaining a first prediction label based on the extracted features, wherein the neural network model is applied to the field of image processing or the field of text processing; Determining a function value of a first loss function corresponding to the first predicted label according to the first predicted label and the true label of the training data, wherein the function value of the first loss function corresponding to the first predicted label is used to indicate a difference between the first predicted label and the true label of the training data; Calculating a first weight gradient of the neural network model according to a function value of a first loss function corresponding to the first predicted label; Obtaining a candidate weight parameter, wherein the partial derivative of a function value of a target loss function with respect to the candidate weight parameter is 0, wherein the function value of the target loss function is determined based on a function value of a second loss function corresponding to the first prediction label, and the function value of the second loss function corresponding to the first prediction label is used to indicate a difference between the candidate weight parameter and the weight parameter of the neural network model and a difference between a weight change and the first weight gradient, wherein the weight change is a difference between the candidate weight parameter and the weight parameter of the neural network model; Update the parameters of the neural network model according to the candidate weight parameters.

18. The device according to claim 17, characterized in that The processing unit is further configured to: Obtain at least one second prediction label according to features output by at least one downsampling layer in the neural network model; Determining a function value of a first loss function corresponding to the at least one second predicted label according to the at least one second predicted label and the true label of the training data, wherein the function value of the first loss function corresponding to the at least one second predicted label is used to indicate a difference between the at least one second predicted label and the true label of the training data; Calculating at least one second weight gradient of the neural network model according to the function value of the first loss function corresponding to the at least one second predicted label; and The function value of the target loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to the at least one second prediction label, and the function value of the second loss function corresponding to the at least one second prediction label is used to indicate the difference between the candidate weight parameters and the weight parameters of the neural network model and the difference between the weight change and the second weight gradient.

19. The device according to claim 17 or 18, characterized in that The function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to the at least one second predicted label, including: The weight corresponding to the function value of the second loss function corresponding to the at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

20. The device according to claim 17 or 18, characterized in that The neural network model includes a first feature normalization layer, which is located before the output layer of the neural network model. The first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

21. The device according to claim 18, characterized in that The neural network model includes at least one second feature normalization layer, each of which is located after the at least one downsampling layer, and the at least one second feature normalization layer is used to perform feature normalization processing on features input into the at least one second feature normalization layer. The processing unit is specifically used to: The features output by at least one downsampling layer in the neural network model are respectively input into the at least one second feature normalization layer for feature normalization processing, and at least one second prediction label is obtained according to the processed results.

22. The device according to claim 17 or 18, characterized in that The second loss function satisfies the following formula: Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function for indicating the difference between the weight change and the weight gradient, d2 represents a function for indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

23. The device according to claim 17 or 18, characterized in that The processing unit is specifically configured to: The parameters of the neural network model are updated according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

24. The device according to claim 23, characterized in that The weight parameters of the updated neural network model satisfy the following formula: w k+1 =(w-w k )*lr+w k ; Among them, w k+1 Represents the weight parameters of the updated neural network model, w k represents the weight parameter of the neural network model, w represents the candidate weight parameter, and lr represents the learning rate.

25. An image processing device, characterized in that: include: An acquisition unit, configured to acquire an image to be processed; An image processing unit, configured to extract features of the image to be processed using a target neural network model, and obtain a processing result of the image to be processed based on the features extracted by the target neural network model; The target neural network model is obtained by inputting training data into the neural network model for feature extraction, obtaining a first predicted label based on the features extracted by the neural network model, determining the function value of the first loss function corresponding to the first predicted label based on the first predicted label and the true label of the training data, the function value of the first loss function corresponding to the first predicted label is used to indicate the difference between the first predicted label and the true label of the training data, calculating the first weight gradient of the neural network model based on the function value of the first loss function corresponding to the first predicted label, obtaining the candidate weight parameter, the partial derivative of the function value of the target loss function with respect to the candidate weight parameter is 0, and updating the parameters of the neural network model based on the candidate weight parameter, wherein the function value of the target loss function is determined based on the function value of the second loss function corresponding to the first predicted label, the function value of the second loss function corresponding to the first predicted label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the first weight gradient, and the weight change is the difference between the candidate weight parameter and the weight parameter of the neural network model.

26. The device according to claim 25, characterized in that The function value of the objective loss function is determined based on the function value of the second loss function corresponding to the first prediction label and the function value of the second loss function corresponding to at least one second prediction label. The function value of the second loss function corresponding to the at least one second prediction label is used to indicate the difference between the candidate weight parameter and the weight parameter of the neural network model and the difference between the weight change and the second weight gradient. The at least one second prediction label is obtained based on the features output by at least one downsampling layer in the neural network model. The function value of the second loss function corresponding to the at least one second prediction label is determined based on the at least one second prediction label and the true label of the training data. The function value of the first loss function corresponding to the at least one second prediction label is used to indicate the difference between the at least one second prediction label and the true label of the training data. The at least one second weight gradient is calculated based on the function value of the first loss function corresponding to the at least one second prediction label.

27. The device according to claim 25 or 26, characterized in that The function value of the target loss function is determined according to the function value of the second loss function corresponding to the first predicted label and the function value of the second loss function corresponding to the at least one second predicted label, including: The weight corresponding to the function value of the second loss function corresponding to the at least one second prediction label in the function value of the target loss function is negatively correlated with the number of updates of the neural network model.

28. The device according to claim 25 or 26, characterized in that The target neural network model includes a first feature normalization layer, which is located before the output layer of the target neural network model. The first feature normalization layer is used to perform feature normalization processing on the features input into the first feature normalization layer.

29. The device according to claim 26, characterized in that The neural network model includes at least one second feature normalization layer, which is located after the at least one downsampling layer. The at least one second feature normalization layer is used to perform feature normalization processing on the features input into the at least one second feature normalization layer. The at least one second prediction label is obtained by inputting the features output by the at least one downsampling layer in the neural network model into the at least one second feature normalization layer for feature normalization processing, and the processing result is obtained.

30. The device according to claim 25 or 26, characterized in that The second loss function satisfies the following formula: Among them, L new The second loss function, w represents the candidate weight parameter, w k Represents the weight parameters of the neural network model; represents the weight gradient of the neural network model, d1 represents a function for indicating the difference between the weight change and the weight gradient, d2 represents a function for indicating the difference between the candidate weight parameters and the weight parameters of the neural network model, λ1 represents the weight of d1, and λ2 represents the weight of d2.

31. The device according to claim 25 or 26, characterized in that The updating of the parameters of the neural network model according to the candidate weight parameters includes: The parameters of the neural network model are updated according to the candidate weight parameters and a learning rate, wherein the learning rate is used to indicate the magnitude of change of the weight parameters of the neural network model.

32. The device according to claim 31, characterized in that The weight parameters of the updated neural network model satisfy the following formula: w k+1 =(w-w k )*lr+w k ; Among them, w k+1 Represents the weight parameters of the updated neural network model, w k represents the weight parameter of the neural network model, w represents the candidate weight parameter, and lr represents the learning rate.

33. A training device for a neural network model, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 8.

34. An image processing device, characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 9 to 16.

35. A computer-readable storage medium, characterized in that The computer-readable medium stores program codes for execution by a device, the program codes including codes for executing the method according to any one of claims 1 to 8 or 9 to 16.

36. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 8 or 9 to 16.

Images