Neural network training method and device

Abstract

The invention provides a neural network training method and device, and the method comprises the steps: training a target neural network in a first time period when the training of the target neural network is started through employing a first training mode, and the training precision corresponding to the first training mode is the first precision; in a second time period of training of the target neural network, a second training mode is used for training the target neural network, the training precision corresponding to the second training mode is second precision, the first precision is lower than the second precision, and the starting moment of the second time period is the ending moment of the first time period; the end moment of the second time period is the training end moment of the target neural network. According to the method provided by the invention, the training time of the neural network can be shortened under the condition that the training precision of the neural network is not influenced.

Description

Method and device for training neural network

[0001] This application claims priority to Russian patent application No. 2023123996 filed on September 18, 2023, entitled “A processing framework and method for downshift training”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI), and more specifically to a method and apparatus for training a neural network. Background Art

[0003] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0004] An AI development framework is a tool library that facilitates rapid AI model development for AI developers. It encapsulates a variety of callable operators and includes the tools necessary for AI model development, training, and deployment. Among various AI development frameworks in the industry (such as PyTorch, TensorFlow, and MindSpore), shortening training time while maintaining neural network training accuracy is a key competitive advantage.

[0005] In a related technical solution, during the entire training cycle of the neural network model, multiple precision modes are used to train the neural network, and the multiple precision training modes are repeatedly switched. Such repeated switching will result in a longer training cycle of the neural network model, thereby making the model training time longer.

[0006] Therefore, how to shorten the training time of neural networks without affecting the training accuracy of neural networks has become a technical problem that needs to be solved urgently.

[0007] Summary of the Invention

[0008] The present application provides a method, apparatus, and computing device for training a neural network, which can shorten the training time of the neural network without affecting the training accuracy of the neural network.

[0009] In a first aspect, a method for neural network training is provided, the method comprising: within a first time period at the start of target neural network training, training the target neural network using a first training mode, the training accuracy corresponding to the first training mode being a first accuracy; within a second time period of the target neural network training, training the target neural network using a second training mode, the training accuracy corresponding to the second training mode being a second accuracy, the first accuracy being lower than the second accuracy, the start time of the second time period being the end time of the first time period, and the end time of the second time period being the end time of the training of the target neural network.

[0010] In the above technical solution, by using a lower-precision training mode for training in the first time period of the target neural network training and using a higher-precision training mode for training in the second time period, the training time of the target neural network can be shortened while ensuring that the training accuracy of the target neural network is not reduced.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: obtaining a current value and a target value of a parameter for training the target neural network, the parameter being a parameter used to characterize a training time dimension during the training of the target neural network; determining that the current training of the target neural network is within the first time period based on a ratio between the current value and the target value of the parameter satisfying a first preset condition; or determining that the current training of the target neural network is within the second time period based on a ratio between the current value and the target value of the parameter satisfying a second preset condition.

[0012] In the above technical solution, the ratio between the current value of the parameter used to characterize the training time dimension and the target value during the training of the target neural network satisfies the first preset condition or the second preset condition, thereby judging whether the current training of the target neural network is in the first time period or the second time period, and then the target neural network can be trained with the corresponding training accuracy in the corresponding time period.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the parameters of the target neural network training include but are not limited to at least one of the following: epoch, iteration step, and loss value loss.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

[0016] In combination with the first aspect, in some implementations of the first aspect, the first coefficient is a value between (0, 1).

[0017] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

[0019] In combination with the first aspect, in some implementations of the first aspect, the second coefficient is a value between (0, 1).

[0020] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

[0022] In combination with the first aspect, in some implementations of the first aspect, the third coefficient is a number greater than 1.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) may be configured by the user himself through the client.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned coefficients (including the first coefficient, the second coefficient, and the third coefficient) may also be automatically set by the system by default. In this way, the user does not need to configure the coefficients himself, so that the user does not perceive the different training modes adopted in the first time period and the second time period, thereby improving the user experience.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the first precision corresponds to a low-precision model compiled from the target neural network, and the second precision corresponds to a high-precision model compiled from the target neural network. During the first time period, the first training mode is used to train the compiled low-precision model; during the second time period, the second training mode is used to train the compiled high-precision model.

[0026] In conjunction with the first aspect, in certain implementations of the first aspect, the precision of the operators included in the compiled low-precision model is the first precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled low-precision model is the first precision.

[0027] In conjunction with the first aspect, in certain implementations of the first aspect, the precision of the operators included in the compiled high-precision model is the second precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled high-precision model is the second precision.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned operator may include but is not limited to: a matrix multiplication operator.

[0029] In a second aspect, a device for neural network training is provided, comprising: a first training module and a second training module, wherein the first training module is used to train the target neural network using a first training mode within a first time period at the start of the target neural network training, and the training accuracy corresponding to the first training mode is a first accuracy; the second training module is used to train the target neural network using a second training mode within a second time period of the target neural network training, and the training accuracy corresponding to the second training mode is a second accuracy, the first accuracy is lower than the second accuracy, the start time of the second time period is the end time of the first time period, and the end time of the second time period is the end time of the training of the target neural network.

[0030] In conjunction with the second aspect, in certain implementations of the second aspect, the apparatus further includes: an acquisition module and a determination module. The acquisition module is configured to acquire a current value and a target value of a parameter for training the target neural network, the parameter being a parameter used to characterize a training time dimension during the training of the target neural network; the determination module is configured to determine, based on a ratio between the current value and the target value of the parameter satisfying a first preset condition, whether the current training of the target neural network is within the first time period; or the determination module is configured to determine, based on a ratio between the current value and the target value of the parameter satisfying a second preset condition, whether the current training of the target neural network is within the second time period.

[0031] In combination with the second aspect, in some implementations of the second aspect, the parameters of the target neural network training include at least one of the following: epoch, iteration step, and loss value loss.

[0032] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

[0034] In combination with the second aspect, in some implementations of the second aspect, the first coefficient is a value between (0, 1).

[0035] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

[0037] In combination with the second aspect, in some implementations of the second aspect, the second coefficient is a value between (0, 1).

[0038] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

[0039] In combination with the second aspect, in certain implementations of the second aspect, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

[0040] In combination with the second aspect, in some implementations of the second aspect, the third coefficient is a number greater than 1.

[0041] In combination with the second aspect, in certain implementations of the first aspect, the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) can be configured by the user himself through the client.

[0042] In combination with the second aspect, in certain implementations of the first aspect, the above-mentioned coefficients (including the first coefficient, the second coefficient, and the third coefficient) can also be automatically defaulted by the system. In this way, the user does not need to configure the coefficients himself, so that the user does not perceive the different training modes adopted in the first time period and the second time period, thereby improving the user experience.

[0043] In combination with the second aspect, in certain implementations of the second aspect, the first precision corresponds to a low-precision model compiled from the target neural network, and the second precision corresponds to a high-precision model compiled from the target neural network. The first training module is specifically used to: use the first training mode to train the compiled low-precision model within the first time period; and the second training module is specifically used to: use the second training mode to train the compiled high-precision model within the second time period.

[0044] In conjunction with the second aspect, in certain implementations of the second aspect, the precision of the operators included in the compiled low-precision model is the first precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled low-precision model is the first precision.

[0045] In conjunction with the second aspect, in certain implementations of the second aspect, the precision of the operators included in the compiled high-precision model is the second precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled high-precision model is the second precision.

[0046] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned operator may include but is not limited to: a matrix multiplication operator.

[0047] It should be understood that the beneficial effects corresponding to the second aspect and each aspect are similar to the beneficial effects of the first aspect and each aspect. Please refer to the beneficial effects of the first aspect and each aspect for details, and no further details will be given here.

[0048] In a third aspect, a computing device is provided, comprising a processor and a memory; the processor is used to execute instructions stored in the memory of the computing device, so that the computing device executes the method in the first aspect or any possible implementation of the first aspect.

[0049] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0050] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device, the computing device executes the method in the first aspect and any one of the implementations of the first aspect.

[0051] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method as described in the first aspect and any one of the implementations of the first aspect.

[0052] By way of example, these computer-readable storages include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0053] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a schematic block diagram of the training phase and inference phase of a deep learning model.

[0055] FIG2 is a schematic block diagram of a deep learning model 100 .

[0056] FIG3 is a schematic block diagram of an application scenario applicable to an embodiment of the present application.

[0057] FIG4 is a schematic flowchart of a neural network training method provided in an embodiment of the present application.

[0058] FIG5 is a schematic flowchart of another neural network training method provided in an embodiment of the present application.

[0059] FIG6 is a schematic block diagram of a neural network training apparatus 600 provided in an embodiment of the present application.

[0060] FIG7 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0061] FIG8 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0062] FIG9 is a schematic diagram of a network connection between computing devices 1500A and 1500B provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solution in this application will be described below with reference to the accompanying drawings.

[0064] This application will present various aspects, embodiments, or features around systems including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0065] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0066] In the embodiments of the present application, "corresponding" and "relevant" can sometimes be used interchangeably.

[0067] It should be pointed out that when the difference is not emphasized, the meaning they express is the same.

[0068] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0069] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0070] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0071] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0072] The basic principle of AI is to combine massive data with super-powerful computing processing capabilities and intelligent algorithms to establish an AI model to solve specific problems, so that the AI ​​model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking close to that of humans.

[0073] An AI model, also known as an AI algorithm (or AI operator), is a general term for mathematical algorithms based on the principles of artificial intelligence. It is also the basis for using AI to solve specific problems. Depending on the specific methods and / or technologies used to implement AI, AI models may also be referred to as machine learning models, deep learning models, or reinforcement learning models. The following details machine learning, machine learning models, deep learning, deep learning models, neural networks, reinforcement learning, and reinforcement learning models.

[0074] Machine learning is an approach to artificial intelligence. Its goal is to design and analyze algorithms (also known as models) that enable computers to automatically "learn." These algorithms are called machine learning models. Machine learning models automatically analyze data to identify patterns and use these patterns to make predictions about unknown data. There are many different types of machine learning models. Depending on whether the model training relies on labels corresponding to the training data, machine learning models can be divided into: 1. supervised learning models; 2. unsupervised learning models.

[0075] 1. Supervised learning model: A model obtained by determining the parameters of the initial AI model based on the data in a given training data set and the labels corresponding to each data in the training data set. The process of determining the parameters of the initial AI model using the data in the training data set and the labels corresponding to the data is also called supervised learning (or supervised training). The labels of the data in the training data set are usually manually annotated to identify the correct answer to the data for a specific task. Typical supervised learning models include: support vector machines, neural network models, logistic regression models, decision trees, naive Bayes models, Gaussian discriminant models, etc. Supervised learning models are usually used for classification or regression.

[0076] 2. Unsupervised learning model: This is a model obtained by determining the parameters of the initial AI model based on the unlabeled data in a given training dataset. The process of using unlabeled training data to determine the parameters of the initial AI model is also called unsupervised learning (or unsupervised training). Through unsupervised learning, the model can discover meaningful information and associations in the data, and then predict the results of the data. There are many types of unsupervised learning models, and the more commonly used ones are: clustering models (clustering), principal component analysis (PCA), anomaly detection models, autoencoders, generative adversarial networks (GANs), etc.

[0077] Deep learning is a new technical field that has emerged in the process of machine learning research. Specifically, deep learning is a method in machine learning based on deep representation learning of data. Deep learning interprets data by establishing a neural network that simulates the human brain for analysis and learning.

[0078] In the field of AI, deep learning is a learning technology based on deep neural network algorithms. Deep learning models consist of input layers, hidden layers, and output layers, and use multiple nonlinear transformations to process data.

[0079] In machine learning, almost all features need to be determined by industry experts and then encoded. However, deep learning algorithms attempt to learn features from data themselves. Algorithms designed based on deep learning are called deep learning models.

[0080] Currently, the typical structure of a deep learning model is a deep neural network. A neural network is a mathematical or computational model that mimics the structure and function of biological neural networks (the central nervous system of animals, particularly the brain). Neural networks perform computations by connecting a large number of neurons. A neural network can include multiple neural network layers with different functions, each with parameters and computational rules. Different layers in a neural network have different names depending on the computational formula or function. For example, a layer that performs convolution is called a convolutional layer, which is often used to extract features from input signals (such as images). A neural network can also be composed of multiple sub-neural networks. Different neural network structures can be applied to different scenarios (such as classification and recognition) or provide different results when used in the same scenario. Specific differences in neural network structure include one or more of the following: different number of layers in the neural network, different order of layers, and different weights, parameters, or computational formulas in each layer. A variety of different neural networks with high accuracy are already available in the industry for applications such as recognition and classification. Some neural networks can be trained with specific datasets and then used independently to complete a task or combined with other neural networks (or other functional modules) to complete a task.

[0081] In other words, deep learning models are actually machine learning models with complex neural network structures. Depending on whether the deep learning model training relies on labels corresponding to the training data, deep learning models can be divided into supervised learning models and unsupervised learning models. This will not be discussed in detail here. Classic deep learning models include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and recursive neural networks (RNNs).

[0082] Reinforcement learning is a special field in machine learning. It is a process of continuously learning optimal strategies, making sequential decisions, and obtaining maximum rewards through the interaction between the agent and the environment.

[0083] In simple terms, reinforcement learning is about learning “what to do (i.e., how to map the current situation into actions) to maximize the numerical reward signal.” The agent is not told what actions to take, but must discover which actions will produce the most lucrative rewards through trial and error.

[0084] Reinforcement learning differs from supervised and unsupervised learning in machine learning. Supervised learning involves learning from externally provided labeled training data (task-driven), while unsupervised learning involves discovering implicit structures in unlabeled data (data-driven). Reinforcement learning involves the process of finding optimal solutions through trial and error. The agent must exploit existing experience to gain benefits while also conducting trials to optimize future action options (i.e., learning from its mistakes).

[0085] Before any AI model can be used to solve a specific technical problem, it must be trained. AI model training involves using a specified initial model to calculate training data. Based on the calculation results, the model's parameters are adjusted using a specific method, allowing the model to gradually learn certain patterns and acquire specific functions. After training, a stable AI model can be used for inference. AI model inference involves using a trained AI model to calculate input data and obtain predicted inference results.

[0086] The most common method for training AI models is supervised training. For example, most deep learning models are trained using supervised training. The following, combined with Figure 1, describes the most widely used supervised training methods for deep learning models.

[0087] As shown in Figure 1, during the training phase, a training set for the deep learning model must first be constructed based on the goal. The training set includes multiple training data, each of which is labeled. The label of the training data is the correct answer to a specific question for that training data. The label can represent the goal of using the training data to train the deep learning model. For example, if a deep learning model is to be trained to recognize different animals, the training set can include multiple images of different animals (i.e., training data). Each image can have a label that identifies the type of animal contained therein, such as a cat or dog. In this example, the type of animal corresponding to each image is the label of the training data.

[0088] When training a deep learning model, training data can be input in batches into the deep learning model after parameter initialization. The deep learning model then performs calculations (i.e., inference) on the training data to obtain predictions for the training data. The predictions obtained through inference and the labels corresponding to the training data are used as data for calculating the loss according to the loss function. The loss function is used to calculate the gap between the model's predictions for the training data and the labels of the training data (i.e., the loss value) during the model training phase. The loss function can be implemented using different mathematical functions. Common loss function expressions include: mean square error loss function, logarithmic loss function, least squares method, etc.

[0089] The loss value calculated based on the loss function can be used to update the parameters of the deep learning model. Gradient descent is often used to update the parameters. Model training is an iterative process, where each iteration infers different training data and calculates the loss value. The goal of multiple iterations is to continuously update the parameters of the deep learning model and find the parameter configuration that minimizes or stabilizes the loss function.

[0090] It should be understood that the loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, in machine learning, the loss function is used for parameter estimation of the model, and the loss value obtained based on the loss function can be used to describe the degree of difference between the model's predicted value and the actual value. Common loss functions include the mean square error loss function, the support vector machine (SVM) hinge loss function, the cross entropy loss function, etc.

[0091] During the training phase, to optimize model training efficiency and post-training performance, it's necessary to set appropriate hyperparameters. Hyperparameters for deep learning models are parameters that can't be learned from the training data or changed by the training data during training. They're a concept relative to the model's parameters. Hyperparameters for deep learning models are typically set manually based on experience or experimentation. These include the learning rate, batch size, and network architecture hyperparameters (e.g., the number of network layers (also known as depth), the interaction between layers, the number and size of convolutional kernels, and activation functions). The learning rate, as a hyperparameter, controls the magnitude of updates to the model's parameter weights during training, significantly impacting both training speed and accuracy.

[0092] As shown in Figure 1, a trained deep learning model is ready to perform inference on input data. During the inference phase, data from actual application scenarios is typically used as input, and the trained deep learning model generates inference results. The inference phase is the practical application of the trained deep learning model, allowing the rapid application of AI capabilities to solve specific technical problems. Today, AI has numerous application scenarios, and inference using deep learning models can be applied across these scenarios, such as personnel recognition in access control systems, video detection, and express delivery waybill number detection and recognition.

[0093] The above only introduces the training of the most typical deep learning model as an example. The training of other types of models has slight differences, but the principles are similar. Most of them involve inferring the training data and adjusting the parameters within the model based on the inference results, with the goal of obtaining a parameter combination that stabilizes the model performance.

[0094] Generally speaking, AI models in machine learning often need to be trained using supervised learning methods. Training AI models using supervised learning methods allows the AI ​​models to more specifically learn the relationship between the training data in the training set and the corresponding annotations in the labeled training set, so that the trained AI model can be used to predict other input data with higher accuracy.

[0095] The AI ​​development framework is a tool library that allows AI developers to quickly develop AI models. It encapsulates a variety of callable operators and includes the tools needed for AI model development, training, and deployment.

[0096] During the process of building, training, and reasoning AI models, you can call the encapsulated operators in the AI ​​framework through API calls, and then combine them with some simple driver codes to complete the corresponding operations.

[0097] AI development frameworks in the industry are generally open source. Typical AI development frameworks used for developing deep learning models, also known as deep learning frameworks, include PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch, MindSpore, and PyTorch. Developers can install AI development frameworks locally and develop AI models locally, or use them on online platforms (such as online open source framework platforms and public cloud AI infrastructure development platforms).

[0098] The following is a detailed description of the training process of a possible deep learning model applied to an embodiment of the present application in conjunction with Figure 2.

[0099] FIG2 is a schematic block diagram of a deep learning model 100 . The deep learning model 100 may include an input layer 110 , a hidden layer 120 , and an output layer 130 .

[0100] It should be understood that in the embodiment of the present application, the hidden layer 120 including n (n is greater than 1) layers of neurons is used as an example for description.

[0101] It should also be understood that each of the input layer 110, the output layer 130, and the hidden layer 120 includes one or more neurons. FIG1 illustrates an example in which the input layer 110 includes two neurons, each of the n layers in the hidden layer 120 includes three neurons, and the output layer 130 includes one neuron.

[0102] The deep learning model 100 shown in FIG2 can be a fully connected neural network or a convolutional neural network (CNN). When all neurons in each layer are connected to all neurons in the next layer (the weight w of each neuron in each layer is not 0), the deep learning model 100 is a fully connected neural network model. When all neurons in each layer are not connected to all neurons in the next layer (the weight w on each neuron in each layer is partially 0), the deep learning model 100 is a CNN model.

[0103] 2 , the deep learning model 100 may include forward propagation (FP) calculation and back propagation (BP) calculation.

[0104] The following describes in detail the process of performing FP calculations in a computing node.

[0105] During the FP calculation process, training data, such as pixel information of the input image, is obtained and used as the input (i1, i2) of the input layer 110 of the deep learning model 100. The input of the input layer 110 can output a prediction result from the output layer 130 after passing through multiple neurons in the hidden layer 120. Specifically, each layer of neurons in the hidden layer 120 corresponds to a parameter matrix. The product of the input of the input layer 110 and the parameter matrix of the first layer neurons serves as the input of the first layer neurons of the hidden layer 120. The input of the first layer neurons of the hidden layer 120 passes through the activation function (for example, a sigmoid function) in the first layer neurons and outputs an output value of the first layer neurons. The output value of the first layer neurons of the hidden layer 120 and the product of the parameter matrix of the second layer neurons serve as the input of the second layer neurons of the hidden layer 120. Similarly, and so on, a prediction result is finally output from the output layer 130.

[0106] The weights in these parameter matrices need to be corrected through a large amount of training in practical applications. The parameter matrices formed by the weights obtained through training can extract pixel information from the image to be inferred input by the user, thereby helping the deep learning model 100 to correctly infer the image to be inferred.

[0107] During the jth iteration of the FP calculation, the input to the first neuron in layer 1 is: The output of the first neuron in layer 1 is: The input to the second neuron in layer 1 is: The output of the second neuron in layer 1 is: The input to the third neuron in layer 1 is: The output of the third neuron in layer 1 is: in, For input The activation function.

[0108] During the jth iteration, the input of the neurons in the first layer is:

[0109] Therefore, the input to a neuron in layer 1 can be represented as The output can be expressed as

[0110] in,

[0111] j is used to represent the number of iterations, which is generally equal to the number of times the input layer 110 obtains the input (i1, i2). Used to represent the parameter matrix of the first layer neurons in the j-th iteration process.

[0112] The product of the output B1 of the neurons in the first layer and the parameter matrix of the neurons in the second layer can be used as the input of the neurons in the second layer. Therefore, during the j-th iteration of FP, the input of the neurons in the second layer can be expressed as The output of the neurons in layer 2 can be expressed as

[0113] Similarly, in the jth iteration of FP, the input of the neurons in the i-th layer can be expressed as The output of the neuron in the i-th layer can be expressed as Among them, 1≤i≤n.

[0114] The following describes in detail the process of performing BP calculation in a computing node.

[0115] In the process of training the deep learning model 100, it is hoped that the predicted value o1 output by the output layer 130 of the deep learning model 100 is as close as possible to the prior knowledge of the training data. Prior knowledge is also called ground truth, which generally includes the prediction results corresponding to the training data provided by humans. Therefore, the parameter matrix of each layer in the deep learning model 100 can be updated based on the difference between the current predicted value and the prior knowledge (of course, there is usually an initialization process before the first update, that is, to initialize the parameter matrix corresponding to each layer of neurons in the hidden layer 120 of the deep learning model 100). The error BP algorithm is used to correct the weights of the parameter matrix in the deep learning model 100 during the training of the deep learning model 100, so that the error loss of the deep learning model 100 becomes smaller and smaller.

[0116] Specifically, there may be an error between the predicted value generated during the FP calculation and the prior knowledge. If the output predicted value exceeds the prior knowledge, the weights in the parameter matrix can be adjusted to lower the output predicted value. If the output predicted value is less than the prior knowledge, the weights in the parameter matrix can be adjusted to increase the output predicted value. Backpropagation is an error-driven backward motion that aims to obtain the optimal parameter matrix for each layer of neurons.

[0117] It should be understood that the training data input by the user may include the input training data and the prediction results corresponding to the training data provided by the human.

[0118] As an example, deep learning model 100 is applied to image recognition. The training data input to deep learning model 100 is image pixel information, and the prior knowledge corresponding to this training data is the image label "dog." The training data is input to input layer 110. After deep learning model 100 performs FP calculations, the predicted value output by output layer 130 is compared with the prior knowledge. For example, if the predicted value output by output layer 130 is "cat," the parameter matrix of each layer in deep learning model 100 can be updated based on the error between the predicted value and the prior knowledge "dog."

[0119] During the jth iteration, the BP calculation can calculate the error E between the output prediction value o1 and the prior knowledge. The weights in the parameter matrix of each layer of neurons in the deep learning model 100 can be corrected according to the error E along the direction of the output layer 130, the hidden layer 120, and the input layer 110. Specifically, the correction of the weights can be calculated by respectively calculating the gradient of the weights in the parameter matrix. The gradient The error E can be used to find the derivative of the weights in the parameter matrix, where 1≤i≤n.

[0120] In the (j+1)th iteration, the deep learning model 100 performs FP calculation first and then BP calculation similarly to the jth iteration. For example, in the FP calculation process of the (j+1)th iteration, the gradient calculated based on the FP of the jth iteration is The weights in the parameter matrix are corrected, and the predicted output value is calculated based on the corrected parameter matrix. During the BP calculation process of the (j+1)th iteration, the gradient of the weights in the parameter matrix is ​​calculated based on the error E between the output value calculated by FP in the (j+1)th iteration and the prior knowledge. So that in the (j+2)th iteration process, The weights in the parameter matrix are modified again. The weights in the parameter matrix are continuously modified during multiple iterations, so that the output value predicted by the deep learning model 100 is as close as possible to the prior knowledge of the training data.

[0121] Specifically, in the FP calculation in the (j+1)th iteration, when calculating the input and output of the neurons in the i-th layer, the parameter matrix of the neurons in the i-th layer becomes according to The process of calculating the input and output of each layer of neurons can be found in the description of the FP calculation of the jth iteration above, which will not be repeated here.

[0122] It should be noted that the parameter matrix calculation formula shown above is a possible implementation method, and other variations of the formula are also possible, all of which are within the protection scope of the embodiments of the present application.

[0123] Under various AI development frameworks in the industry (such as pytorch / tensorflow / mindspore, etc.), shortening the training time while ensuring that the training accuracy of the neural network is not reduced is the core competitiveness.

[0124] In a related technical solution, a single-precision mode is used to train a neural network. For example, the single-precision mode is a high-precision training mode, that is, the operators in the neural network training use the full-precision floating-point mode (for example, FP32) for calculation. Although this training mode can ensure the training accuracy of the neural network, the full-precision floating-point number (for example, FP32) occupies more storage space, which greatly reduces the throughput of the operator calculation, resulting in a decrease in training speed and a longer training time for the model. For another example, the single-precision mode is a low-precision training mode, that is, the operators in the neural network training use the half-precision floating-point mode (for example, FP16) for calculation. For example, the input and output precision of the operator (for example, matrix multiplication operator) in the neural network training is FP16. In this training mode, although the half-precision floating-point number (for example, FP16) occupies less storage space, it greatly improves the throughput of the operator (for example, matrix multiplication operator) calculation, thereby improving the training time of the model, but due to the low precision of the half-precision floating-point number (for example, FP16), the training accuracy of the neural network is reduced. The main difference here is that the input and output precision of the matrix multiplication operator is FP16.

[0125] In another related technical solution, during the entire training cycle of the neural network model, multiple precision modes are used to train the neural network, and the multiple precision training modes are repeatedly switched. Such repeated switching will result in a longer training cycle of the neural network model, thereby making the training time of the model longer.

[0126] In view of this, an embodiment of the present application provides a model training method, in which a lower-precision training mode is used for training in the first part of the entire training cycle of the model, and a higher-precision training mode is used for training in the second part. In this way, the training time of the model can be shortened while ensuring that the training accuracy of the neural network is not reduced.

[0127] For ease of description, the following first illustrates an application scenario of the technical solution applicable to the embodiment of the present application in conjunction with FIG3 . It should be understood that the application scenario described below is only for the purpose of illustrating the embodiment of the present application and is not intended to be limiting. During specific implementation, the technical solution provided in the embodiment of the present application can be flexibly applied according to actual needs.

[0128] Referring to Figure 3, the application scenario includes a model training system 10 and a terminal device 20. The terminal device 20 and the model training system 10 are communicatively connected through a network. The terminal device 20 is a user-side device. Users who need model training can log in to the model training system 10 through the terminal device 20.

[0129] The model training system 10 includes multiple data processing units 101, such as data processing unit 101-1, data processing unit 101-2, ..., data processing unit 101-N, etc. shown in Figure 3. The model training system 10 also requires at least one control unit 102, such as control unit 102-1, control unit 102-2, ..., control unit 102-M, etc. shown in Figure 1, and at least one storage unit 103 for storing data, such as storage unit 103-1, ..., storage unit 103-X, etc. The model training system 10 communicates with the terminal device 20 via a network interface unit 105, such as network interface unit 105-1 and network interface unit 105-2 shown in Figure 3. The model training system 10 also includes at least one processing unit 106, and each unit can be connected to each other via a bus 104. In the application scenario shown in Figure 1, the units are relatively close and can be connected via a bus. If the units are distributed in different regions, remote communication between the units can also be achieved via a network. In the application scenario shown in Figure 3, the processing unit 106 can control other units in the model training system 10 (including the data processing unit 101, the control unit 102, the network interface unit 105 and the storage unit 103, etc.) through the bus 104. One control unit 102 can execute a training task issued by the processing unit 106. One control unit 102 can call multiple data processing units 101 at the same time to coordinate multiple data processing units 101 to complete a training task together. The processing unit 106 and multiple control units 102 can be combined. The storage unit 103 is used to store sample data and deep learning models, and can also be used to store other data that needs to be stored in the model training system 10, such as some intermediate data generated during the training process, user information, user-corresponding training tasks, and system parameters, etc.

[0130] The user can log in to the model training system 10 through the terminal device 20. The model training system 10 displays the optional online model training services to the user through the terminal device 20. The display content includes information such as the number of computing resources available for each online service, training accuracy, training speed, and service price. The user can purchase the appropriate online service according to their needs. The terminal device 20 sends an online service purchased by the user to the processing unit 106 through the network interface unit 105. The processing unit 106 generates a corresponding training task based on the online service purchased by the user and configures a corresponding number of data processing units 101 and control units 102 for the training task. The user can use the configured data processing units 101 and control units 102 to perform model training.

[0131] After the user purchases the online service, the model training system 10 provides the user with an interface for uploading deep learning models and sample data through the terminal device 20, so that the user can upload the deep learning model to be trained and the sample data for training to the model training system 10 through the interface displayed on the terminal device 20, and the processing unit 106 stores the uploaded deep learning model and sample data in the storage unit 103. At the same time, the user can also set the configuration parameters of the training task corresponding to the purchased online service through the terminal device 20. Of course, the storage unit 103 of the model training system 10 can also pre-store shared sample data and some common deep learning models for user use, and the user can choose to use the sample data and deep learning model provided by the model training system 10. The processing unit 106 also needs to configure the storage address of the sample data and deep learning model uploaded or selected by the user for the training task. Of course, the sample data and configuration parameters can also be provided by the training system 10 without the need for user input.

[0132] After the processing unit 106 configures the various parameters of the training task, it imports the configuration parameters into the corresponding control unit 102, and the control unit 102 calls the corresponding data processing unit 101 and controls the data processing unit 101 to train the deep learning model using sample data. Specifically, the data processing unit 101 reads the deep learning model and sample data from the corresponding storage unit 103 according to the configured storage address, and trains the deep learning model using the sample data. After the training is completed, the trained deep learning model is stored in the corresponding storage unit 103. At the same time, the processing unit 106 ends the training task and releases the corresponding data processing unit 101 and control unit 102. After the training is completed, the processing unit 106 can send a prompt message indicating that the training is completed to the terminal device 20 through the network. The user can log in to the model training system 10 again through the terminal device 20, access the storage unit 103 corresponding to the user account through the network interface unit 105, and obtain the trained deep learning model.

[0133] In the above application scenario, the terminal device 20 and the model training system 10 are connected to each other through a network, which can be a local area network, a wide area network, etc. The terminal device 20 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a personal computer, etc. Regardless of the type of terminal device, the user can log in through the client installed in the terminal device 20 for logging in to the model training system 10, or can access the homepage of the model training system 20 through the browser in the terminal device 20 and log in to the model training system 10 through the homepage of the model training system 20. The model training system 10 can be deployed on a single server, a server cluster consisting of several servers, or a cloud computing center.

[0134] It should be noted that in practical applications, a data processing unit may correspond to a single chip, or a set number of data processing units may be integrated into the same chip. Each data processing unit integrated into the same chip can independently perform different computing tasks. Data processing units with the same data processing precision can be integrated into the same chip, or data processing units with different data processing precisions can be integrated into the same chip.

[0135] In an embodiment of the present application, the chip integrated with the data processing unit 101 can be any chip with the computing capability to train a deep learning model, for example, a V100 chip, a Vota, a neural network processing unit (NPU), a tensor processing unit (TPU), or a field programmable gate array (FPGA).

[0136] In practical applications, the control unit 102 and the processing unit 106 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The control unit 102 may also be a plurality of independent processes set in the processing unit 106, and the processing unit 106 allocates a process for each training task to control the plurality of data processing units 101 to perform the training task.

[0137] Of course, the method provided in the embodiment of the present application is not limited to the application scenario shown in Figure 3, and can also be used in other possible application scenarios, which are not limited by the embodiment of the present application. The functions that can be implemented by each device in the application scenario shown in Figure 3 will be described in the subsequent method embodiments, and will not be described in detail here.

[0138] To further illustrate the technical solution provided by the embodiment of the present application, a detailed description is provided below in conjunction with FIG4 . Although the embodiment of the present application provides the method operation steps shown in FIG4 , the method may include more or fewer operation steps based on routine or no creative effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0139] Figure 4 is a schematic flow chart of a neural network training method provided by an embodiment of the present application. As shown in Figure 4, the method may include steps 410-420, which are described in detail below.

[0140] Step 410: In a first time period after the training of the target neural network begins, the target neural network is trained using a first training mode, where the training accuracy corresponding to the first training mode is a first accuracy.

[0141] In the embodiments of the present application, the target neural network can be a model used in different fields and for processing different tasks. For example, the model can be an image processing model, image recognition model, image detection model, etc. used in the field of image processing; the model can also be a speech recognition model, speech synthesis model, etc. used in the field of speech processing; the model can also be a text processing model, semantic understanding model, machine translation model, etc. used in the field of natural language processing, and the present application does not make specific limitations here.

[0142] In an embodiment of the present application, the target neural network can be trained using a first precision within a first time period after the training of the target neural network begins. The training precision corresponding to the first training mode is the first precision.

[0143] Step 420: During the second time period of the target neural network training, the target neural network is trained using the second training mode. The training accuracy corresponding to the second training mode is a second accuracy, and the first accuracy is lower than the second accuracy.

[0144] In an embodiment of the present application, during the second time period of the target neural network training, the target neural network can be trained using the second precision until the target neural network training is completed.

[0145] In a specific implementation, the first precision corresponds to a low-precision model compiled by the target neural network, and the second precision corresponds to a high-precision model compiled by the target neural network. During the first time period, the compiled low-precision model is trained using the first training mode, and the precision of the operators included in the compiled low-precision model is the first precision. Specifically, in one example, the input and output precision of the operators in the compiled low-precision model is the first precision. During the second time period, the compiled high-precision model is trained using the second training mode, and the precision of the operators included in the compiled high-precision model is the second precision. Specifically, in one example, the input and output precision of the operators in the compiled high-precision model is the second precision.

[0146] For example, the above-mentioned operators may include but are not limited to: matrix multiplication operators.

[0147] For example, the above-mentioned compiled model (compiled low-precision model, compiled high-precision model) can be a computational graph compiled by the target neural network, or it can also be a program compiled by the target neural network. This application does not make specific limitations on this.

[0148] It should be understood that the start time of the first time period is the start time of the target neural network training, and the end time of the first time period is the start time of the second time period. The start time of the second time period is the end time of the first time period, and the end time of the second time period is the end time of the target neural network training.

[0149] In one example, the time represented by the second time period is shorter than the time represented by the first time period, that is, the first time period is greater than the second time period. For example, the first time period accounts for more than 50% of the sum of the first time period and the second time period. Preferably, the first time period accounts for 80%-90% of the sum of the first time period and the second time period.

[0150] It should be understood that the sum of the first time period and the second time period is the entire training time of the target neural network.

[0151] It should also be understood that the end time of the target neural network training refers to the performance of the trained target neural network meeting the conditions. For example, the target neural network can be tested using a test sample. If the test results show that the accuracy of the target neural network reaches a preset accuracy, then the target neural network can be considered to have met the training end conditions. For another example, the training end condition here can also refer to the iteration steps of the target neural network training reaching a preset number of iteration steps. For example, assuming the preset number of iteration steps is 500 steps, after the server completes the 500th iteration step of training for the target neural network, it can be considered that the target neural network has met the training end conditions. Of course, in actual applications, the training end conditions can also be other conditions, and this application does not specifically limit the training end conditions.

[0152] The aforementioned training precision refers to the precision corresponding to the data type of the target neural network during training. The first precision being lower than the second precision means that the precision corresponding to the data type used by the operators during training of the target neural network during the first time period is lower than the precision corresponding to the data type used by the operators during training of the target neural network during the second time period.

[0153] For example, the above-mentioned operators may include but are not limited to: matrix multiplication operators.

[0154] The embodiments of the present application do not specifically limit the data type, and the data type may include but is not limited to: int1, int2, int3, int4, int5, int6, int7, int8, hif8, fp8, bf16, fp16, int16, int32, fp32, fp64, int64, etc. Among them, the precision corresponding to these data types increases from left to right. For example, the precision of int1 is lower than the precision of int2, the precision of hif8 is lower than the precision of fp16, the precision of bf16 is lower than the precision of fp32, and the precision of fp32 is lower than the precision of fp64. In this way, under different chips, even if the precision range supported by the chips is different, the method provided in the embodiments of the present application can be used for downshift training. Here, the same number of bits is considered to be the same precision level.

[0155] The following is an explanation of some of the above data types.

[0156] Bf16, also known as a half-precision floating-point number, is a binary floating-point data type used in computers. Half-precision numbers are stored using 2 bytes (16 bits). In IEEE 754-2008, half-precision floating-point numbers are called binary16. Half-precision numbers are suitable for storing data that does not require high precision.

[0157] fp32, also known as single-precision floating-point number (single), is a binary floating-point data type used by computers. It is stored as an IEEE 32-bit (4-byte) floating-point value. The range of single-precision numbers is from -3.402823E38 to -1.401298E-45 for negative numbers and from 1.401298E-45 to 3.402823E38 for positive numbers.

[0158] fp64, also known as double-precision floating-point numbers (double), is a binary floating-point data type used by computers, using 64 bits (8 bytes) to store a floating-point number. Double-precision numbers can represent 15 or 16 significant digits in the decimal system, and the absolute value range of the numbers they can represent is approximately: 2.23×10-308 to 1.79×10308.

[0159] In the embodiments of the present application, the accuracy of the data in the first training mode may be referred to as low precision, and the accuracy of the data in the second training mode may be referred to as high precision. The difference between the low precision and the high precision may be one level of precision, or may be multiple levels of precision, which is not specifically limited in the embodiments of the present application.

[0160] It should be noted that the low-precision method can be a single low-precision method or a mixture of multiple low-precision methods. The high-precision method can be a single high-precision method or a mixture of multiple high-precision methods. This is not specifically limited in the present embodiment. As long as the accuracy corresponding to the second training mode is higher than the accuracy corresponding to the first training mode, the requirements of different users for training accuracy and training duration can be met.

[0161] As an example, the high precision and low precision can be arbitrarily selected and combined from the following: int1, int2, int3, int4, int5, int6, int7, int8, hif8, fp8, bf16, fp16, int16, int32, fp32, fp64, int64, etc. It should be understood that, when making a selection, the high precision is the precision of the neural network model that the user inputs to train, and the high precision is the high precision supported by the chip that performs the neural network model training. The low precision is the low precision supported by the chip that performs the neural network model training that the user inputs.

[0162] For example, during a first time period, the target neural network is trained using the precision of hif8, and during a second time period, the target neural network is trained using the precision of fp64. For another example, during a first time period, the target neural network is trained using the precision of fp16, and during a second time period, the target neural network is trained using the precision of fp32. For another example, during a first time period, the target neural network is trained using the precision of bf16, and during a second time period, the target neural network is trained using the precision of bf32. For another example, during a first time period, the target neural network is trained using the precision of fp16, and during a second time period, the target neural network is trained using the precision of bf32. For another example, during a first time period, the target neural network is trained using the precision of bf16, and during a second time period, the target neural network is trained using the precision of fp32.

[0163] In the embodiment of the present application, a lower-precision training mode is used for training in the first time period of the entire training cycle of the target neural network, and a higher-precision training mode is used for training in the latter time period. Since the gradient of the weight (the rate of change of the weight) is large in the early stage of the target neural network training, when the target neural network is trained using a lower-precision training mode, not only will the accuracy of the training of the target neural network not be affected, but also the storage space occupied by the lower-precision parameters is less, which greatly improves the throughput of the calculation, thereby increasing the training time of the target neural network. Therefore, for the entire training cycle of the target neural network, the training time of the target neural network can be shortened while ensuring that the training accuracy of the target neural network is not reduced.

[0164] In an embodiment of the application, the current value and target value of the parameters of the target neural network training can also be obtained, and based on the ratio between the current value and the target value of the parameters of the target neural network training, it can be determined whether the current training of the target neural network is in the first time period or the second time period, and then the target neural network can be trained using the corresponding training accuracy in the corresponding time period.

[0165] It should be understood that the present embodiment does not impose any specific restrictions on this parameter, as long as it can represent the training time dimension during the training of the target neural network. As an example, this parameter may include but is not limited to at least one of the following: epoch, iteration step, and loss value.

[0166] An epoch is a training process that passes all examples in the training dataset once (and only once). In an epoch, the training algorithm feeds all examples into the model in a predefined order for forward propagation, loss calculation, backpropagation, and parameter updates. An epoch typically consists of multiple iterations.

[0167] An iterative step represents a parameter update performed by the model during an epoch. In other words, during training, each step is completed after training a batch of data. A batch represents a group of samples fed into the model at once. When training neural networks, the amount of training data is often substantial, perhaps tens of thousands or even hundreds of thousands of examples. If we were to feed all of these data points into the model at once, the demands on computer performance and the neural network's learning capabilities would be prohibitive. Instead, the training data can be divided into multiple batches, and the samples from each batch are then fed into the model in batches for forward propagation, loss calculation, backpropagation, and parameter updates.

[0168] The loss value represents the gap or error between the predicted results of the training data and the actual results of the training data. For a detailed description of the loss value, please refer to the description above and will not be repeated here.

[0169] One implementation of the above-mentioned determination of whether the current training of the target neural network is within the first time period or the second time period based on the ratio between the current value and the target value of the parameter for training the target neural network is as follows: determining whether the current training of the target neural network is within the first time period based on the ratio between the current value and the target value of the parameter satisfying a first preset condition; or determining whether the current training of the target neural network is within the second time period based on the ratio between the current value and the target value of the parameter satisfying a second preset condition. This implementation is described in detail below in conjunction with FIG5 .

[0170] In one example, assuming that the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

[0171] In another example, assuming that the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

[0172] In another example, assuming that the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

[0173] In another example, assuming that the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

[0174] In another example, assuming that the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to the third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

[0175] In another example, assuming that the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than the third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

[0176] It should be noted that the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) can be configured by the user through the client, or can be automatically configured by the system by default. In the implementation method where the system automatically configures the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) by default, the user does not need to configure the above coefficients, so that the user is not aware of the different training modes used in the first time period and the second time period, thereby improving the user experience.

[0177] For example, the first coefficient is a value between (0, 1), the second coefficient is a value between (0, 1), and the third coefficient is a number greater than 1.

[0178] Figure 5 is a schematic flow chart of another neural network training method provided by an embodiment of the present application. As shown in Figure 5, the method may include steps 510-560, and steps 510-560 are described in detail below.

[0179] Step 510: The user inputs termination parameters for model training.

[0180] As an example, the user can input a termination parameter of the model training to the model training system 10 through the terminal device 20, and the termination parameter is used to indicate the end of the model training. The termination parameter can be, for example, a target Epoch / steps or a target loss (final loss).

[0181] Step 520: The user configures the coefficient corresponding to the termination parameter.

[0182] As an example, the user may input coefficients corresponding to termination parameters into the model training system 10 through the terminal device 20 .

[0183] Taking the termination parameter as the target steps as an example, the user can configure the coefficient value range is [0.2, 0.9]. The user can select a coefficient from the value range and input it to the model training system 10. If the user does not configure the coefficient, the model training system 10 can select the default parameter 0.9.

[0184] Taking the termination parameter as the target loss as an example, the range of the coefficient that the user can configure is a number greater than 1, and the user can select a coefficient from the range and input it into the model training system 10. If the user does not configure the coefficient, the model training system 10 can select the default parameter 1.05.

[0185] For ease of description, if the termination parameter is the target steps, the coefficient corresponding to the user-configured termination parameter is 0.9. If the termination parameter is the target loss, the coefficient corresponding to the user-configured termination parameter is 1.05.

[0186] For example, assume that the target steps value entered by the user is N and the target loss value is M.

[0187] The following describes the case where the termination parameter is the target steps in conjunction with steps 530-540.

[0188] Step 530: If the current step number is less than or equal to 0.9*N, perform training in a low-precision training mode.

[0189] As an example, if the current step number obtained by the model training system 10 is <= 0.9*N, it can be determined that the current time period is the first time period mentioned above, and the neural network model can be trained according to the low-precision training mode.

[0190] Step 540: If the current number of steps is greater than 0.9*N, perform training in a high-precision training mode.

[0191] As an example, if the current step number obtained by the model training system 10 is greater than 0.9*N, it can be determined that the current time period is the second time period mentioned above, and the neural network model can be trained according to the high-precision training mode.

[0192] The following describes the case where the termination parameter is final loss in conjunction with steps 550-560.

[0193] Step 550: If the current loss is greater than or equal to 1.05*M, perform training in a low-precision training mode.

[0194] As an example, if the current loss obtained by the model training system 10 is greater than or equal to 1.05*M, it can be determined that the current time period is the first time period mentioned above, and the neural network model can be trained according to the low-precision training mode.

[0195] Step 560: If the current loss is less than 1.05*M, perform training in high-precision training mode.

[0196] As an example, if the current loss obtained by the model training system 10 is less than 1.05*M, it can be determined that the current time period is the second time period mentioned above, and the neural network model can be trained according to the high-precision training mode.

[0197] For example, a pseudo code for implementing the method shown in FIG5 is listed below.

[0198] 1. Add an experiment option to the adaptation layer of the software stack:

[0199] -accelerate{fast / ultrafast / fast1 / ultrafast1}{step / loss}[ratio0.2~0.9 / 1.05](default value if not set)

[0200] 2. Adding APIs to the Adaptation Layer of the Software Stack

[0201] 3. Each step of the user script calls the API:

[0202] train_accelerate(step, iteration_per_loop, 10000)

[0203] train_accelerate(now_loss,final_loss)

[0204] For example, another pseudo code for implementing the method shown in FIG5 is listed below.

[0205] 1. Add an experiment option to the adaptation layer of the software stack: -accelerate[fast / ultrafast / fast1 / ultrafast1]{step / loss}[0.2~0.9] (default value if not set)

[0206] 2. In the training script, in the for loop, set two accelerate_var environment variables = step, iteration_per_loop, epochs / now_loss, final_loss (stopping criteria)

[0207] And let the user set the environment variables:

[0208] 1) If judging by step: set the step environment variable: step, epochs (0.9)

[0209] 2) If judging by loss: set the loss environment variables: now_loss, final_loss

[0210] 3. Get this environment variable in the adaptation layer of the software stack:

[0211] 1), get(accelerate_var)

[0212] 2), if(accelerate_var<=0.9steps)

[0213] set precision_mode=mix_precision

[0214] else

[0215] set precision_mode=fp32

[0216] The above describes in detail the method provided by the embodiment of the present application in conjunction with Figures 1 to 5. The following describes in detail the embodiment of the device of the present application in conjunction with Figures 6 and 7. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0217] Figure 6 is a schematic block diagram of a neural network training device 600 provided in an embodiment of the present application. The device 600 can be implemented by software, hardware, or a combination of the two. The device 600 provided in an embodiment of the present application can implement the method flow shown in Figure 4 or Figure 5 of the embodiment of the present application. The device 600 includes: a first training module 610 and a second training module 620, wherein the first training module 610 is used to train the target neural network using a first training mode within a first time period at the beginning of the target neural network training, and the training accuracy corresponding to the first training mode is a first accuracy; the second training module 620 is used to train the target neural network using a second training mode within a second time period of the target neural network training, and the training accuracy corresponding to the second training mode is a second accuracy, the first accuracy is lower than the second accuracy, the start time of the second time period is the end time of the first time period, and the end time of the second time period is the end time of the training of the target neural network.

[0218] Optionally, the apparatus 600 further includes: an acquisition module and a determination module. The acquisition module is configured to acquire a current value and a target value of a parameter for training the target neural network, where the parameter is a parameter used to characterize a training time dimension during the training of the target neural network; the determination module is configured to determine whether the current training of the target neural network is within the first time period based on a ratio between the current value and the target value of the parameter meeting a first preset condition; or the determination module is configured to determine whether the current training of the target neural network is within the second time period based on a ratio between the current value and the target value of the parameter meeting a second preset condition.

[0219] Optionally, the parameters of the target neural network training include at least one of the following: epoch, iteration step, and loss value loss.

[0220] Optionally, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

[0221] Optionally, the parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

[0222] Optionally, the first coefficient is a value between (0, 1).

[0223] Optionally, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

[0224] Optionally, the parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than the second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

[0225] Optionally, the second coefficient is a value between (0, 1).

[0226] Optionally, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

[0227] Optionally, the parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

[0228] Optionally, the third coefficient is a number greater than 1.

[0229] Optionally, the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) can be configured by the user through the client.

[0230] Optionally, the above coefficients (including the first coefficient, the second coefficient, and the third coefficient) can also be automatically set by the system by default. In this way, the user does not need to configure the coefficients himself, so that the user does not perceive the different training modes used in the first time period and the second time period, thereby improving the user experience.

[0231] Optionally, the first precision corresponds to a low-precision model compiled from the target neural network, and the second precision corresponds to a high-precision model compiled from the target neural network. The first training module 610 is specifically used to: use the first training mode to train the compiled low-precision model within the first time period; the second training module 620 is specifically used to: use the second training mode to train the compiled high-precision model within the second time period.

[0232] Optionally, the precision of the operators included in the compiled low-precision model is the first precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled low-precision model is the first precision.

[0233] Optionally, the precision of the operators included in the compiled high-precision model is the second precision. Specifically, for example, the input precision and / or output precision of the operators in the compiled high-precision model is the second precision.

[0234] Optionally, the above-mentioned operators may include but are not limited to: matrix multiplication operators.

[0235] The device 600 herein can be embodied in the form of a functional module. The term "module" herein can be implemented in software and / or hardware form, and is not specifically limited thereto. For example, a "module" can be a software program, a hardware circuit, or a combination thereof that implements the aforementioned functions. As an example of a hardware functional unit, a "module" can include at least one computing device, such as a server. Alternatively, a "module" can also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0236] The modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0237] The device provided in the embodiment of the present application adopts the same inventive concept as the above method and can achieve the same beneficial effects, which will not be described in detail here.

[0238] All relevant contents of each step involved in the embodiment of the aforementioned method can be referred to the functional description of the functional modules corresponding to the device in the embodiment of the present application, and will not be repeated here.

[0239] The division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0240] Based on the same inventive concept as the above method, the embodiment of the present application also provides a computing device, and the method provided in the embodiment of the present application can be executed by the computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer. This application is not particularly limited, as long as it can be implemented by the method provided in the embodiment of the present application. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0241] As an example, specifically, the computing device may be a device or system (not shown in FIG3 ) used for model training in the model training system 10 .

[0242] 7 , a computing device provided in an embodiment of the present application is described in detail below.

[0243] FIG7 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in FIG7 includes: at least one processor 1510 and a memory 1520.

[0244] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0245] The processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the method provided in this application. Alternatively, the processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0246] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0247] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 7, but this does not mean that there is only one bus or one type of bus.

[0248] In one possible implementation, the processor 1510 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1520 or the cache 1516.

[0249] Optionally, the processor 1510 may be an integrated circuit chip having signal processing capabilities. By way of example and not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0250] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0251] Processing unit 1512, also known as a core, is the most important component of a processor. Processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All processor calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, leveraging parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure. For example, a processing unit includes logical units such as a level 1 cache, a level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0252] In one implementation example, the memory control unit 1514 is configured to control data exchange between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to the memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0253] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbiter (not shown in FIG. 7 ) is configured in the system bus to handle and coordinate competing accesses by multiple processing units 1512 .

[0254] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other via connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0255] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When a processing unit 1512 needs to read data, it first searches the cache for the required data. If the data is found, it executes the request directly; if not, it searches the memory. Because the cache runs much faster than the memory, the cache helps the processing unit 1512 run faster.

[0256] Memory 1520 can provide runtime space for processes in computing device 1500. For example, memory 1520 stores computer programs (specifically, program code) used to generate processes. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in memory 1520. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during the execution of the process, such as intermediate data or process data, in the storage space corresponding to the aforementioned process.

[0257] Optionally, the memory is also called the internal memory. Its function is to temporarily store the data processed by the processor 1510 and the data exchanged with external storage such as a hard disk. As long as the computer is running, the processor 1510 will load the data to be calculated into the internal memory for calculation. When the calculation is completed, the processing unit 1512 will transmit the result.

[0258] By way of example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0259] The structure of the computing device 1500 listed above is only an example, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 7.

[0260] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0261] As shown in Figure 8, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0262] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0263] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement the functions of one or more modules in the above-mentioned apparatus.

[0264] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN), a local area network (LAN), or the like. FIG9 illustrates one possible implementation. As shown in FIG9 , two computing devices 1500A and 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0265] It should be understood that the functionality of the computing device 1500A shown in FIG9 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0266] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the apparatus provided above.

[0267] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.

[0268] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0269] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0270] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0271] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0272] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0273] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0274] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0275] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for training a neural network, characterized in that: include: In a first time period after the training of the target neural network begins, the target neural network is trained using a first training mode, wherein the training accuracy corresponding to the first training mode is a first accuracy; In a second time period of the target neural network training, the target neural network is trained using a second training mode, the training accuracy corresponding to the second training mode is a second accuracy, the first accuracy is lower than the second accuracy, the start time of the second time period is the end time of the first time period, and the end time of the second time period is the end time of the training of the target neural network.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining current values ​​and target values ​​of parameters of the target neural network training, wherein the parameters are parameters used to characterize the training time dimension during the target neural network training process; According to the ratio between the current value of the parameter and the target value satisfying the first preset condition, determining that the current training of the target neural network is within the first time period; or According to the ratio between the current value of the parameter and the target value satisfying the second preset condition, it is determined that the current training of the target neural network is in the second time period.

3. The method according to claim 2, characterized in that The parameters of the target neural network training include at least one of the following: Epoch, iteration step, and loss value loss.

4. The method according to claim 3, characterized in that The parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

5. The method according to claim 3, characterized in that: The parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

6. The method according to claim 3, characterized in that The parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

7. The method according to claim 3, characterized in that The parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

8. The method according to claim 3, characterized in that The parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

9. The method according to claim 3, characterized in that: The parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

10. The method according to any one of claims 1 to 9, characterized in that The first precision corresponds to a low-precision model compiled from the target neural network, and the second precision corresponds to a high-precision model compiled from the target neural network. The step of training the target neural network using a first training mode within a first time period starting from the training of the target neural network includes: In the first time period, the compiled low-precision model is trained using the first training mode; The step of training the target neural network using a second training mode during a second time period of the target neural network training includes: During the second time period, the compiled high-precision model is trained using the second training mode.

11. A model training device, characterized in that: include: A first training module is used to train the target neural network using a first training mode within a first time period at the beginning of the training of the target neural network, wherein the training accuracy corresponding to the first training mode is a first accuracy; The second training module is used to train the target neural network using a second training mode during a second time period of the target neural network training. The network is trained, the training accuracy corresponding to the second training mode is the second accuracy, the first accuracy is lower than the second accuracy, the starting time of the second time period is the end time of the first time period, and the end time of the second time period is the end time of the training of the target neural network.

12. The device according to claim 11, characterized in that The device also includes: An acquisition module, used to acquire current values ​​and target values ​​of parameters of the target neural network training, wherein the parameters are parameters used to characterize the training time dimension during the training of the target neural network; a determination module, configured to determine whether the current training of the target neural network is within the first time period according to a ratio between the current value of the parameter and the target value satisfying a first preset condition; or The determination module is used to determine whether the current training of the target neural network is in the second time period according to whether the ratio between the current value of the parameter and the target value meets the second preset condition.

13. The device according to claim 12, characterized in that The parameters of the target neural network training include at least one of the following: Epoch, iteration step, and loss value loss.

14. The device according to claim 13, characterized in that The parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than or equal to the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than the first coefficient.

15. The device according to claim 13, characterized in that The parameter is the Epoch, the first preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is less than the first coefficient, and the second preset condition is that the ratio between the current value of the Epoch and the target value of the Epoch is greater than or equal to the first coefficient.

16. The device according to claim 13, characterized in that The parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than or equal to a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than the second coefficient.

17. The device according to claim 3, characterized in that The parameter is the step, the first preset condition is that the ratio between the current value of the step and the target value of the step is less than a second coefficient, and the second preset condition is that the ratio between the current value of the step and the target value of the step is greater than or equal to the second coefficient.

18. The device according to claim 13, characterized in that The parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than or equal to a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than the third coefficient.

19. The device according to claim 13, characterized in that The parameter is the loss, the first preset condition is that the ratio between the current value of the loss and the target value of the loss is greater than a third coefficient, and the second preset condition is that the ratio between the current value of the loss and the target value of the loss is less than or equal to the third coefficient.

20. The device according to any one of claims 11 to 19, characterized in that The first precision corresponds to a low-precision model compiled from the target neural network, and the second precision corresponds to a high-precision model compiled from the target neural network. The first training module is specifically used to: train the compiled low-precision model using the first training mode within the first time period; The second training module is specifically used to: train the compiled high-precision model using the second training mode within the second time period.

21. A computing device, characterized in that including a processor and a memory; The processor is configured to execute instructions stored in the memory so that the computing device performs the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, which, when executed by a computing device, perform the method according to any one of claims 1 to 10.

Images