Model training method and related device

By using the global gradient norm moving average of the previous iteration during the training process of BERT model, the problems of slow convergence speed and high training cost during the training process of BERT model are solved, and the iteration time and reduction are achieved The effect of total time cost.

WO2024239927A1PCT designated stage expired Publication Date: 2024-11-28HUAWEI TECH CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/090594
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-04-29
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The loss convergence speed during the training process of the BERT model is slow, resulting in high training costs. The calculation and use of global gradient norms need to be performed after the model backpropagation is completed, which increases the tailing time of the model training process.

Method used

In each iteration of the model, the moving average (MGGN) of the global gradient norm of the previous iteration is used for gradient normalization, backpropagation and gradient normalization are performed in advance, reducing the tailing time during the iteration process.

Benefits of technology

It effectively shortens the time of each iteration during the model training process and reduces the total time cost of the entire training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024090594_28112024_PF_FP_ABST
    Figure CN2024090594_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application discloses a model training method and a related device, capable of reducing the total time costs required for a model training process. The method of the present application comprises: in a t-th iteration of a model to be trained, once a gradient of an N-th layer of said model in the t-th iteration is obtained, the gradient of the N-th layer in the t-th iteration can be immediately normalized on the basis of a moving average of global gradient norm (MGGN) in a (t-1)-th iteration, so that a normalized gradient of the N-th layer in the t-th iteration is obtained; and by analogy, once a gradient of a first layer in the t-th iteration is obtained, the gradient of the first layer in the t-th iteration can be immediately normalized on the basis of the MGGN in the (t-1)-th iteration, so that a normalized gradient of the first layer in the t-th iteration is obtained. Thus, parameters of the first layer to the N-th layer can be updated on the basis of the normalized gradients of the first layer to the N-th layer in the t-th iteration, so that the t-th iteration of said model is completed.
Need to check novelty before this filing date? Find Prior Art

Description

A model training method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 4, 2023, with application number 202310492136.8 and invention name “A model training method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a model training method and related equipment. Background Art

[0003] The bidirectional encoder representations from transformers (BERT) model, due to its powerful language representation capabilities, has demonstrated excellent performance in tasks such as machine reading comprehension. However, during BERT training, the loss convergence rate is often slow, resulting in high training costs.

[0004] In order to speed up the convergence of loss during model training, related technologies have proposed a method based on global gradient normalization to complete the training of the BERT model. Specifically, after obtaining the loss based on the training data input to the model to be trained, backpropagation for the model to be trained can be completed based on the loss, thereby obtaining the gradients of each layer in the model to be trained. Then, the global gradient norm can be calculated based on the gradients of each layer, and the global gradient norm can be used to normalize the gradients of each layer. Then, the normalized gradients of each layer are used to update the parameters of each layer. Continuously repeating the above process can accelerate the convergence of the loss and complete the entire training process of the model to be trained, thereby obtaining the BERT model.

[0005] However, the calculation and use of the global gradient norm must be performed after the backpropagation of the model is completed, which undoubtedly increases the tail time of the model training process and causes the time cost of the model training process to be too high.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a model training method and related equipment, which can effectively shorten the tail time in each iteration of the model, thereby reducing the total time cost required for the entire training process of the model.

[0008] A first aspect of the embodiments of the present application provides a model training method, the method comprising:

[0009] Assume that the entire training process of the model to be trained includes K iterations, K ≥ 2. After completing the first iteration of the model to be trained, the second iteration of the model to be trained is entered to the Kth iteration of the model to be trained.

[0010] From the 2nd iteration to the Kth iteration of the model to be trained, since the process of each iteration is similar, any one of the iterations is schematically introduced below, and the iteration is called the tth iteration, K≥t≥2. In the t-th iteration of the model to be trained, based on the training data input to the model to be trained, after obtaining the gradient of the N-th layer of the model to be trained in the t-th iteration, the gradient of the N-th layer in the t-th iteration can be immediately normalized based on the moving average of the global gradient norm (MGGN) in the t-1-th iteration, thereby obtaining the normalized gradient of the N-th layer in the t-th iteration. Then, after obtaining the gradient of the N-1-th layer of the model to be trained in the t-th iteration, the gradient of the N-1-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-1-th layer in the t-th iteration. ..., after obtaining the gradient of the 1st layer of the model to be trained in the t-th iteration, the gradient of the 1st layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the 1st layer in the t-th iteration. Then, the parameters of the 1st to Nth layers can be updated based on the normalized gradients of the 1st to Nth layers in the tth iteration. At this point, the tth iteration of the model to be trained is completed, so we can enter the t+1th iteration of the model to be trained.

[0011] By repeating the above process continuously, the second iteration to the Kth iteration of the model to be trained can be gradually completed, and the target model can be obtained.

[0012] From the above method, it can be seen that: since the MGGN in the t-1th iteration can be obtained by the gradient from the 1st layer to the Nth layer in the t-1th iteration, that is, the MGGN in the t-1th iteration has been obtained in advance in the t-1th iteration of the model to be trained, after entering the t-1th iteration of the model to be trained, once the gradient of any layer in the model to be trained in the t-1th iteration is obtained, the MGGN in the t-1th iteration can be immediately used to normalize the gradient of the layer in the t-1th iteration, thereby obtaining the normalized gradient of the layer in the t-1th iteration. In this way, the backpropagation and gradient normalization in the t-1th iteration of the model to be trained can be carried out simultaneously, which can effectively shorten the tail time of the t-1th iteration of the model to be trained, thereby reducing the total time cost required for the entire training process of the model to be trained.

[0013] In one possible implementation, the method further includes: obtaining the MGGN at the tth iteration based on the gradient of the Nth layer at the tth iteration. In the aforementioned implementation, since the gradients of the first to Nth layers at the tth iteration have already been obtained, the gradients of the first to Nth layers at the tth iteration can be used to obtain the MGGN at the tth iteration.

[0014] In one possible implementation, obtaining the MGGN at iteration t based on the gradients of layer N at iteration t includes: performing a first calculation on the gradients of layer N at iteration t to obtain a global gradient norm (GGN) at iteration t; and performing a second calculation on the MGGN at iteration t-1 and the GGN at iteration t to obtain the MGGN at iteration t. In the aforementioned implementation, since the gradients of layers 1 to N at iteration t have already been obtained, the GGN at iteration t can be obtained by calculating the gradients of layers 1 to N at iteration t, and the MGGN at iteration t-1 and the GGN at iteration t can be calculated to obtain the MGGN at iteration t.

[0015] In one possible implementation, the first calculation includes at least one of the following: exponentiation, summation, and square root calculation. In the aforementioned implementation, the squares of the gradients of the first layer at iteration t, ..., and the squares of the gradients of the Nth layer at iteration t can be calculated. The squares of the gradients of the first layer at iteration t, ..., and the squares of the gradients of the Nth layer at iteration t are then added together. Finally, the square root of the sum of these squares is taken to obtain the GGN at iteration t.

[0016] In one possible implementation, the second calculation is a weighted sum calculation. In the above implementation, the GGN in the tth iteration and the MGGN in the t-1th iteration may be weighted summed using preset weights to obtain the MGGN in the tth iteration.

[0017] In one possible implementation, obtaining the gradient of the i-th layer of the model to be trained at the t-th iteration based on the training data input to the model to be trained includes: obtaining M pending gradients of the i-th layer of the model to be trained at the t-th iteration based on the M training data input to the model to be trained, where M ≥ 2; and averaging the M pending gradients to obtain the gradient of the i-th layer at the t-th iteration. In the aforementioned implementation, at the t-th iteration of the model to be trained, there are M training data input to the model to be trained. Based on these M data, M candidate gradients of the N-th layer at the t-th iteration can be obtained accordingly. The M candidate gradients of the N-th layer at the t-th iteration are directly averaged to obtain the gradient of the N-th layer at the t-th iteration, and the gradient of the N-th layer at the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the N-th layer at the t-th iteration. Then, based on these M data, the M candidate gradients of the N-1th layer in the t-th iteration can be obtained accordingly, and the M candidate gradients of the N-1th layer in the t-th iteration are directly averaged to obtain the gradient of the N-1th layer in the t-th iteration, and the gradient of the N-1th layer in the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the N-1th layer in the t-th iteration, ..., finally, based on these M data, the M candidate gradients of the 1st layer in the t-th iteration can be obtained accordingly, and the M candidate gradients of the 1st layer in the t-th iteration are directly averaged to obtain the gradient of the 1st layer in the t-th iteration, and the gradient of the 1st layer in the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the 1st layer in the t-th iteration.

[0018] In one possible implementation, the method further includes: in the first iteration of the model to be trained, based on the training data input to the model to be trained, obtaining the gradient of the N layer in the first iteration; based on the gradient of the N layer in the first iteration, obtaining the GGN in the first iteration, and using the GGN in the first iteration as the MGGN in the first iteration; based on the GGN in the first iteration, normalizing the gradient of the N layer in the first iteration to obtain the normalized gradient of the N layer in the first iteration; and based on the normalized gradient of the N layer in the first iteration, updating the parameters of the N layer to enter the second iteration of the model to be trained. In the aforementioned implementation, in the first iteration of the model to be trained, the training data can be input to the model to be trained. Then, based on the training data, the gradients of the first iteration of the model to be trained from the first layer to the N layer can be gradually obtained. Then, since the gradients of layers 1 to N in the first iteration have been obtained, the gradients of layers 1 to N in the first iteration can be calculated to obtain the GGN in the first iteration. The GGN in the first iteration can be directly used as the MGGN in the first iteration. Subsequently, the gradients of layers 1 to N in the first iteration can be normalized based on the GGN in the first iteration to obtain the normalized gradients of layers 1 to N in the first iteration. Finally, the parameters of layers 1 to N can be updated based on the normalized gradients of layers 1 to N in the first iteration. At this point, the first iteration of the model to be trained is completed, and the second iteration of the model to be trained can be entered.

[0019] A second aspect of an embodiment of the present application provides a model training device, which includes: a first acquisition module, used to obtain the gradient of the i-th layer of the model to be trained in the t-th iteration of the model to be trained based on the training data input to the model to be trained, t≥2, i=1,...,N, N≥1; a normalization module, used to normalize the gradient of the i-th layer in the t-th iteration based on the MGGN in the t-1-th iteration, and obtain the normalized gradient of the i-th layer in the t-1 iteration, and the MGGN in the t-1 iteration is obtained based on the gradient of the N layer of the model to be trained in the t-1 iteration; an update module, used to update the parameters of the i-th layer based on the normalized gradient of the i-th layer in the t-th iteration, so as to enter the t+1-th iteration of the model to be trained.

[0020] It can be seen from the above device that: since the MGGN in the t-1th iteration can be obtained by the gradient from the 1st layer to the Nth layer in the t-1th iteration, that is, the MGGN in the t-1th iteration has been obtained in advance in the t-1th iteration of the model to be trained, after entering the tth iteration of the model to be trained, once the gradient of any layer in the model to be trained in the tth iteration is obtained, the MGGN in the t-1th iteration can be immediately used to normalize the gradient of the layer in the tth iteration, thereby obtaining the normalized gradient of the layer in the t-1th iteration. In this way, the backpropagation and gradient normalization in the tth iteration of the model to be trained can be carried out simultaneously, which can effectively shorten the tail time of the tth iteration of the model to be trained, thereby reducing the total time cost required for the entire training process of the model to be trained.

[0021] In a possible implementation, the apparatus further includes: a second acquisition module, configured to acquire the MGGN in the t-th iteration based on the gradient of the N layer in the t-th iteration.

[0022] In one possible implementation, the second acquisition module is used to: perform a first calculation on the gradient of the N layer in the t-th iteration to obtain the global gradient norm GGN in the t-th iteration; and perform a second calculation on the MGGN in the t-1-th iteration and the GGN in the t-th iteration to obtain the MGGN in the t-th iteration.

[0023] In a possible implementation, the first calculation includes at least one of the following: a power calculation, a sum calculation, and a square root calculation.

[0024] In one possible implementation, the second calculation is a weighted sum calculation.

[0025] In one possible implementation, the first acquisition module is used to: obtain M unprocessed gradients of the i-th layer of the to-be-trained model in the t-th iteration based on M training data input to the to-be-trained model, where M ≥ 2; and average the M unprocessed gradients to obtain the gradient of the i-th layer in the t-th iteration.

[0026] In one possible implementation, the first acquisition module is further used to obtain the gradient of the N layer in the first iteration based on the training data input into the model to be trained in the first iteration of the model to be trained; the second acquisition module is used to obtain the GGN in the first iteration based on the gradient of the N layer in the first iteration, and the GGN in the first iteration is used as the MGGN in the first iteration; the normalization module is further used to normalize the gradient of the N layer in the first iteration based on the GGN in the first iteration to obtain the normalized gradient of the N layer in the first iteration; the update module is further used to update the parameters of the N layer based on the normalized gradient of the N layer in the first iteration to enter the second iteration of the model to be trained.

[0027] A third aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the first aspect or any possible implementation method of the first aspect.

[0028] A fourth aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or any possible implementation of the first aspect.

[0029] A fifth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a possible implementation, the processor is coupled to the memory through an interface.

[0031] In one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.

[0032] A sixth aspect of the embodiments of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, enables the computer to implement the method described in the first aspect or any possible implementation of the first aspect.

[0033] A seventh aspect of the embodiments of the present application provides a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the method described in the first aspect or any possible implementation of the first aspect.

[0034] In an embodiment of the present application, in the t-th iteration of the model to be trained, after obtaining the gradient of the N-th layer of the model to be trained in the t-th iteration, the gradient of the N-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-th layer in the t-th iteration. Similarly, after obtaining the gradient of the N-1-th layer in the t-th iteration, the gradient of the N-1-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-1-th layer in the t-th iteration, ..., after obtaining the gradient of the 1st layer in the t-th iteration, the gradient of the 1st layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the 1st layer in the t-th iteration. In this way, the parameters of layers 1 through N can be updated based on the normalized gradients from layer 1 to layer N at iteration t. At this point, iteration t of the model to be trained is complete, and the model can proceed to iteration t+1. In the aforementioned process, the MGGN at iteration t-1 can be derived from the gradients from layer 1 to layer N at iteration t-1. That is, the MGGN at iteration t-1 has already been obtained at iteration t-1 of the model to be trained. Therefore, after entering iteration t of the model to be trained, once the gradients of any layer in the model to be trained at iteration t are obtained, the MGGN at iteration t-1 can be immediately used to normalize the gradients of that layer at iteration t, thereby obtaining the normalized gradients of that layer at iteration t-1. This allows backpropagation and gradient normalization to be performed simultaneously at iteration t of the model to be trained, effectively shortening the tail time of iteration t of the model to be trained, thereby reducing the total time cost required for the entire training process of the model to be trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0036] FIG2a is a schematic structural diagram of an information processing system provided in an embodiment of the present application;

[0037] FIG2 b is another schematic diagram of the structure of the information processing system provided in an embodiment of the present application;

[0038] FIG2c is a schematic diagram of an information processing device according to an embodiment of the present application;

[0039] FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;

[0040] FIG4 is a flow chart of a model training method provided in an embodiment of the present application;

[0041] FIG5 is a schematic diagram of a comparison result provided in an embodiment of the present application;

[0042] FIG6 is another schematic diagram of the comparison results provided by an embodiment of the present application;

[0043] FIG7 is a schematic diagram of an application example of the model training method provided in an embodiment of the present application;

[0044] FIG8 is another schematic diagram of a comparative structure provided in an embodiment of the present application;

[0045] FIG9 is a flow chart of an information processing method according to an embodiment of the present application;

[0046] FIG10 is a schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0047] FIG11 is a schematic structural diagram of an information processing device provided in an embodiment of the present application;

[0048] FIG12 is a schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0049] FIG13 is a schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0050] FIG14 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application provide a model training method and related equipment, which can effectively shorten the tail time in each iteration of the model, thereby reducing the total time cost required for the entire training process of the model.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] The BERT model, with its powerful language representation capabilities, has demonstrated excellent performance in tasks such as machine reading comprehension. However, during BERT training, the loss convergence rate is often slow, resulting in high training costs.

[0054] In order to speed up the convergence of loss during model training, related technologies have proposed a method based on global gradient normalization to complete the training of the BERT model. Specifically, the training process of the BERT model includes multiple iterations. In a certain iteration of the model to be trained, after the loss is obtained based on the training data input to the model to be trained, backpropagation for the model to be trained can be completed based on the loss, thereby obtaining the gradients of each layer in the model to be trained. Next, the global gradient norm can be calculated based on the gradients of each layer, and the global gradient norm can be used to normalize the gradients of each layer. Then, the normalized gradients of each layer are used to update the parameters of each layer. At this point, the iteration of the model to be trained is completed, and the next iteration of the model to be trained can be entered. In the process of multiple iterations, the convergence speed of the loss can be accelerated until the loss converges, and the entire training process of the model to be trained can be completed, thereby obtaining the BERT model.

[0055] However, the calculation of the global gradient norm and gradient normalization need to be performed after the backpropagation of the model is completed. That is to say, in each iteration of the model to be trained, the global gradient norm must be calculated after the gradients of all layers in the model to be trained are obtained, and this norm is used to normalize the gradients of all layers. This undoubtedly increases the tail time of each iteration in the model training process, resulting in an excessively high total time cost for the model training process.

[0056] In order to solve the above problems, the embodiments of the present application provide a model training method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.

[0057] First, we will describe the overall workflow of an AI system. See Figure 1, which illustrates a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0058] (1) Infrastructure

[0059] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0060] (2) Data

[0061] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0062] (3) Data processing

[0063] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0064] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0065] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0066] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0067] (4) General ability

[0068] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Smart products and industry applications

[0070] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0071] Next, several application scenarios of this application are introduced.

[0072] Figure 2a is a schematic diagram of the structure of an information processing system provided in an embodiment of the present application. The information processing system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of information processing and is the initiator of information processing requests, typically initiated by a user through the user device.

[0073] The aforementioned data processing devices can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing devices receive information processing requests from smart terminals via interactive interfaces and then perform information processing methods such as machine learning, deep learning, search, reasoning, and decision-making through data storage and data processing processors. The memory in a data processing device is a general term that includes local storage and databases that store historical data. The database can be located on the data processing device or on other network servers.

[0074] In the information processing system shown in FIG2a, a user device can receive a user's instruction. For example, the user device can obtain information input / selected by the user, and then initiate a request to a data processing device, so that the data processing device performs information processing on the information obtained by the user device, thereby obtaining a processing result for the information. For example, the user device can obtain information input by the user (for example, a question to be answered, a prompt to be subjected to sentiment analysis, an incomplete text to be supplemented, and other types of language information to be processed), and then initiate an information processing request to the data processing device, so that the data processing device performs a series of processing on the information based on the information processing request, thereby obtaining a processing result for the information (for example, an answer to a question, a sentiment belonging to a prompt, a supplementary part of an incomplete text, etc.).

[0075] In FIG. 2 a , the data processing device may execute the information processing method according to the embodiment of the present application.

[0076] Figure 2b is another structural diagram of the information processing system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.

[0077] In the information processing system shown in Figure 2b, the user device can receive user instructions. For example, the user device can obtain information input by the user (for example, a question to be answered, a prompt to be sentimentally analyzed, an incomplete text to be supplemented, and other types of language information to be processed), and then perform a series of processing on the information to obtain the processing result of the information (for example, an answer to a question, the emotion belonging to a prompt, a supplementary part of an incomplete text, etc.).

[0078] In FIG2b , the user equipment itself can execute the information processing method of the embodiment of the present application.

[0079] FIG2c is a schematic diagram of information processing related equipment provided in an embodiment of the present application.

[0080] The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or it can be set on the cloud or other network servers.

[0081] The processors in Figures 2a and 2b can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute information processing applications on the image, thereby obtaining corresponding processing results.

[0082] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data can include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.

[0083] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0084] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0085] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from training samples collected by the data collection device 160.

[0086] In the scenario shown in FIG3 , the user can manually input data, which can be performed through the interface provided by I / O interface 112. Alternatively, client device 140 can automatically send input data to I / O interface 112. If user authorization is required for client device 140 to automatically send input data, the user can set the corresponding permissions in client device 140. The user can view the output of execution device 110 on client device 140, which can be presented in a display, sound, action, or other specific form. Client device 140 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, collection can also be performed without client device 140, with I / O interface 112 directly storing the input data and output results from I / O interface 112 as new sample data in database 130.

[0087] It is worth noting that FIG3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG3, a neural network can be trained according to the training device 120.

[0088] The present application also provides a chip including a neural network processor (NPU). The chip can be provided in the execution device 110 shown in FIG3 to complete the computational work of the computation module 111. The chip can also be provided in the training device 120 shown in FIG3 to complete the training work of the training device 120 and output the target model / rule.

[0089] The neural network processor (NPU) is a coprocessor mounted on the host central processing unit (CPU), which assigns tasks to it. The core of the NPU is the arithmetic circuit, which is controlled by a controller to extract data from memory (weight memory or input memory) and perform calculations.

[0090] In some implementations, the arithmetic circuit includes multiple processing engines (PEs). In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

[0091] For example, consider an input matrix A, a weight matrix B, and an output matrix C. The computational circuit retrieves the corresponding data for matrix B from the weight memory and caches it on each PE within the computational circuit. The computational circuit then performs a matrix operation on the matrix A data from the input memory and matrix B, storing the partial or final matrix results in the accumulator.

[0092] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network calculations in non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, and local response normalization.

[0093] In some implementations, the vector computation unit can store the processed output vectors in a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vectors can be used as activation inputs to the computation circuit, such as for use in subsequent layers in a neural network.

[0094] The unified memory is used to store input data and output data.

[0095] The weight data is directly transferred from the external memory to the input memory and / or unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

[0096] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.

[0097] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

[0098] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

[0099] Generally, unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and external memory is memory outside the NPU, which can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

[0100] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0101] (1) Neural Network

[0102] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0103] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0104] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) into the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. "Bending." Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. That is, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.

[0105] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0106] (2) Backpropagation algorithm

[0107] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0108] (3) Global gradient normalization

[0109] During the training process of a neural network model, after completing backpropagation for the neural network model, the gradients of each layer in the neural network model can be obtained. To prevent gradient explosion during model training and accelerate the convergence of losses in the process, the gradients of each layer can be normalized. Specifically, after obtaining the gradients of each layer, a global gradient norm can be calculated using the gradients of each layer. The gradients of each layer are then divided by the global gradient norm to obtain the normalized gradients of each layer. This is used to update the parameters of each layer in the neural network model, thereby completing the entire training process of the model.

[0110] The method provided in this application is described below from the training side of the neural network and the application side of the neural network.

[0111] The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning, and performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on the training data (for example, the training data in the model training method provided in the embodiment of the present application), and finally obtains a trained neural network (for example, a target model obtained after training the training model based on the model training method provided in the embodiment of the present application); and the information processing method provided in the embodiment of the present application can use the above-mentioned trained neural network to input input data (for example, the information in the information processing method provided in the embodiment of the present application) into the trained neural network to obtain output data (for example, the processing result of the information in the information processing method provided in the embodiment of the present application). It should be noted that the model training method and information processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0112] FIG4 is a flow chart of a model training method provided in an embodiment of the present application. As shown in FIG4 , the method includes:

[0113] 401. After completing the first iteration of the model to be trained, enter the second iteration of the model to be trained to the Kth iteration of the model to be trained, where K ≥ 2;

[0114] In this embodiment, it is assumed that the training process of the model to be trained (for example, the BERT model to be trained) includes K iterations (K is a positive integer greater than or equal to 2), and the K iterations of the model to be trained can be divided into two parts, one part is the first iteration of the model to be trained, and the other part is the second iteration to the Kth iteration of the model to be trained. It should be noted that since the training process of the model to be trained includes K iterations, before training the model to be trained, a data set can be obtained first, which includes K batches of training data (for example, these training data can be language information in various tasks, such as questions in question-answering tasks, prompts in sentiment analysis tasks, incomplete texts in text supplementation tasks, etc.). Among these K batches of training data, one batch of training data can be used to complete one iteration of the model to be trained.

[0115] Specifically, the first iteration of the model to be trained can be completed in the following ways:

[0116] (1) In the first iteration of the model to be trained, the first batch of training data can be input into the model to be trained so that the first batch of training data can be processed by the model to be trained, thereby obtaining the processing results of the first batch of training data, and based on the processing results of the first batch of training data, obtaining the loss in the first iteration. Then, based on the loss in the first iteration, the gradients of the first layer to the Nth layer of the model to be trained in the first iteration can be gradually obtained (N is a positive integer greater than or equal to 1). Then, since the gradients of the first layer to the Nth layer in the first iteration have been obtained, the gradients of the first layer to the Nth layer in the first iteration can be calculated to obtain the GGN in the first iteration, and the GGN in the first iteration can be directly used as the MGGN in the first iteration. Subsequently, based on the GGN in the first iteration, the gradients of the first layer to the Nth layer in the first iteration can be normalized to obtain the normalized gradients of the first layer to the Nth layer in the first iteration. Finally, the parameters of the 1st to Nth layers can be updated based on the normalized gradients of the 1st to Nth layers in the 1st iteration. At this point, the 1st iteration of the model to be trained is completed, so we can enter the 2nd iteration of the model to be trained.

[0117] More specifically, the gradients from the first layer to the Nth layer in the first iteration can be calculated using a preset formula (for example, the formula can implement at least one of the following calculations: exponentiation calculation, summation calculation, and square root calculation), thereby obtaining the GGN in the first iteration and the MGGN in the first iteration. For example, the formula can be expressed as:

[0118] In the above formula, GGN1 is the GGN in the first iteration, grad 1,1 is the gradient of the first layer in the first iteration, grad 1,N is the gradient of the Nth layer in the first iteration.

[0119] (2) In the first iteration of the model to be trained, the first batch of training data can be input into the model to be trained so that the first batch of training data can be processed by the model to be trained, thereby obtaining the processing result of the first batch of training data, and based on the processing result of the first batch of training data, the loss in the first iteration is obtained. Then, based on the loss in the first iteration, the gradient of the Nth layer of the to-be-trained model in the first iteration is obtained, and the gradient of the Nth layer in the first iteration is immediately normalized based on the preset MGGN, so as to obtain the normalized gradient of the Nth layer in the first iteration. Then, based on the loss in the first iteration, the gradient of the N-1th layer of the to-be-trained model in the first iteration can be obtained, and the gradient of the N-1th layer in the first iteration is immediately normalized based on the preset MGGN, so as to obtain the normalized gradient of the N-1th layer in the first iteration, ..., finally, based on the loss in the first iteration, the gradient of the 1st layer of the to-be-trained model in the first iteration can be obtained, and the gradient of the 1st layer in the first iteration is immediately normalized based on the preset MGGN, so as to obtain the normalized gradient of the 1st layer in the first iteration. Then, since the gradients of layers 1 to N in the first iteration have been obtained, the gradients of layers 1 to N in the first iteration can be calculated to obtain the GGN in the first iteration. The preset MGGN and the GGN in the first iteration are also calculated to obtain the MGGN in the first iteration. Finally, the parameters of layers 1 to N can be updated based on the normalized gradients of layers 1 to N in the first iteration. At this point, the first iteration of the model to be trained is completed, and the second iteration of the model to be trained can be entered.

[0120] More specifically, the gradients from the first layer to the Nth layer in the first iteration can be calculated using a preset formula (for example, the formula can implement at least one of the following calculations: exponentiation calculation, sum calculation, square root calculation, and weighted sum calculation), thereby obtaining the GGN in the first iteration and the MGGN in the first iteration. For example, the formula can be expressed as:

[0121] In the above formula, MGGN p is the preset MGGN, and β is the preset weight.

[0122] Furthermore, the model to be trained can be deployed on M graphics processing units (GPUs). Then, in the first iteration of the model to be trained, the first batch of training data input to the model to be trained may include M training data (also referred to as M pieces of training data, M is a positive integer greater than or equal to 2), and these M training data can be input into the model to be trained on the M GPUs accordingly. After the models to be trained on the M GPUs process these M training data respectively, the processing results of these M training data can be obtained accordingly. Then, based on the processing results of these M training data, the M losses in the first iteration can be obtained accordingly. At this time, there are multiple situations:

[0123] Based on the above situation (1), we can first obtain M candidate gradients of the Nth layer in the first iteration based on the M losses, and directly average the M candidate gradients of the Nth layer in the first iteration to obtain the gradient of the Nth layer in the first iteration. Then, we can obtain M candidate gradients of the N-1th layer in the first iteration based on the M losses, and directly average the M candidate gradients of the N-1th layer in the first iteration to obtain the gradient of the N-1th layer in the first iteration, ..., and finally, we can obtain M candidate gradients of the 1st layer in the first iteration based on the M losses, and directly average the M candidate gradients of the 1st layer in the first iteration to obtain the gradient of the 1st layer in the first iteration. Since the gradients of the 1st to Nth layers in the first iteration have been obtained, the gradients of the 1st to Nth layers in the first iteration can be calculated to obtain the GGN in the first iteration. Subsequently, the gradients from the 1st layer to the Nth layer in the 1st iteration may be normalized based on the GGN in the 1st iteration, thereby obtaining the normalized gradients from the 1st layer to the Nth layer in the 1st iteration.

[0124] Based on the above situation (2), M candidate gradients of the Nth layer in the first iteration can be obtained based on the M losses, and the M candidate gradients of the Nth layer in the first iteration can be directly averaged to obtain the gradient of the Nth layer in the first iteration, and the gradient of the Nth layer in the first iteration can be immediately normalized based on the preset MGGN to obtain the normalized gradient of the Nth layer in the first iteration. Then, based on the M losses, M candidate gradients of the N-1th layer in the first iteration can be obtained accordingly, and the M candidate gradients of the N-1th layer in the first iteration are directly averaged to obtain the gradient of the N-1th layer in the first iteration, and the gradient of the N-1th layer in the first iteration is immediately normalized based on the preset MGGN to obtain the normalized gradient of the N-1th layer in the first iteration, ..., finally, based on the M losses, M candidate gradients of the 1st layer in the first iteration can be obtained accordingly, and the M candidate gradients of the 1st layer in the first iteration are directly averaged to obtain the gradient of the 1st layer in the first iteration, and the gradient of the 1st layer in the first iteration is immediately normalized based on the preset MGGN to obtain the normalized gradient of the 1st layer in the first iteration.

[0125] 402. In the t-th iteration of the model to be trained, based on the training data input to the model to be trained, obtain the gradient of the i-th layer of the model to be trained in the t-th iteration, K≥t≥2, i=1,...,N, N≥1.

[0126] 403. Based on the MGGN in the t-1th iteration, the gradient of the i-th layer in the t-th iteration is normalized to obtain the normalized gradient of the i-th layer in the t-th iteration, and the MGGN in the t-1th iteration is obtained based on the gradient of the N layer of the to-be-trained model in the t-1th iteration.

[0127] 404. Based on the normalized gradient of the i-th layer in the t-th iteration, update the parameters of the i-th layer to enter the t+1-th iteration of the to-be-trained model.

[0128] After completing the first iteration of the model to be trained, the second iteration of the model to be trained can be entered, and after completing the second iteration of the model to be trained, the third iteration of the model to be trained can be entered, and so on, until the K-th iteration of the model to be trained is completed. At this time, the model training conditions are met (assuming that the loss reaches convergence in the K-th iteration), and the target model (for example, the trained BERT model) can be obtained.

[0129] Specifically, the second iteration to the Kth iteration of the model to be trained can be completed in the following manner:

[0130] From the 2nd iteration to the Kth iteration of the model to be trained, since the process of each iteration is similar, any one of the iterations is schematically introduced below, and the iteration is called the tth iteration (K≥t≥2).

[0131] In the tth iteration of the model to be trained, the tth batch of training data can be input into the model to be trained, so that the tth batch of training data can be processed by the model to be trained, thereby obtaining the processing result of the tth batch of training data, and based on the processing result of the tth batch of training data, the loss in the tth iteration is obtained. Then, based on the loss in the t-th iteration, the gradient of the N-th layer of the to-be-trained model in the t-th iteration can be obtained, and the gradient of the N-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, so as to obtain the normalized gradient of the N-th layer in the t-th iteration. Then, based on the loss in the t-th iteration, the gradient of the N-1-th layer of the to-be-trained model in the t-th iteration can be obtained, and the gradient of the N-1-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration (the MGGN in the t-1-th iteration can be obtained from the gradients from the 1st layer to the N-th layer in the t-1-th iteration), so as to obtain the normalized gradient of the N-1-th layer in the t-th iteration, ..., finally, based on the loss in the t-th iteration, the gradient of the 1st layer of the to-be-trained model in the t-th iteration can be obtained, and the gradient of the 1st layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, so as to obtain the normalized gradient of the 1st layer in the t-th iteration. Then, since the gradients of layers 1 to N at iteration t have been obtained, the gradients of layers 1 to N at iteration t can be calculated to obtain the GGN at iteration t. The MGGN at iteration t-1 and the GGN at iteration t can also be calculated to obtain the MGGN at iteration t. Finally, the parameters of layers 1 to N can be updated based on the normalized gradients of layers 1 to N at iteration t. At this point, iteration t of the model to be trained is completed, and the t+1 iteration of the model to be trained can be entered.

[0132] It should be noted that when t = K, the loss in the K-th iteration has converged, so there is no need to calculate the GGN in the K-th iteration and the MGGN in the K-th iteration. After updating the parameters of the 1st to Nth layers of the model to be trained, there is no need to enter the K+1th iteration, and the final target model can be obtained directly.

[0133] More specifically, the gradients of the first to Nth layers at the tth iteration can be calculated using a preset formula (e.g., a formula that can implement at least one of the following calculations: exponentiation, summation, square root, and weighted summation), thereby obtaining the GGN at the tth iteration and the MGGN at the tth iteration. For example, the formula can be expressed as:

[0134] In the above formula, GGN t is the GGN in the tth iteration, MGGN t-1 is the MGGN in the t-1th iteration, MGGN t is the MGGN in the t-th iteration, and β is the preset weight.

[0135] Furthermore, the model to be trained can be deployed on M GPUs. Then, in the tth iteration of the model to be trained, the tth batch of training data input to the model to be trained may include M training data, and these M training data can be input to the models to be trained on the M GPUs. After the models to be trained on the M GPUs process these M training data, processing results for these M training data can be obtained. Then, based on the processing results of these M training data, M losses for the tth iteration can be obtained.

[0136] Then, based on the M losses, M candidate gradients of the N-th layer in the t-th iteration can be obtained accordingly, and the M candidate gradients of the N-th layer in the t-th iteration are directly averaged to obtain the gradient of the N-th layer in the t-th iteration, and the gradient of the N-th layer in the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the N-th layer in the t-th iteration. Then, based on the M losses, the M candidate gradients of the N-1th layer in the t-th iteration can be obtained accordingly, and the M candidate gradients of the N-1th layer in the t-th iteration are directly averaged to obtain the gradient of the N-1th layer in the t-th iteration, and the gradient of the N-1th layer in the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the N-1th layer in the t-th iteration, ..., finally, based on the M losses, the M candidate gradients of the 1st layer in the t-th iteration can be obtained accordingly, and the M candidate gradients of the 1st layer in the t-th iteration are directly averaged to obtain the gradient of the 1st layer in the t-th iteration, and the gradient of the 1st layer in the t-th iteration is immediately normalized based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the 1st layer in the t-th iteration.

[0137] In order to better understand the model training method provided by the embodiment of the present application, the method provided by the embodiment of the present application is compared with the method provided by the related art 1 below. The comparison results are shown in Figures 5 and 6 (Figure 5 is a schematic diagram of the comparison results provided by the embodiment of the present application, and Figure 6 is another schematic diagram of the comparison results provided by the embodiment of the present application). In the embodiment of the present application, the MGGN in the t-1th iteration is used to complete the gradient normalization in the tth iteration, and the related art 1 uses the GGN in the tth iteration to complete the gradient normalization in the tth iteration. Based on Figures 5 and 6, it can be seen that the degree of fit between the MGGN in the t-1th iteration and the GGN in the tth iteration is very high, and the loss curves obtained by the two methods are also almost overlapping. It can be seen that the two methods have a very high degree of equivalent matching.

[0138] In the method provided by the embodiment of the present application, gradient normalization and backpropagation are performed simultaneously. As shown in Figure 7 (Figure 7 is a schematic diagram of an application example of the model training method provided by the embodiment of the present application), assume that in a certain iteration of the model, when backpropagation reaches a certain layer in the model, GPU0 can obtain the gradient G0 of the layer, GPU1 can obtain the gradient G1 of the layer, GPU2 can obtain the gradient G2 of the layer, and GPU3 can obtain the gradient G3 of the layer. After these four GPUs communicate with each other and obtain the four gradients G0, G1, G2, and G3 of the layer, they can all average G0, G1, G2, and G3 and divide them by the MGGN in the previous iteration to obtain the normalized gradient of the layer. Then, the four GPUs can perform similar operations on the layer above the layer until the entire backpropagation stage and gradient normalization stage are completed. Then, the MGGN in the iteration can be updated, and the normalized gradients of each layer can be used to complete the parameter update of each layer. At this point, the iteration is completed and the next iteration can be entered.

[0139] In addition, the method provided in the embodiment of the present application can be compared with the method provided in the related technology 2. The comparison results are shown in Figure 8 (Figure 8 is another schematic diagram of the comparison structure provided in the embodiment of the present application). Both have the purpose of training the BERT model. Based on Figure 8, it can be seen that compared with the loss curve of the method provided in the related technology 2, the loss curve of the method provided in the embodiment of the present application can significantly accelerate the loss convergence, and when achieving the same accuracy, the method provided in the embodiment of the present application can save about 25% of the number of iterations.

[0140] In an embodiment of the present application, in the t-th iteration of the model to be trained, after obtaining the gradient of the N-th layer of the model to be trained in the t-th iteration, the gradient of the N-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-th layer in the t-th iteration. Similarly, after obtaining the gradient of the N-1-th layer in the t-th iteration, the gradient of the N-1-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-1-th layer in the t-th iteration, ..., after obtaining the gradient of the 1st layer in the t-th iteration, the gradient of the 1st layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the 1st layer in the t-th iteration. In this way, the parameters of layers 1 through N can be updated based on the normalized gradients from layer 1 to layer N at iteration t. At this point, iteration t of the model to be trained is complete, and the model can proceed to iteration t+1. In the aforementioned process, the MGGN at iteration t-1 can be derived from the gradients from layer 1 to layer N at iteration t-1. That is, the MGGN at iteration t-1 has already been obtained at iteration t-1 of the model to be trained. Therefore, after entering iteration t of the model to be trained, once the gradients of any layer in the model to be trained at iteration t are obtained, the MGGN at iteration t-1 can be immediately used to normalize the gradients of that layer at iteration t, thereby obtaining the normalized gradients of that layer at iteration t-1. This allows backpropagation and gradient normalization to be performed simultaneously at iteration t of the model to be trained, effectively shortening the tail time of iteration t of the model to be trained, thereby reducing the total time cost required for the entire training process of the model to be trained.

[0141] The above is a detailed description of the model training method provided by the embodiment of the present application. The following is a brief introduction to the information processing method provided by the embodiment of the present application. As shown in Figure 9 (Figure 9 is a flow chart of the information processing method provided by the embodiment of the present application), as shown in Figure 9, the method is implemented by the target model obtained by training in the embodiment shown in Figure 4. The method includes:

[0142] 901. Obtain information to be processed.

[0143] 902. Input the information to be processed into the target model so as to process the information through the target model, thereby obtaining a processing result of the information.

[0144] For example, suppose the target model is a trained BERT model. After a question is input into the BERT model, the BERT model can output the answer to the question.

[0145] For example, suppose the target model is a trained BERT model. After a prompt is input into the BERT model, the BERT model can output the emotion belonging to the prompt.

[0146] For example, assuming that the target model is a trained BERT model, after inputting an incomplete text into the BERT model, the BERT model can output the supplementary part of the incomplete text, and so on.

[0147] In addition, the target model trained by the model training method provided in the embodiment of the present application and the model trained by the method of the second related technology can be fine-tuned and tested on the four downstream tasks of SQUAD, MRPC, SST-2, and MNLI, and the test results are compared, as shown in Table 1:

[0148] Table 1

[0149] Based on Table 1, it can be seen that the model trained in the embodiment of the present application has better accuracy, that is, it has better performance in various downstream tasks.

[0150] The above is a detailed description of the model training method and information processing method provided in the embodiment of the present application. The following is an introduction to the model training device and information processing device provided in the embodiment of the present application. Figure 10 is a structural diagram of the model training device provided in the embodiment of the present application. As shown in Figure 10, the device includes:

[0151] A first acquisition module 1001 is configured to acquire, in the t-th iteration of the model to be trained, the gradient of the i-th layer of the model to be trained in the t-th iteration based on the training data input to the model to be trained, where t≥2, i=1, ..., N, and N≥1;

[0152] a normalization module 1002 for normalizing the gradient of the i-th layer at the t-th iteration based on the MGGN at the t-1-th iteration to obtain the normalized gradient of the i-th layer at the t-th iteration, wherein the MGGN at the t-1-th iteration is obtained based on the gradient of the N-th layer of the model to be trained at the t-1-th iteration;

[0153] The updating module 1003 is used to update the parameters of the i-th layer based on the normalized gradient of the i-th layer in the t-th iteration to enter the t+1-th iteration of the model to be trained.

[0154] In an embodiment of the present application, in the t-th iteration of the model to be trained, after obtaining the gradient of the N-th layer of the model to be trained in the t-th iteration, the gradient of the N-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-th layer in the t-th iteration. Similarly, after obtaining the gradient of the N-1-th layer in the t-th iteration, the gradient of the N-1-th layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the N-1-th layer in the t-th iteration, ..., after obtaining the gradient of the 1st layer in the t-th iteration, the gradient of the 1st layer in the t-th iteration can be immediately normalized based on the MGGN in the t-1-th iteration, thereby obtaining the normalized gradient of the 1st layer in the t-th iteration. In this way, the parameters of layers 1 to N can be updated based on the normalized gradients from layer 1 to layer N at iteration t. At this point, iteration t of the model to be trained is complete, and the model can proceed to iteration t+1. In the aforementioned process, the MGGN at iteration t-1 can be derived from the gradients from layer 1 to layer N at iteration t-1. In other words, the MGGN at iteration t-1 has already been obtained at iteration t-1 of the model to be trained. Therefore, after entering iteration t of the model to be trained, once the gradients of a layer in the model to be trained at iteration t are obtained, the MGGN at iteration t-1 can be immediately used to normalize the gradients of that layer at iteration t, thereby obtaining the normalized gradients of that layer at iteration t-1. This allows backpropagation and gradient normalization to be performed simultaneously in iteration t of the model to be trained, effectively shortening the tail time of iteration t of the model to be trained, thereby reducing the total time cost required for the entire training process of the model to be trained.

[0155] In a possible implementation, the apparatus further includes: a second acquisition module, configured to acquire the MGGN in the t-th iteration based on the gradient of the N layer in the t-th iteration.

[0156] In one possible implementation, the second acquisition module is used to: perform a first calculation on the gradient of the N layer in the t-th iteration to obtain the global gradient norm GGN in the t-th iteration; and perform a second calculation on the MGGN in the t-1-th iteration and the GGN in the t-th iteration to obtain the MGGN in the t-th iteration.

[0157] In a possible implementation, the first calculation includes at least one of the following: a power calculation, a sum calculation, and a square root calculation.

[0158] In one possible implementation, the second calculation is a weighted sum calculation.

[0159] In one possible implementation, the first acquisition module 1001 is used to: obtain M unprocessed gradients of the i-th layer of the to-be-trained model in the t-th iteration based on M training data input to the to-be-trained model, where M ≥ 2; and perform averaging calculation on the M unprocessed gradients to obtain the gradient of the i-th layer in the t-th iteration.

[0160] In one possible implementation, the first acquisition module 1001 is further used to obtain the gradient of the N layer in the first iteration based on the training data input into the model to be trained in the first iteration of the model to be trained; the second acquisition module is used to obtain the GGN in the first iteration based on the gradient of the N layer in the first iteration, and the GGN in the first iteration serves as the MGGN in the first iteration; the normalization module 1002 is further used to normalize the gradient of the N layer in the first iteration based on the GGN in the first iteration, and obtain the normalized gradient of the N layer in the first iteration; the update module 1003 is further used to update the parameters of the N layer based on the normalized gradient of the N layer in the first iteration, so as to enter the second iteration of the model to be trained.

[0161] FIG11 is a schematic diagram of the structure of an information processing device provided in an embodiment of the present application. As shown in FIG11 , the device includes a target model obtained by training according to the embodiment shown in FIG10 . The device includes:

[0162] The acquisition module 1101 is used to acquire information to be processed.

[0163] The processing module 1102 is used to input the information to be processed into the target model so as to process the information through the target model and obtain the processing result of the information.

[0164] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0165] The embodiment of the present application also relates to an execution device, and Figure 12 is a structural diagram of the execution device provided by the embodiment of the present application. As shown in Figure 12, the execution device 1200 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which are not limited here. Among them, the information processing device described in the embodiment corresponding to Figure 11 can be deployed on the execution device 1200 to implement the function of information processing in the embodiment corresponding to Figure 9. Specifically, the execution device 1200 includes: a receiver 1201, a transmitter 1202, a processor 1203 and a memory 1204 (wherein the number of processors 1203 in the execution device 1200 can be one or more, and Figure 12 takes one processor as an example), wherein the processor 1203 may include an application processor 12031 and a communication processor 12032. In some embodiments of the present application, the receiver 1201, the transmitter 1202, the processor 1203 and the memory 1204 can be connected via a bus or other means.

[0166] The memory 1204 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1203. A portion of the memory 1204 may also include non-volatile random access memory (NVRAM). The memory 1204 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0167] Processor 1203 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0168] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1203. Processor 1203 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1203. The above processor 1203 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1203 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1204, and processor 1203 reads the information in memory 1204 and, in conjunction with its hardware, completes the steps of the above method.

[0169] Receiver 1201 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1202 can be used to output digital or character information through the first interface. Transmitter 1202 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1202 can also include a display device such as a display screen.

[0170] In an embodiment of the present application, in one case, the processor 1203 is used to obtain the processing result of the information through the target model in the embodiment corresponding to Figure 9.

[0171] The present application also relates to a training device. FIG13 is a schematic diagram of the structure of the training device provided by the present application. As shown in FIG13 , the training device 1300 is implemented by one or more servers. The training device 1300 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1313 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) storing application programs 1342 or data 1344. The memory 1332 and storage medium 1330 may be temporary storage or permanent storage. The program stored in the storage medium 1330 may include one or more modules (not shown), each module may include a series of instruction operations in the training device. Furthermore, the CPU 1313 may be configured to communicate with the storage medium 1330 to execute a series of instruction operations in the storage medium 1330 on the training device 1300.

[0172] The training device 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1358; or, one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0173] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 4 to obtain the target model.

[0174] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the computer storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0175] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.

[0176] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0177] Specifically, see Figure 14 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip may be a neural network processor (NPU) 1400. NPU 1400 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.

[0178] In some implementations, the arithmetic circuit 1403 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional systolic array. The arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0179] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.

[0180] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 through the Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 through the DMAC.

[0181] BIU stands for Bus Interface Unit, i.e., bus interface unit 1413 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1409 .

[0182] The bus interface unit 1413 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0183] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.

[0184] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.

[0185] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the predicted label plane extracted by the convolution layer, or another example is a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in subsequent layers in a neural network.

[0186] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0187] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0188] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0189] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0190] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0191] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0192] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A model training method, characterized in that: The method comprises: In the t-th iteration of the model to be trained, based on the training data input into the model to be trained, the gradient of the i-th layer of the model to be trained in the t-th iteration is obtained, t≥2, i=1,...,N, N≥1; Based on the moving average MGGN of the global gradient norm in the t-1th iteration, the gradient of the i-th layer in the t-th iteration is normalized to obtain the normalized gradient of the i-th layer in the t-th iteration, and the MGGN in the t-1th iteration is obtained based on the gradient of the N layer of the model to be trained in the t-1th iteration; Based on the normalized gradient of the i-th layer in the t-th iteration, the parameters of the i-th layer are updated to enter the t+1-th iteration of the model to be trained.

2. The method according to claim 1, characterized in that The method further comprises: Based on the gradient of the N layers in the t-th iteration, an MGGN in the t-th iteration is obtained.

3. The method according to claim 2, characterized in that The obtaining the MGGN in the t-th iteration based on the gradient of the N layer in the t-th iteration includes: Performing a first calculation on the gradient of the N layer in the t-th iteration to obtain a global gradient norm GGN in the t-th iteration; A second calculation is performed on the MGGN in the t-1th iteration and the GGN in the tth iteration to obtain the MGGN in the tth iteration.

4. The method according to claim 3, characterized in that The first calculation includes at least one of the following: exponentiation calculation, summation calculation and square root calculation.

5. The method according to claim 3 or 4, characterized in that: The second calculation is a weighted sum calculation.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the gradient of the i-th layer of the model to be trained in the t-th iteration based on the training data input into the model to be trained comprises: Based on the M training data input to the model to be trained, obtaining M to-be-processed gradients of the i-th layer of the model to be trained in the t-th iteration, where M≥2; The M to-be-processed gradients are averaged to obtain the gradient of the i-th layer in the t-th iteration.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In a first iteration of the model to be trained, based on the training data input into the model to be trained, obtaining the gradient of the N layers in the first iteration; Based on the gradient of the N layer in the first iteration, obtaining the GGN in the first iteration, and using the GGN in the first iteration as the MGGN in the first iteration; Based on the GGN in the first iteration, normalizing the gradients of the N layers in the first iteration to obtain the normalized gradients of the N layers in the first iteration; Based on the normalized gradient of the N layer in the first iteration, the parameters of the N layer are updated to enter the second iteration of the model to be trained.

8. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire, in the t-th iteration of the model to be trained, the gradient of the i-th layer of the model to be trained in the t-th iteration based on the training data input into the model to be trained, t≥2, i=1,...,N, N≥1; A normalization module, configured to normalize the gradient of the i-th layer in the t-th iteration based on the MGGN in the t-1-th iteration to obtain the normalized gradient of the i-th layer in the t-th iteration, wherein the MGGN in the t-1-th iteration is obtained based on the gradient of the N layer of the model to be trained in the t-1-th iteration; An updating module is used to update the parameters of the i-th layer based on the normalized gradient of the i-th layer in the t-th iteration to enter the t+1-th iteration of the model to be trained.

9. The device according to claim 8, characterized in that The device also includes: The second acquisition module is used to acquire the MGGN in the t-th iteration based on the gradient of the N layer in the t-th iteration.

10. The device according to claim 9, characterized in that The second acquisition module is used to: Performing a first calculation on the gradient of the N layer in the t-th iteration to obtain a global gradient norm GGN in the t-th iteration; A second calculation is performed on the MGGN in the t-1th iteration and the GGN in the tth iteration to obtain the MGGN in the tth iteration.

11. The device according to claim 10, characterized in that The first calculation includes at least one of the following: exponentiation calculation, summation calculation and square root calculation.

12. The device according to claim 10 or 11, characterized in that The second calculation is a weighted sum calculation.

13. The device according to any one of claims 8 to 12, characterized in that The first acquisition module is used to: Based on the M training data input to the model to be trained, obtaining M to-be-processed gradients of the i-th layer of the model to be trained in the t-th iteration, where M≥2; The M to-be-processed gradients are averaged to obtain the gradient of the i-th layer in the t-th iteration.

14. The device according to any one of claims 8 to 13, characterized in that The first acquisition module is further used to, in the first iteration of the model to be trained, acquire the gradient of the N layers in the first iteration based on the training data input into the model to be trained; A second acquisition module is configured to acquire a GGN in the first iteration based on the gradient of the N layer in the first iteration, and the GGN in the first iteration is used as the MGGN in the first iteration; The normalization module is further used to normalize the gradient of the N layer in the first iteration based on the GGN in the first iteration to obtain the normalized gradient of the N layer in the first iteration; The updating module is also used to update the parameters of the N layers based on the normalized gradients of the N layers in the first iteration to enter the second iteration of the model to be trained.

15. A model training device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the model training device performs the method according to any one of claims 1 to 7.

16. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Thermal power plant condenser vacuum degree prediction method based on multi-layer LSTM

    CN111353631A

  • Neural network model training method and apparatus, and computing device

    CN115860100A

  • Model training method and related equipment thereof

    CN117422122A