Method, device and equipment for updating network model, and non-volatile storage medium

Abstract

The present application discloses a method, apparatus, and device for updating a network model, as well as a non-volatile storage medium. The method comprises: obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution comprises: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; selecting multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain gradient estimation results; and updating the deep neural network model using the gradient estimation results. The present application solves the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus and device for updating a network model, and a non-volatile storage medium. Background Art

[0002] With the continuous advancement of artificial intelligence (AI) technology, distributed training of large datasets has become an important tool for AI computing platforms. In large-scale distributed training scenarios, communication efficiency between AI computing machines is low due to limited bandwidth. Furthermore, if communication latency between AI computing machines is excessive, the overall training efficiency and multi-machine scalability of the training system cannot be guaranteed.

[0003] Therefore, it is necessary to optimize the communication efficiency between artificial intelligence computing machines through technical means, reduce the communication overhead between artificial intelligence computing machines, and thus improve the throughput of the training system while ensuring that the accuracy is not reduced as much as possible.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, and device, as well as a non-volatile storage medium for updating a network model, to at least solve the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0006] According to one aspect of an embodiment of the present application, a method for updating a network model is provided, comprising: obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution comprises gradient distribution information of some or all layers in the deep neural network model; sampling gradients generated by the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; selecting multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; and updating the deep neural network model using the gradient estimation result.

[0007] According to another aspect of an embodiment of the present application, a method for updating a network model is also provided, including: obtaining a global gradient distribution of a deep neural network model from a server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; displaying the gradient position information on a client to select multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; displaying the gradient estimation result on the client, and updating the deep neural network model using the gradient estimation result.

[0008] According to another aspect of an embodiment of the present application, a method for updating a network model is also provided, including: obtaining a global gradient distribution of a deep neural network model stored locally on a server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; sending the gradient position information to a client, and displaying the gradient position information on the client, so as to select multiple target gradients on the client using the gradient position information; receiving the multiple target gradients returned by the client, and performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; sending the gradient estimation result to the client to display the gradient estimation result on the client, and updating the deep neural network model through the gradient estimation result.

[0009] According to another aspect of an embodiment of the present application, a device for updating a network model is also provided, including: an acquisition module for acquiring a global gradient distribution of a deep neural network model, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; a sampling module for sampling the above-mentioned deep neural network model in a distributed training scenario based on the above-mentioned global gradient distribution to obtain gradient position information; a selection module for selecting multiple target gradients using the above-mentioned gradient position information; an estimation module for performing gradient estimation based on the above-mentioned multiple target gradients to obtain a gradient estimation result; and an update module for updating the above-mentioned deep neural network model according to the above-mentioned gradient estimation result.

[0010] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned methods for updating the network model.

[0011] According to another aspect of an embodiment of the present application, a device for updating a network model is also provided, including: a processor; and a memory, connected to the above-mentioned processor, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining a global gradient distribution of a deep neural network model, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; sampling the above-mentioned deep neural network model in a distributed training scenario based on the above-mentioned global gradient distribution to obtain gradient position information; selecting multiple target gradients using the above-mentioned gradient position information; performing gradient estimation based on the above-mentioned multiple target gradients to obtain a gradient estimation result; and updating the above-mentioned deep neural network model through the above-mentioned gradient estimation result.

[0012] In an embodiment of the present application, the global gradient distribution of the deep neural network model is obtained, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; based on the above-mentioned global gradient distribution, the above-mentioned deep neural network model is sampled in a distributed training scenario to obtain gradient position information; multiple target gradients are selected using the above-mentioned gradient position information; gradient estimation is performed based on the above-mentioned multiple target gradients to obtain gradient estimation results; and the above-mentioned deep neural network model is updated according to the above-mentioned gradient estimation results.

[0013] It is easy to note that in the embodiments of the present application, since some layers in the deep neural network model have fewer parameters, gradient sampling can be omitted for layers with parameters less than a predetermined threshold, and all gradient data for that layer can be directly communicated. By selecting the target gradient for gradient estimation and using the gradient estimation result to update the parameters in the deep neural network model, the gradient compression method based on gradient sampling can significantly reduce the amount of communication, lower the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0014] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for updating a network model according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of a method for updating a network model according to an embodiment of the present application;

[0018] Figure 3 This is a schematic diagram of a scenario for updating a network model in a distributed training scenario according to an embodiment of the present application;

[0019] Figure 4 is a flowchart of another method for updating a network model according to an embodiment of the present application;

[0020] Figure 5 is a flowchart of another method for updating a network model according to an embodiment of the present application;

[0021] Figure 6 is a structural diagram of an apparatus for updating a network model according to an embodiment of the present application;

[0022] Figure 7 is a structural diagram of a device for updating a network model according to an embodiment of the present application;

[0023] Figure 8 This is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0027] Scalability: Specifically refers to the performance improvement achieved by increasing the number of training throughput and machines in a multi-machine distributed synchronous training scenario.

[0028] Gradient compression: refers to a method to reduce the amount of gradient synchronization communication in multi-machine distributed training scenarios.

[0029] All-Reduce: A multi-machine communication method.

[0030] SGD (Stochastic Gradient Descent): Stochastic gradient descent is a commonly used optimization method in the field of machine learning. Stochastic gradient descent randomly selects a data point in the direction of fastest descent for calculation, rather than scanning the entire training set, thereby speeding up the iteration speed.

[0031] Distributed SGD optimization: refers to a distributed optimization method commonly used in the field of deep learning. It communicates gradients and updates algorithm model parameters through synchronous SGD or asynchronous SGD. Among them, the synchronous SGD method can guarantee convergence performance, but in bandwidth-constrained scenarios, the scalability of multi-machine training is greatly restricted due to the large communication volume. The asynchronous SGD method has poor convergence performance.

[0032] Gradient sparsification is a gradient compression method that selects a subset of gradients for communication. It achieves a compression rate of 0.1% on the ImageNet-1K classification task without compromising accuracy. During training, gradients are selected layer by layer based on their magnitude for communication, with the remaining gradients accumulated locally. Computing the top-k gradients layer by layer incurs additional computational overhead, and the need to communicate the location and value of the top-k gradients limits scalability across multiple machines.

[0033] Example 1

[0034] According to an embodiment of the present application, an embodiment of a method for updating a network model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiment provided in Example 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for updating a network model is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for updating the network model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for updating the network model. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0040] Under the above operating environment, this application provides Figure 2 An embodiment of a method for updating a network model is shown. Figure 2 is a flow chart of a method for updating a network model according to an embodiment of the present application. Figure 2 As shown, the method for updating the network model includes:

[0041] Step S202: obtaining a global gradient distribution of the deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model;

[0042] Step S204: sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information;

[0043] Step S206, selecting multiple target gradients using the above gradient position information;

[0044] Step S208, performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result;

[0045] Step S210: Update the deep neural network model using the gradient estimation result.

[0046] In an embodiment of the present application, the global gradient distribution of the deep neural network model is obtained, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; based on the above-mentioned global gradient distribution, the above-mentioned deep neural network model is sampled in a distributed training scenario to obtain gradient position information; multiple target gradients are selected using the above-mentioned gradient position information; gradient estimation is performed based on the above-mentioned multiple target gradients to obtain gradient estimation results; and the above-mentioned deep neural network model is updated according to the above-mentioned gradient estimation results.

[0047] It is easy to notice that, in the embodiment of the present application, since the number of parameters of certain layers in the deep neural network model is small, gradient sampling can be omitted for layers with parameter amounts less than a predetermined threshold. By selecting the target gradient for gradient estimation, and using the gradient estimation result to update the parameters in the deep neural network model, the gradient compression method based on gradient sampling can significantly reduce the amount of communication, reduce the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0048] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0049] It should be noted that the method embodiment for updating the network model provided in this application can be applied to, but is not limited to, multi-machine distributed synchronous training scenarios. By reducing the gradient synchronization communication volume through gradient compression, the communication overhead between machines can be reduced, thereby improving the throughput of the training system.

[0050] In an optional embodiment, sampling the above-mentioned deep neural network model in the above-mentioned distributed training scenario based on the above-mentioned global gradient distribution to obtain the above-mentioned gradient position information includes:

[0051] Step S302: Based on the global gradient distribution at the same sampling position of each training scene in the distributed training scene, gradient sampling is performed on the deep neural network model to obtain the gradient position information.

[0052] In an optional embodiment, gradient sampling is performed on the layers in the above-mentioned deep neural network model whose parameter quantity is greater than a preset threshold.

[0053] In an optional embodiment, the above method further includes:

[0054] Step S402 : Buffering the gradient position information of the plurality of target gradients into a communication queue.

[0055] In an optional embodiment, the following Figure 3 Taking the gradient sampling scheme in the distributed training scenario (N workers) shown as an example, the embodiment of the method for updating the network model provided in this application is schematically described:

[0056] As an optional embodiment, the training system can maintain additional memory space for storing gradient distribution information for some or all layers of the deep neural network model. Alternatively, the training system can communicate with the system through, but is not limited to, an asynchronous process, iterating and synchronizing the gradient distribution information every predetermined period k to determine the gradient distribution information for some or all layers.

[0057] like Figure 3 In the scenario diagram of updating the network model shown in , that is, Bernoulli sampling is performed based on the global gradient distribution to obtain the corresponding sampled gradient position information. It is necessary to ensure that the sampling position of each layer worker is consistent, and gradient sampling is performed on the layers with parameter quantities greater than a preset threshold in the above-mentioned deep neural network model to obtain gradient position information; based on the gradient position information obtained after gradient sampling of the above-mentioned deep neural network model, multiple target gradients are selected, and the gradient position information of the target gradients is added to the communication queue for caching.

[0058] In an optional embodiment, performing gradient estimation based on the multiple target gradients to obtain the gradient estimation result includes:

[0059] Step S502: When the gradient position information of the plurality of target gradients cached in the communication queue reaches a cache threshold, a communication operation of distributed training is performed on the plurality of target gradients stored in the communication queue to obtain a communication operation result.

[0060] Step S504, calculating the average value of the above communication operation results to obtain a mean result;

[0061] Step S506 , performing a division operation on the above-mentioned mean value result and the sampling probability value corresponding to the same sampling position to obtain the above-mentioned gradient estimation result.

[0062] Optionally, the communication operation of the above distributed training is all-reduce communication.

[0063] In the above optional embodiment, when the gradient position information added to the communication queue cache reaches the cache threshold, all-reduce communication is performed on the multiple target gradients stored in the above communication queue; until all target gradients have completed communication, the gradient position information obtained through the above communication operation results is averaged, and the average result is divided by the sampling probability value corresponding to the corresponding position to obtain the final gradient estimation result, and the parameters in the deep neural network model are updated according to the corresponding optimization method.

[0064] It should be noted that in the embodiments of the present application, since the number of parameters in some layers of the deep neural network is small, gradient sampling is usually not performed on layers with a parameter value less than a certain threshold (for example, a parameter value less than 2048), and gradients are obtained through all-reduce communication for parameter update.

[0065] The gradient compression method based on gradient sampling provided in the embodiments of the present application can optimize the problem of large communication volume in distributed SGD, especially synchronous SGD optimization methods, and greatly improve bandwidth utilization; gradient sampling and all-reduce communication schemes are introduced to address the problems of high computational overhead and poor multi-machine scalability in gradient sparsification, significantly reducing computational overhead and improving multi-machine communication efficiency.

[0066] In an optional embodiment, obtaining the global gradient distribution of the deep neural network model includes:

[0067] Step S602 : communicating through a training process or an asynchronous process, iteratively synchronizing the global gradient distribution at intervals of a preset period.

[0068] Optionally, communication can be performed through a training process or an asynchronous process, and the gradient distribution information can be iterated and synchronized once every preset period k to obtain the gradient distribution information of some or all of the above layers.

[0069] In an optional embodiment, iteratively synchronizing the global gradient distribution at intervals of the preset period includes:

[0070] Step S702: Obtaining the gradient corresponding to each training scenario in the distributed training scenario at intervals of the preset period;

[0071] Step S704 , performing mean calculation using the gradient corresponding to each training scene to obtain the above-mentioned global gradient distribution.

[0072] As an optional embodiment, Figure 3 As shown, assuming that the k0th time needs to synchronize the gradient distribution information, the gradient corresponding to worker-i is expressed as After synchronization, the global gradient distribution information can be expressed as

[0073] The gradient compression method based on gradient sampling in a distributed training scenario provided by the embodiments of the present application greatly reduces the communication volume and significantly reduces the computational overhead compared to other gradient sparsification methods such as DGC. In addition, in the gradient compression method, only all-reduce communication is sampled for gradient synchronization, which can effectively improve the scalability of multi-machine training and only uses the native API interface of the open source software PyTorch, without involving any code modification of the underlying software.

[0074] In an optional embodiment, the above method further includes:

[0075] Step S802 : updating the gradient distribution corresponding to at least one layer in the global gradient distribution based on a preset compression rate.

[0076] In the embodiment of the present application, for the preset compression rate ρ, the gradient distribution corresponding to each layer is solved in, Represents the sampling probability vector corresponding to the lth layer.

[0077] In an optional embodiment, updating the gradient distribution corresponding to at least one layer in the global gradient distribution based on the preset compression rate includes:

[0078] Step S902, an acquisition step, obtaining the gradient and sampling probability vector corresponding to the current layer of the deep neural network model, and initializing the probability value and number of iterations at each dimensional position in the sampling probability vector based on the preset compression rate, the gradient corresponding to the current layer, and the dimension of the sampling probability vector;

[0079] Step S904, a processing step, selecting a dimension set, and calculating a target coefficient based on the dimension set, the preset compression rate, the dimension of the sampling probability vector, and the probability and value corresponding to the dimension set, calculating the product of the probability value of each dimension position in the sampling probability vector in the current iteration and the target coefficient, and determining the probability value of each dimension position in the sampling probability vector in the next iteration based on the product, until the target coefficient is less than or equal to a preset threshold or the maximum number of iterative calculations is reached, thereby obtaining a vector iteration result;

[0080] Step S906, an updating step, updating the sampling probability vector to the vector iteration result, and using the updated sampling probability vector to update the gradient distribution corresponding to at least one layer in the global gradient distribution;

[0081] Step S908 is a judgment step, which determines whether each layer of the above-mentioned deep neural network model has been processed. If not, return to the above-mentioned acquisition step; if yes, end the update process.

[0082] Optionally, the specific solution of the above sampling probability vector is as follows: Assume that the gradient corresponding to a certain layer is g∈R d , initially j = 0, let the element solution of the sampling probability vector p of this layer be: For all i∈[1, d], the following calculation is repeated until the target coefficient c≤1 (i.e., the preset threshold) or the maximum number of iterative calculations is reached, and the set dimension set is selected And calculate the target coefficient Calculate the sampling probability vector j=j+1; find the sampling probability vector p=p of this layer; j And update the memory space of the global gradient distribution.

[0083] As an optional embodiment, the communication volume can also be reduced by methods such as gradient quantization or gradient low-rank. Gradient quantization is orthogonal to the gradient compression method in the embodiment of the present application, so they can be used in combination; it should be noted here that as the number of layers of the deep neural network increases, the computational overhead of the gradient low-rank method also gradually increases, so the applicable scenarios of the gradient low-rank method are relatively limited.

[0084] In the above operating environment, this application also provides Figure 4 Another embodiment of a method for updating a network model is shown. Figure 4 is a flow chart of another method for updating a network model according to an embodiment of the present application. Figure 4 As shown, the method for updating the network model includes:

[0085] Step S1002: Obtaining a global gradient distribution of the deep neural network model from the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model;

[0086] Step S1004: sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information;

[0087] Step S1006: displaying the gradient position information on the client, so as to select a plurality of target gradients using the gradient position information;

[0088] Step S1008, performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result;

[0089] Step S1010: Display the gradient estimation result on the client, and update the deep neural network model according to the gradient estimation result.

[0090] In an embodiment of the present application, a global gradient distribution of a deep neural network model is obtained from a server through a client, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; based on the global gradient distribution, the deep neural network model is sampled in a distributed training scenario to obtain gradient position information; the gradient position information is displayed on the client to select multiple target gradients using the gradient position information; gradient estimation is performed based on the multiple target gradients to obtain a gradient estimation result; the gradient estimation result is displayed on the client, and the deep neural network model is updated according to the gradient estimation result.

[0091] It is easy to notice that, in the embodiment of the present application, since the number of parameters of certain layers in the deep neural network model in the server side is small, gradient sampling can be omitted for layers with parameter amounts less than a predetermined threshold. By using the gradient position information on the client to select multiple target gradients for gradient estimation, and using the gradient estimation results to update the parameters in the deep neural network model, the gradient compression method based on the gradient sampling provided in the embodiment of the present application can significantly reduce the communication volume, reduce the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0092] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0093] The above steps S1002 to S1010 provide an embodiment of a method for updating a network model from the perspective of SaaS (Software as a Service) service form and gradient information visualization. The execution subject of this embodiment is the client.

[0094] SaaS platforms (i.e., platforms that operate SaaS software) provide enterprises with all the necessary infrastructure and software and hardware platforms for information technology. Users can access information systems online without having to purchase hardware and software, build computer rooms, or hire IT personnel. As a software deployment model, SaaS applications are designed for network delivery, making them easy to host, deploy, and access online.

[0095] It should be noted that the method embodiment for updating the network model provided in this application can be applied to, but is not limited to, multi-machine distributed synchronous training scenarios. By reducing the gradient synchronization communication volume through gradient compression, the communication overhead between machines can be reduced, thereby improving the throughput of the training system.

[0096] As an optional embodiment, based on the global gradient distribution at the same sampling position of each training scene in the distributed training scene, gradient sampling is performed on the deep neural network model to obtain the gradient position information.

[0097] As an optional embodiment, the training system can maintain additional memory space for storing gradient distribution information for some or all layers of the deep neural network model. Alternatively, the training system can communicate with the system through, but is not limited to, an asynchronous process, iterating and synchronizing the gradient distribution information every predetermined period k to determine the gradient distribution information for some or all layers.

[0098] For example, by performing Bernoulli sampling based on the global gradient distribution to obtain the corresponding sampled gradient position information, it is necessary to ensure that the sampling position of each layer worker is consistent, and to perform gradient sampling on the layers in the above-mentioned deep neural network model whose parameter quantity is greater than a preset threshold to obtain gradient position information; based on the gradient position information obtained after gradient sampling of the above-mentioned deep neural network model, multiple target gradients are selected, and the gradient position information of the target gradients is added to the communication queue for caching.

[0099] In the above optional embodiment, when the gradient position information added to the communication queue cache reaches the cache threshold, all-reduce communication is performed on the multiple target gradients stored in the above communication queue; until all target gradients have completed communication, the gradient position information obtained through the above communication operation results is averaged, and the average result is divided by the sampling probability value corresponding to the corresponding position to obtain the final gradient estimation result, and the parameters in the deep neural network model are updated according to the corresponding optimization method.

[0100] It should be noted that in the embodiments of the present application, since the number of parameters in some layers of the deep neural network is small, gradient sampling is usually not performed on layers with a parameter value less than a certain threshold (for example, a parameter value less than 2048), and gradients are obtained through all-reduce communication for parameter update.

[0101] The gradient compression method based on gradient sampling provided in the embodiments of the present application can optimize the problem of large communication volume in distributed SGD, especially synchronous SGD optimization methods, and greatly improve bandwidth utilization; gradient sampling and all-reduce communication schemes are introduced to address the problems of high computational overhead and poor multi-machine scalability in gradient sparsification, significantly reducing computational overhead and improving multi-machine communication efficiency.

[0102] In the above operating environment, this application also provides Figure 5 Another embodiment of a method for updating a network model is shown. Figure 5is a flow chart of another method for updating a network model according to an embodiment of the present application. Figure 5 As shown, the method for updating the network model includes:

[0103] Step S1102: Obtain the global gradient distribution of the deep neural network model stored locally on the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model;

[0104] Step S1104: sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information;

[0105] Step S1106 , sending the gradient position information to the client, and displaying the gradient position information on the client, so that the client can select multiple target gradients using the gradient position information;

[0106] Step S1108: receiving the multiple target gradients returned by the client, and performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result;

[0107] Step S1110: Send the above-mentioned gradient estimation result to the above-mentioned client to display the above-mentioned gradient estimation result on the above-mentioned client, and update the above-mentioned deep neural network model according to the above-mentioned gradient estimation result.

[0108] In an embodiment of the present application, a global gradient distribution of a deep neural network model stored locally on the server is obtained through the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; based on the global gradient distribution, the deep neural network model is sampled in a distributed training scenario to obtain gradient position information; the gradient position information is sent to the client, and the gradient position information is displayed on the client, so as to select multiple target gradients on the client using the gradient position information; the multiple target gradients returned by the client are received, and gradient estimation is performed based on the multiple target gradients to obtain a gradient estimation result; the gradient estimation result is sent to the client to display the gradient estimation result on the client, and the deep neural network model is updated according to the gradient estimation result.

[0109] It is easy to notice that, in the embodiment of the present application, since the number of parameters of certain layers in the deep neural network model in the server side is small, gradient sampling can be omitted for layers with parameter amounts less than a predetermined threshold. By using the gradient position information on the client to select multiple target gradients for gradient estimation, and using the gradient estimation results to update the parameters in the deep neural network model, the gradient compression method based on the gradient sampling provided in the embodiment of the present application can significantly reduce the communication volume, reduce the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0110] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0111] The above steps S1102 to S1110 provide an embodiment of a method for updating a network model from the perspective of SaaS (Software as a Service) service form and gradient information visualization, and the execution subject of this embodiment is the server.

[0112] SaaS platforms (i.e., platforms that operate SaaS software) provide enterprises with all the necessary infrastructure and software and hardware platforms for information technology. Users can access information systems online without having to purchase hardware and software, build computer rooms, or hire IT personnel. As a software deployment model, SaaS applications are designed for network delivery, making them easy to host, deploy, and access online.

[0113] It should be noted that the method embodiment for updating the network model provided in this application can be applied to, but is not limited to, multi-machine distributed synchronous training scenarios. By reducing the gradient synchronization communication volume through gradient compression, the communication overhead between machines can be reduced, thereby improving the throughput of the training system.

[0114] As an optional embodiment, based on the global gradient distribution at the same sampling position of each training scene in the distributed training scene, gradient sampling is performed on the deep neural network model to obtain the gradient position information.

[0115] As an optional embodiment, the training system can maintain additional memory space for storing gradient distribution information for some or all layers of the deep neural network model. Alternatively, the training system can communicate with the system through, but is not limited to, an asynchronous process, iterating and synchronizing the gradient distribution information every predetermined period k to determine the gradient distribution information for some or all layers.

[0116] For example, by performing Bernoulli sampling based on the global gradient distribution to obtain the corresponding sampled gradient position information, it is necessary to ensure that the sampling position of each layer worker is consistent, and to perform gradient sampling on the layers in the above-mentioned deep neural network model whose parameter quantity is greater than a preset threshold to obtain gradient position information; based on the gradient position information obtained after gradient sampling of the above-mentioned deep neural network model, multiple target gradients are selected, and the gradient position information of the target gradients is added to the communication queue for caching.

[0117] In the above optional embodiment, when the gradient position information added to the communication queue cache reaches the cache threshold, all-reduce communication is performed on the multiple target gradients stored in the above communication queue; until all target gradients have completed communication, the gradient position information obtained through the above communication operation results is averaged, and the average result is divided by the sampling probability value corresponding to the corresponding position to obtain the final gradient estimation result, and the parameters in the deep neural network model are updated according to the corresponding optimization method.

[0118] It should be noted that in the embodiments of the present application, since the number of parameters in some layers of the deep neural network is small, gradient sampling is usually not performed on layers with a parameter value less than a certain threshold (for example, a parameter value less than 2048), and gradients are obtained through all-reduce communication for parameter update.

[0119] The gradient compression method based on gradient sampling provided in the embodiments of the present application can optimize the problem of large communication volume in distributed SGD, especially synchronous SGD optimization methods, and greatly improve bandwidth utilization; gradient sampling and all-reduce communication schemes are introduced to address the problems of high computational overhead and poor multi-machine scalability in gradient sparsification, significantly reducing computational overhead and improving multi-machine communication efficiency.

[0120] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a non-volatile storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above-mentioned methods of each embodiment of the present application.

[0122] Example 2

[0123] According to an embodiment of the present application, there is also provided an embodiment of a device for implementing the above-mentioned method for updating the network model. Figure 6 is a structural diagram of a device for updating a network model according to an embodiment of the present application, such as Figure 6 As shown, the apparatus includes: an acquisition module 400, a sampling module 402, a selection module 404, an estimation module 406 and an update module 408, wherein:

[0124] An acquisition module 400 is used to obtain the global gradient distribution of the deep neural network model, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; a sampling module 402 is used to sample the gradient data generated by the above-mentioned deep neural network model in a distributed training scenario based on the above-mentioned global gradient distribution to obtain gradient position information; a selection module 404 is used to select multiple target gradients using the above-mentioned gradient position information; an estimation module 406 is used to perform gradient estimation based on the above-mentioned multiple target gradients to obtain a gradient estimation result; and an update module 408 is used to update the above-mentioned deep neural network model according to the above-mentioned gradient estimation result.

[0125] It should be noted that the acquisition module 400, sampling module 402, selection module 404, estimation module 406, and update module 408 described above correspond to steps S202 to S210 in Example 1. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0126] It should also be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0127] Example 3

[0128] According to an embodiment of the present application, an embodiment of a device for updating a network model is also provided. The device for updating a network model can be any computing device in a computing device group. Figure 7 is a structural diagram of a device for updating a network model according to an embodiment of the present application, such as Figure 7 As shown, the device for updating the network model includes: a processor 500 and a memory 502, wherein:

[0129] A processor 500; and a memory 502, connected to the above-mentioned processor 500, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining a global gradient distribution of a deep neural network model, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; sampling the above-mentioned deep neural network model in a distributed training scenario based on the above-mentioned global gradient distribution to obtain gradient position information; selecting multiple target gradients using the above-mentioned gradient position information; performing gradient estimation based on the above-mentioned multiple target gradients to obtain a gradient estimation result; and updating the above-mentioned deep neural network model according to the above-mentioned gradient estimation result.

[0130] In an embodiment of the present application, the global gradient distribution of the deep neural network model is obtained, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; based on the above-mentioned global gradient distribution, the above-mentioned deep neural network model is sampled in a distributed training scenario to obtain gradient position information; multiple target gradients are selected using the above-mentioned gradient position information; gradient estimation is performed based on the above-mentioned multiple target gradients to obtain gradient estimation results; and the above-mentioned deep neural network model is updated according to the above-mentioned gradient estimation results.

[0131] It is easy to notice that, in the embodiment of the present application, since the number of parameters of certain layers in the deep neural network model is small, gradient sampling can be omitted for layers with parameter amounts less than a predetermined threshold. By selecting the target gradient for gradient estimation, and using the gradient estimation result to update the parameters in the deep neural network model, the gradient compression method based on gradient sampling can significantly reduce the amount of communication, reduce the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0132] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0133] It should also be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0134] Example 4

[0135] According to an embodiment of the present application, a computer terminal embodiment is also provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0136] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0137] In this embodiment, the computer terminal can execute the program code of the following steps in the method for updating the network model: obtaining the global gradient distribution of the deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; selecting multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; and updating the deep neural network model using the gradient estimation result.

[0138] Optionally, Figure 8 is a structural block diagram of a computer terminal according to an embodiment of the present application, such as Figure 8 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 602 , a memory 604 , and a peripheral interface 606 .

[0139] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for updating the network model in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned method for updating the network model. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the global gradient distribution of the deep neural network model, wherein the above-mentioned global gradient distribution includes: gradient distribution information of some or all layers in the above-mentioned deep neural network model; based on the above-mentioned global gradient distribution, the above-mentioned deep neural network model is sampled in a distributed training scenario to obtain gradient position information; multiple target gradients are selected using the above-mentioned gradient position information; gradient estimation is performed based on the above-mentioned multiple target gradients to obtain gradient estimation results; and the above-mentioned deep neural network model is updated according to the above-mentioned gradient estimation results.

[0141] Optionally, the processor may also execute the program code of the following steps: performing gradient sampling on the deep neural network model based on the global gradient distribution at the same sampling position of each training scene in the distributed training scene to obtain the gradient position information.

[0142] Optionally, the processor may further execute the program code of the following steps: performing gradient sampling on the layers in the deep neural network model whose parameter quantity is greater than a preset threshold.

[0143] Optionally, the processor may further execute program code of the following steps: caching the gradient position information of the plurality of target gradients in a communication queue.

[0144] Optionally, the processor may also execute the program code of the following steps: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; calculating the average value of the communication operation results to obtain a mean result; and dividing the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

[0145] Optionally, the processor may further execute program code of the following steps: communicating through a training process or an asynchronous process, and iteratively synchronizing the global gradient distribution at intervals of a preset period.

[0146] Optionally, the processor may also execute the program code of the following steps: obtaining the gradient corresponding to each training scenario in the distributed training scenario at intervals of the preset period; performing mean calculation using the gradient corresponding to each training scenario to obtain the global gradient distribution.

[0147] Optionally, the processor may further execute program code of the following steps: updating the gradient distribution corresponding to at least one layer in the global gradient distribution based on a preset compression rate.

[0148] Optionally, the processor may also execute the following program code: an acquisition step, acquiring the gradient and sampling probability vector corresponding to the current layer of the deep neural network model, initializing the probability value and number of iterations for each dimensional position in the sampling probability vector based on the preset compression rate, the gradient corresponding to the current layer, and the dimension of the sampling probability vector; a processing step, selecting a dimension set, and calculating a target coefficient based on the dimension set, the preset compression rate, the dimension of the sampling probability vector, and the probability and value corresponding to the dimension set, calculating the product of the probability value of each dimensional position in the sampling probability vector in the current iteration and the target coefficient, and determining the probability value of each dimensional position in the sampling probability vector in the next iteration based on the product result, until the target coefficient is less than or equal to a preset threshold or reaches the maximum number of iterative calculations, thereby obtaining a vector iteration result; an updating step, updating the sampling probability vector to the vector iteration result, and using the updated sampling probability vector to update the gradient distribution corresponding to at least one layer in the global gradient distribution; a judging step, judging whether each layer of the deep neural network model has been processed. If not, returning to the acquisition step; if yes, terminating the update process.

[0149] An embodiment of the present application provides a scheme for updating a network model. This scheme involves obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution includes gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; selecting multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain gradient estimation results; and updating the deep neural network model using the gradient estimation results.

[0150] It is easy to notice that, in the embodiment of the present application, since the number of parameters of certain layers in the deep neural network model is small, gradient sampling can be omitted for layers with parameter amounts less than a predetermined threshold. By selecting the target gradient for gradient estimation, and using the gradient estimation result to update the parameters in the deep neural network model, the gradient compression method based on gradient sampling can significantly reduce the amount of communication, reduce the communication overhead between artificial intelligence computing machines, and improve the communication efficiency between artificial intelligence computing machines and the scalability of multi-machine training.

[0151] Therefore, the embodiment of the present application achieves the purpose of improving the communication efficiency between artificial intelligence computing machines, thereby achieving the technical effect of improving the throughput of the training system and the scalability of multi-machine training, and further solving the technical problem of low communication efficiency between artificial intelligence computing machines in related technologies.

[0152] Optionally, the processor may also execute the program code of the following steps: obtaining the global gradient distribution of the deep neural network model from the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; displaying the gradient position information on the client to select multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; displaying the gradient estimation result on the client, and updating the deep neural network model through the gradient estimation result.

[0153] Optionally, the processor may also execute the program code of the following steps: obtaining the global gradient distribution of the deep neural network model stored locally on the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; sending the gradient position information to the client, and displaying the gradient position information on the client, so as to select multiple target gradients on the client using the gradient position information; receiving the multiple target gradients returned by the client, and performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; sending the gradient estimation result to the client to display the gradient estimation result on the client, and updating the deep neural network model through the gradient estimation result.

[0154] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 8 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 8 Different configurations shown.

[0155] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable non-volatile storage medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0156] Example 5

[0157] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the method for updating the network model provided in the first embodiment.

[0158] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0159] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; selecting multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; and updating the deep neural network model using the gradient estimation result.

[0160] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: based on the above-mentioned global gradient distribution at the same sampling position of each training scene in the above-mentioned distributed training scene, gradient sampling is performed on the above-mentioned deep neural network model to obtain the above-mentioned gradient position information.

[0161] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: performing gradient sampling on layers in the above-mentioned deep neural network model whose parameter quantity is greater than a preset threshold.

[0162] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: caching the gradient position information of the plurality of target gradients in a communication queue.

[0163] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: when the gradient position information of the above-mentioned multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the above-mentioned multiple target gradients stored in the above-mentioned communication queue to obtain a communication operation result; calculating the average of the above-mentioned communication operation results to obtain a mean result; and performing a division operation on the above-mentioned mean result and the sampling probability value corresponding to the same sampling position to obtain the above-mentioned gradient estimation result.

[0164] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: communicating through a training process or an asynchronous process, and iteratively synchronizing the above-mentioned global gradient distribution at intervals of a preset period.

[0165] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining the gradient corresponding to each training scenario in the above-mentioned distributed training scenario at intervals of the above-mentioned preset period; performing mean calculation using the gradient corresponding to each training scenario to obtain the above-mentioned global gradient distribution.

[0166] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: updating the gradient distribution corresponding to at least one layer in the global gradient distribution based on a preset compression rate.

[0167] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: an acquisition step, obtaining the gradient and sampling probability vector corresponding to the current layer of the above-mentioned deep neural network model, and initializing the probability value and the number of iterations at each dimensional position in the above-mentioned sampling probability vector based on the above-mentioned preset compression rate, the gradient corresponding to the above-mentioned current layer, and the dimension of the above-mentioned sampling probability vector; a processing step, selecting a dimension set, and calculating the target coefficient based on the above-mentioned dimension set, the above-mentioned preset compression rate, the dimension of the above-mentioned sampling probability vector, and the probability and value corresponding to the above-mentioned dimension set, and calculating the probability of each dimensional position in the above-mentioned sampling probability vector. The product of the probability value in the current iteration and the above-mentioned target coefficient is obtained, and the probability value of each dimensional position in the above-mentioned sampling probability vector in the next iteration is determined based on the above-mentioned product result, until the above-mentioned target coefficient is less than or equal to the preset threshold or the maximum number of iterative calculations is reached, and the vector iteration result is obtained; an updating step, updating the above-mentioned sampling probability vector to the above-mentioned vector iteration result, and using the updated sampling probability vector to update the gradient distribution corresponding to at least one layer in the above-mentioned global gradient distribution; a judgment step, judging whether each layer of the above-mentioned deep neural network model has been processed, if not, returning to the above-mentioned acquisition step, if yes, ending the update process.

[0168] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining the global gradient distribution of the deep neural network model from the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; displaying the gradient position information on the client to select multiple target gradients using the gradient position information; performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; displaying the gradient estimation result on the client, and updating the deep neural network model through the gradient estimation result.

[0169] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining the global gradient distribution of the deep neural network model stored locally on the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; sending the gradient position information to the client and displaying the gradient position information on the client to select multiple target gradients on the client using the gradient position information; receiving the multiple target gradients returned by the client, and performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; sending the gradient estimation result to the client to display the gradient estimation result on the client, and updating the deep neural network model through the gradient estimation result.

[0170] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0171] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0173] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0176] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for updating a network model, characterized in that: include: Obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; Sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; Selecting a plurality of target gradients using the gradient position information; Performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; Updating the deep neural network model using the gradient estimation result; The step of sampling the deep neural network model in the distributed training scenario based on the global gradient distribution to obtain the gradient position information includes: performing gradient sampling on the deep neural network model at the same sampling position of each training scenario in the distributed training scenario based on the global gradient distribution to obtain the gradient position information; Among them, performing gradient estimation based on the multiple target gradients to obtain the gradient estimation result includes: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; averaging the communication operation results to obtain a mean result; and dividing the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

2. The method according to claim 1, characterized in that Gradient sampling is performed on layers in the deep neural network model whose parameter quantity is greater than a preset threshold.

3. The method according to claim 1, characterized in that The method further comprises: The gradient position information of the plurality of target gradients is cached in a communication queue.

4. The method according to claim 1, wherein Obtaining the global gradient distribution of the deep neural network model includes: Communicating through a training process or an asynchronous process, iteratively synchronizing the global gradient distribution at intervals of a preset period.

5. The method according to claim 4, characterized in that Iteratively synchronizing the global gradient distribution at intervals of the preset period includes: At intervals of the preset period, obtaining the gradient corresponding to each training scenario in the distributed training scenario; The global gradient distribution is obtained by performing mean calculation using the gradient corresponding to each training scene.

6. The method according to claim 5, characterized in that The method further comprises: The gradient distribution corresponding to at least one layer in the global gradient distribution is updated based on a preset compression rate.

7. The method according to claim 6, characterized in that Updating the gradient distribution corresponding to at least one layer in the global gradient distribution based on the preset compression rate includes: An acquisition step of acquiring the gradient and sampling probability vector corresponding to the current layer of the deep neural network model, and initializing the probability value and number of iterations at each dimensional position in the sampling probability vector based on the preset compression rate, the gradient corresponding to the current layer, and the dimension of the sampling probability vector; A processing step of selecting a dimension set, and calculating a target coefficient based on the dimension set, the preset compression rate, the dimension of the sampling probability vector, and the probability and value corresponding to the dimension set, calculating a product of the probability value of each dimension position in the sampling probability vector in the current iteration and the target coefficient, and determining a probability value of each dimension position in the sampling probability vector in the next iteration based on the product result, until the target coefficient is less than or equal to a preset threshold or the maximum number of iterative calculations is reached, thereby obtaining a vector iteration result; An updating step of updating the sampling probability vector to the vector iteration result, and using the updated sampling probability vector to update the gradient distribution corresponding to at least one layer in the global gradient distribution; A judgment step is performed to determine whether each layer of the deep neural network model has been processed. If not, the process returns to the acquisition step; if so, the update process ends.

8. A method for updating a network model, characterized in that: include: Obtaining a global gradient distribution of a deep neural network model from a server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; Sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; Displaying the gradient position information on the client to select multiple target gradients using the gradient position information; Performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; Displaying the gradient estimation result on the client, and updating the deep neural network model according to the gradient estimation result; The step of sampling the deep neural network model in the distributed training scenario based on the global gradient distribution to obtain the gradient position information includes: performing gradient sampling on the deep neural network model at the same sampling position of each training scenario in the distributed training scenario based on the global gradient distribution to obtain the gradient position information; Among them, performing gradient estimation based on the multiple target gradients to obtain the gradient estimation result includes: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; averaging the communication operation results to obtain a mean result; and dividing the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

9. A method for updating a network model, characterized in that: include: Obtaining a global gradient distribution of a deep neural network model stored locally on the server, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; Sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; Sending the gradient position information to a client and displaying the gradient position information on the client, so that the client can select multiple target gradients using the gradient position information; receiving the multiple target gradients returned by the client, and performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; Sending the gradient estimation result to the client to display the gradient estimation result on the client, and updating the deep neural network model according to the gradient estimation result; The step of sampling the deep neural network model in the distributed training scenario based on the global gradient distribution to obtain the gradient position information includes: performing gradient sampling on the deep neural network model at the same sampling position of each training scenario in the distributed training scenario based on the global gradient distribution to obtain the gradient position information; Among them, performing gradient estimation based on the multiple target gradients to obtain the gradient estimation result includes: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; averaging the communication operation results to obtain a mean result; and dividing the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

10. A device for updating a network model, characterized in that: include: An acquisition module is used to obtain a global gradient distribution of a deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; A sampling module, configured to sample the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; a selection module, configured to select a plurality of target gradients using the gradient position information; An estimation module, configured to perform gradient estimation based on the multiple target gradients to obtain a gradient estimation result; An updating module, configured to update the deep neural network model using the gradient estimation result; The sampling module is further configured to: perform gradient sampling on the deep neural network model based on the same sampling position of the global gradient distribution in each training scene in the distributed training scene to obtain the gradient position information; The estimation module is further configured to: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, perform a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; average the communication operation results to obtain a mean result; and divide the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the method for updating the network model according to any one of claims 1 to 9.

12. A device for updating a network model, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Obtaining a global gradient distribution of a deep neural network model, wherein the global gradient distribution includes: gradient distribution information of some or all layers in the deep neural network model; Sampling the deep neural network model in a distributed training scenario based on the global gradient distribution to obtain gradient position information; Selecting a plurality of target gradients using the gradient position information; Performing gradient estimation based on the multiple target gradients to obtain a gradient estimation result; Updating the deep neural network model using the gradient estimation result; The step of sampling the deep neural network model in the distributed training scenario based on the global gradient distribution to obtain the gradient position information includes: performing gradient sampling on the deep neural network model at the same sampling position of each training scenario in the distributed training scenario based on the global gradient distribution to obtain the gradient position information; Among them, performing gradient estimation based on the multiple target gradients to obtain the gradient estimation result includes: when the gradient position information of the multiple target gradients cached in the communication queue reaches a cache threshold, performing a communication operation of distributed training on the multiple target gradients stored in the communication queue to obtain a communication operation result; averaging the communication operation results to obtain a mean result; and dividing the mean result by the sampling probability value corresponding to the same sampling position to obtain the gradient estimation result.

Images