Computing method and device of neural network model, electronic equipment and program product

Abstract

This application relates to the field of neural network technology, and provides a method, apparatus, electronic device, and program product for calculating a neural network model. The method for calculating the neural network model includes multiple computational layers, each layer comprising neurons. The method includes: dividing the weight data of the neural network model into weight packets corresponding to each neuron, and storing these weight packets in a storage unit; during the calculation of the current computational layer, extracting feature information from the intermediate calculation results of the current computational layer, and predicting the set of neurons to be activated in the next computational layer based on the feature information; before entering the next computational layer, reading the weight packets corresponding to each neuron in the set of neurons from the storage unit for use in the calculation of the next computational layer. Embodiments of this application can improve the inference efficiency of neural network models.

Description

Technical Field

[0001] This application belongs to the field of neural network technology, and in particular relates to a method, apparatus, electronic device and program product for calculating a neural network model. Background Technology

[0002] During the inference process of a neural network model, the computation of each computational layer depends on the reading of weight data. There are generally two ways to read weight data: one is to preload all weight data at once, which increases the time consumption due to loading a large amount of weight data that will not be used, and also wastes storage bandwidth; the other is a serial computation method, in which the loading and inference of the weights of the next layer are started after the current computational layer has finished its calculation. This method requires waiting for the weights to be loaded after each calculation. Neither of these methods affects the inference efficiency of the neural network model. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and program product for calculating a neural network model, which can improve the inference efficiency of the neural network model.

[0004] The first aspect of this application provides a method for calculating a neural network model. The neural network model includes multiple computational layers, each computational layer including neurons. The method for calculating the neural network model includes: dividing the weight data of the neural network model into weight packets corresponding to each neuron, and storing the weight packets corresponding to each neuron in a storage unit; extracting feature information of the intermediate computation results of the current computational layer during computation, and predicting the set of neurons to be activated in the next computational layer based on the feature information; and reading the weight packets corresponding to each neuron in the set of neurons from the storage unit before entering the next computational layer for computation in the next computational layer.

[0005] In some implementations of the first aspect, the weight data of the neural network model is divided into weight packets corresponding to each neuron, including: performing structural analysis on the feedforward network layer of the neural network model to identify the gated weight matrix, the up-projection weight matrix, and the down-projection weight matrix in each computational layer; and, taking a single neuron as a unit, aggregating the weight data, quantized weight values, and scale information of the neuron in the gated weight matrix, the up-projection weight matrix, and the down-projection weight matrix to obtain the weight packet corresponding to the neuron.

[0006] In some embodiments of the first aspect, storing the weight packets corresponding to each neuron in a storage unit includes: writing multiple weight packets sequentially into the storage unit in a preset order so that each weight packet is physically stored consecutively; assigning a unique physical address identifier to each weight packet and establishing a mapping relationship between the neuron index and the physical address.

[0007] In some embodiments of the first aspect, reading the weight packets corresponding to each neuron in the neuron set from the storage unit includes: querying the target physical address corresponding to each neuron index in the neuron index list according to the neuron index list and mapping relationship corresponding to the neuron set; generating a read request based on the target physical address and adding it to the request queue; and initiating a read operation to the storage unit based on the request queue to load the weight packets corresponding to each neuron in the neuron set into a preset cache area for computation.

[0008] In some embodiments of the first aspect, the computation method of the neural network model further includes: assigning an initial priority value to each read request in the request queue; adjusting the priority of each read request in the request queue in real time, and sorting the read requests in the request queue according to the adjusted priority; the priority adjustment rules include one or more of the following: adjusting the priority of read requests of the preceding computation layer to be higher than the priority of read requests of the following computation layer according to the execution order of each computation layer of the neural network model; raising the priority of the read request to the highest level when the computation layer corresponding to the read request is idle due to waiting for weight packets; increasing the priority of each read request according to the waiting time in the request queue; and adjusting the priority according to the order in which the read requests arrive in the request queue when multiple read requests have the same priority value.

[0009] In some embodiments of the first aspect, reading the weight packet corresponding to each neuron in the neuron set from the storage unit includes: prioritizing the loading of the gating weights in the weight packet; after completing the gating calculation, determining whether the corresponding neuron is activated based on the gating value; if the neuron is activated, continuing to load the remaining weight data in the weight packet; if the neuron is not activated, terminating the subsequent loading operation of the weight packet.

[0010] In some embodiments of the first aspect, the computation method for the neural network model further includes: after the neural network model completes one forward propagation, recording the matching between the predicted set of neurons and the actual set of activated neurons; adjusting the prediction model used to generate the set of neurons based on the matching, and / or adjusting the priority of the read request.

[0011] A second aspect of this application provides a computing device for a neural network model. The neural network model includes multiple computing layers, each computing layer including neurons. The computing device for the neural network model includes: a storage unit, used to divide the weight data of the neural network model into weight packets corresponding to each neuron, and store the weight packets corresponding to each neuron in the storage unit; a prediction unit, used to extract feature information of the intermediate computing results of the current computing layer during the computing process, and predict the set of neurons to be activated in the next computing layer based on the feature information; and a pre-fetch unit, used to read the weight packets corresponding to each neuron in the set of neurons from the storage unit before entering the next computing layer, for use in the computing of the next computing layer.

[0012] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a computation method for a neural network model as described in any of the first aspects.

[0013] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the calculation method for the aforementioned neural network model.

[0014] The fifth aspect of this application provides a computer program product that, when run, causes the calculation method of the neural network model described above to be executed.

[0015] In the embodiments of this application, by extracting the feature information of intermediate calculation results during the current calculation layer, the set of neurons that may be activated in the next layer can be predicted in advance, and the weight packets corresponding to these neurons can be read before entering the next layer. This allows the calculation to be performed directly using the weight data that is already in place when entering the next layer of neurons without waiting for the weights to be loaded, thus eliminating the idle waiting time of the computing unit. At the same time, only the weight data of the predicted activated neurons are loaded, avoiding the time cost and storage bandwidth waste caused by loading all weight data. Therefore, the inference efficiency of the neural network model is significantly improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a schematic diagram illustrating the implementation flow of a calculation method for a neural network model provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a computing device for a neural network model provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are protected by this application.

[0019] There are generally two existing methods for reading weight data: one is to preload all weight data at once, which increases time consumption due to loading a large amount of unused weight data and wastes storage bandwidth; the other is a serial calculation method, where the next layer's weight loading and inference are started only after the current layer has finished calculating, which requires waiting for weight loading after each calculation. Neither method affects the inference efficiency of the neural network model. Therefore, this application proposes a calculation method for neural network models that can accurately read weight data in advance, thereby improving the inference efficiency of the neural network model.

[0020] To illustrate the technical solution of this application, specific embodiments are described below.

[0021] Figure 1 This illustration shows a schematic flowchart of a computation method for a neural network model provided in an embodiment of this application. This method can be applied to electronic devices. The electronic devices can be intelligent devices such as robots, unmanned vehicles, and computers; this application does not limit the scope of the application.

[0022] In some scenarios, electronic devices may be configured with a Neural Processing Unit (NPU), and the computational methods of neural network models can be used to perform calculations on the neural network models. In other embodiments, the neural network model may also run on a Central Processing Unit (CPU) or other processors, and this application does not impose any restrictions on this.

[0023] A neural network model is a computational model consisting of multiple computational layers connected according to a certain topology. It is used to extract and transform features from input data and ultimately output prediction results. Neural network models include, but are not limited to, large-scale language models, deep feedforward networks, recurrent neural networks, the Transformer architecture, and its variants.

[0024] Computational layers are the basic computational units that make up a neural network model. Each layer receives the feature data output from the previous layer, performs operations such as weight matrix operations and activation function transformations, and then outputs it to the next layer. Computational layers include, but are not limited to: feedforward network layers (fully connected layers), convolutional layers, normalization layers, gated recurrent layers (such as LSTM and GRU), attention layers, etc.

[0025] Each computational layer contains multiple neurons, and the activation states of different neurons may differ during the computation process. A neuron is the smallest computational granularity within a computational layer.

[0026] Specifically, the calculation method of the above neural network model may include the following steps S101 to S103.

[0027] Step S101: Divide the weight data of the neural network model into weight packets corresponding to each neuron, and store the weight packets corresponding to each neuron in the storage unit.

[0028] In this context, weight data refers to the set of parameters obtained after the neural network model has been trained and used to calculate the output of neurons. Each neuron corresponds to a set of weight data, which is used to perform weighted summation and nonlinear transformation on the input features. For example, in a feedforward network layer, a neuron corresponds to one dimension of the output feature vector, and its weight data includes gating weights, upward projection weights, downward projection weights, etc.

[0029] In the embodiments of this application, taking a single neuron as a unit, all weight parameters required by that neuron at different computational stages are extracted from the total weight data and aggregated into an independent weight package. A weight package is a data unit formed by aggregating and encapsulating all weight data corresponding to a neuron. These weight packages are written into storage units according to a preset storage layout, enabling subsequent inference processes to use neurons as the basic scheduling unit and read the weight package corresponding to the neuron that needs to be computed.

[0030] The storage unit is a hardware medium or its logical partition used for persistent or temporary storage of weighted data, including but not limited to memory, external storage, on-chip cache, disk, etc.

[0031] Step S102: During the calculation of the current calculation layer, extract the feature information of the intermediate calculation results of the current calculation layer, and predict the set of neurons to be activated in the next calculation layer based on the feature information.

[0032] In this context, the current computational layer refers to the computational layer that is currently performing computation during the neural network's inference process. The inference of the neural network model proceeds layer by layer from the input layer to the output layer in the order of computation; the computational layer currently performing computation is called the current computational layer. Intermediate computation results refer to the partial output data generated by the current computational layer during the forward computation process, which has not yet undergone activation function processing or has not yet fully completed all computations at that layer. These data reflect the intermediate state of the current computational layer after transforming the input data.

[0033] Since the computation of neurons in the next computational layer depends on the output value and weights of the current computational layer, feature information reflecting the final output distribution can be extracted from the intermediate computation results when the current layer is not yet fully completed (but has obtained partial data). Feature information is data characteristics extracted from intermediate computation results that reflect the output distribution or activation patterns of the current computational layer. These characteristics can be statistics, numerical distributions, sparsity patterns, or other quantifiable attributes of the intermediate computation results. Based on this feature information, combined with the structural relationships of neurons in the next computational layer, it is possible to predict which neurons in the next computational layer are likely to be activated in subsequent computations. The set of activated neurons refers to the set of neurons in the next computational layer that, based on the output results or feature information of the current computational layer, are determined to participate in the computation (i.e., output values ​​are non-zero or exceed a threshold).

[0034] In this way, predictions are made before the current computation layer has finished, so that the prediction results can be obtained before the current computation layer has finished. This provides a time window for loading the weights required for the next computation layer in advance, avoiding the waiting delay caused by starting weight loading only after entering the next computation layer.

[0035] Step S103: Before entering the next computational layer, read the weight packets corresponding to each neuron in the neuron set from the storage unit for computation in the next computational layer.

[0036] The period before entering the next computational layer refers to the time when the current computational layer's computation is not yet fully completed but is about to end. After obtaining the set of neurons activated in the next computational layer, the weight packets of these neurons need to be loaded before the next computational layer actually starts its computation. Since the weight packets are stored independently in storage units according to the neuron granularity, the storage location of the corresponding weight packet can be queried based on the neuron, and a read operation can be initiated. Utilizing the remaining execution time of the current computational layer as a hidden delay, the transmission process of the weight packets overlaps with the tail computation of the current computational layer. When the control flow enters the next computational layer, the required weight packets are already in the area accessible to the computational layer, and the computational layer can directly call these weight data for the computation of the next computational layer.

[0037] In the embodiments of this application, by extracting the feature information of intermediate calculation results during the current calculation layer, the set of neurons that may be activated in the next layer can be predicted in advance, and the weight packets corresponding to these neurons can be read before entering the next layer. This allows the calculation to be performed directly using the weight data that is already in place when entering the next layer of neurons without waiting for the weights to be loaded, thus eliminating the idle waiting time of the computing unit. At the same time, only the weight data of the predicted activated neurons are loaded, avoiding the time cost and storage bandwidth waste caused by loading all weight data. Therefore, the inference efficiency of the neural network model is significantly improved.

[0038] In some embodiments of this application, dividing the weight data of the neural network model into weight packets corresponding to each neuron may include: performing structural analysis on the feedforward network layer of the neural network model to identify the gated weight matrix, the up-projection weight matrix, and the down-projection weight matrix in each computational layer; and, taking a single neuron as a unit, aggregating the weight data, quantized weight values, and scale information of that neuron in the gated weight matrix, the up-projection weight matrix, and the down-projection weight matrix to obtain the weight packet corresponding to that neuron.

[0039] In a neural network model, the feedforward layer is a computational layer where information propagates unidirectionally. The output of each neuron depends only on the input of the previous layer, with no feedback connections. Each neuron in the feedforward layer corresponds to three sets of weight parameters: gating weights (controlling whether information passes through), upward projection weights (mapping the input to a higher-dimensional space), and downward projection weights (mapping the high-dimensional features back to the output). By analyzing the structure of the feedforward layer, the parameters and connections of each computational layer can be analyzed, identifying whether a gating mechanism exists and the corresponding three weight matrices (gating weight matrix, upward projection weight matrix, and downward projection weight matrix). Using a single neuron as an index, the three weight matrices are traversed to find the weight vector corresponding to that neuron in each matrix. These weight vectors are extracted, along with the quantized weight values ​​corresponding to the neuron (i.e., the low-bit integer representation obtained after quantizing the original floating-point weight parameters) and the scaling information required to restore the original values ​​(the scaling factor and zero-point offset required to restore the quantized integer values ​​to the original floating-point values), and all are collected into a single data packet. This data packet is the weight packet for that neuron.

[0040] In this way, through the aggregation operation, all the parameters of the same neuron that were originally scattered in multiple weight matrices are stored in a single weight package, decoupling the data dependencies between neurons and allowing subsequent reading of only the weight package corresponding to a specific neuron without having to read the entire weight matrix.

[0041] In some embodiments of this application, storing the weight packets corresponding to each neuron in a storage unit may include: writing multiple weight packets into the storage unit in a preset order so that each weight packet is physically stored consecutively; assigning a unique physical address identifier to each weight packet and establishing a mapping relationship between the neuron index and the physical address.

[0042] After all the weight data for each neuron has been encapsulated into independent weight packets, these weight packets have independent access capabilities in physical storage. At this point, multiple weight packets can be written sequentially to storage units in a preset order. The preset order refers to the pre-determined arrangement of the weight packets when written to storage units, which can be set based on the natural order of the neuron indexes, the execution order of the model's computation layers, or other design rules. The weight packets in this preset order are written sequentially to the address space of the storage units, ensuring that the bytes within each weight packet are stored contiguously and that different weight packets are closely adjacent to each other. In this way, the weight packets are arranged linearly throughout the entire storage area.

[0043] After writing is complete, a physical address identifier can be assigned to each weight packet, and a mapping relationship can be established from the neuron index to the physical address identifier of the weight packet. The neuron index identifies the neuron, while the physical storage location can be the start or end address of the weight packet and the data length. The mapping relationship can be organized into a data structure that allows for fast querying and deployed on the storage controller or its accessible area to enable direct addressing from the neuron to the physical storage location.

[0044] In some embodiments of this application, reading the weight packets corresponding to each neuron in the neuron set from the storage unit may include: querying the target physical address corresponding to each neuron index in the neuron index list and mapping relationship according to the neuron set; generating a read request based on the target physical address and adding it to the request queue; and initiating a read operation to the storage unit based on the request queue to load the weight packets corresponding to each neuron in the neuron set into a preset cache area for computation.

[0045] Specifically, as the current computation layer nears completion, feature information is extracted from the intermediate computation results and input into a pre-trained prediction model. The prediction model outputs a set of neurons that may be activated in the next computation layer. This set of neurons is output as a list of neuron indices, containing the index of each neuron. The predicted list of neuron indices is then sent to the prefetch execution unit. The prefetch execution unit calls the address resolution module based on the neuron indices in the list. Based on the aforementioned mapping relationship, the address resolution module queries the target physical address corresponding to each neuron index and organizes these target physical addresses into a sequence of read requests, adding them to the request queue. The target physical address is the specific location information of the weight packet corresponding to the neuron in the storage unit.

[0046] The prefetching execution unit internally includes a request management module to maintain the request queue. This queue, acting as a buffer pool, is managed by the storage controller or a built-in scheduling unit. The scheduling unit sequentially retrieves read requests from the queue and initiates the actual read operation to the storage unit. Specifically, it reads all the data from the weight packet at the target physical address via the bus or memory access interface and transfers the data to a preset cache area. This preset cache area is specifically reserved for prefetched data; the computation unit can directly access this cache area to obtain weight data and perform calculations when executing the next computational layer.

[0047] In some embodiments of this application, the calculation method of the neural network model may further include: assigning an initial priority value to each read request in the request queue, adjusting the priority of each read request in the request queue in real time, and sorting each read request in the request queue according to the adjusted priority.

[0048] Specifically, in the request queue, each read request corresponds to a weight packet that needs to be loaded from the storage unit. When multiple read requests exist in the queue simultaneously, using a fixed first-in-first-out order may cause some critical requests to be delayed due to being at the back of the queue, affecting the startup time of subsequent calculations. Therefore, this step first assigns an initial priority value to each read request. The initial priority value can be a uniform default value, or it can be adjusted according to the dependency order of the weight packets in the calculation. In particular, when the request queue maintains both read requests and activation value calculation requests, the initial priority value of the read request can be higher than the initial priority value of the activation value calculation request.

[0049] During the maintenance of the request queue, priorities can be adjusted periodically, and the read requests in the request queue can be sorted according to the adjusted priorities.

[0050] The rules for priority adjustment may include one or more of the following: 1. According to the execution order of each computation layer in the neural network model, adjust the priority of the read request of the previous computation layer to be higher than the priority of the read request of the subsequent computation layer; for example, the read request of Layer N has a higher priority than the read request of Layer N+1, so as to ensure that the dependency between the prefetch operation and the computation operation is not broken.

[0051] 2. When the computation layer corresponding to the read request is in an idle state due to waiting for the weight packet, the priority of the read request is raised to the highest level. That is, the read request required by the computation layer in the waiting state is raised to the highest priority.

[0052] 3. Prioritize each read request based on its waiting time in the request queue. For example, as the waiting time of a read request in the request queue increases, its priority value is dynamically increased by 1 for each clock cycle, thus preventing low-priority read requests from waiting for a long time.

[0053] 4. When multiple read requests have the same priority value, adjust the priority according to the order in which the read requests arrive in the request queue to avoid long-term starvation of the read requests that arrive earlier.

[0054] In some embodiments of this application, reading the weight packets corresponding to each neuron in the neuron set from the storage unit can be done inside the neural network processor without the intervention of the host CPU.

[0055] Thus, the steps of loading the weight packets corresponding to each neuron in the neuron set into the preset cache area and the computation steps of the current computation layer can be executed in parallel. Specifically, the main processor can write the organizational structure information of the weight packets (including the storage address, size, and dependencies of each weight packet) into the metadata register file inside the neural network processor. The neural network processor can then autonomously perform read operations. By reading the information in the metadata register file and combining it with the computation progress of the current computation layer (which can be collected in real time by hardware performance counters), it predicts the weight packets that the next computation layer needs to prefetch based on feature information when the current computation layer is about to finish, and generates a read request. The neural network processor and memory can have a dedicated Direct Memory Access (DMA) channel, and the scheduler for this channel is located inside the neural network processor. When a read request is generated, the DMA scheduler directly initiates data transfer without the participation of the main processor. The main processor can handle other tasks (such as performing computation operations in the current computation layer, preparing the next batch of input data, handling operating system interrupts, etc.) without participating in the internal read process of the neural network processor, thereby reducing overall inference latency through parallel execution. The loaded weight data can be marked with a status for subsequent fast access.

[0056] In some embodiments of this application, reading the weight packet corresponding to each neuron in the neuron set from the storage unit may include: first loading the gate weights in the weight packet; after completing the gate calculation, determining whether the corresponding neuron is activated based on the gate value; if the neuron is activated, then continuing to load the remaining weight data in the weight packet; if the neuron is not activated, then terminating the subsequent loading operation of the weight packet.

[0057] Specifically, gating weights from the gating weight matrix can be loaded first from the neuron weight package. Gating computation refers to the process of performing gating operations and obtaining gating values ​​using the gating weights and the input features of the current layer. After completing the gating computation, the gating output can be used to determine whether the neuron is activated.

[0058] The output of the gating calculation is a continuous value between 0 and 1 (i.e., the gating value). The gating value determines the proportion of information retained.

[0059] Specifically, in practical applications, not all neurons need to be activated at every time step or for every input sample. For example, in a gated recurrent neural network, when the output value of the forget gate is below a threshold (e.g., 0.1), it means that the historical information at that time step does not need to be retained, and the corresponding hidden state can be directly discarded without participating in the calculation of the current time step; the neuron is not activated. Similarly, in a sparse neural network, when the weight of a neuron is pruned to 0 and its input activation value is below a threshold, the output of that neuron will definitely be 0, the result of the gating calculation can be directly determined as invalid, the activation function calculation is skipped, and the neuron is not activated. Furthermore, in the attention mechanism, when the attention weight of an attention head is below a threshold (e.g., 0.01), it means that the attention head is not important to the current input, its corresponding value vector can be directly ignored without participating in the weighted summation, and the neuron is not activated.

[0060] Therefore, inside the neural network processor, the output of the gating unit is connected to a threshold comparator, which compares the gating value with a preset threshold (e.g., 0.5). When the gating value is lower than the preset threshold, the corresponding neuron output can be determined to be invalid (i.e., approximately 0), and an invalid flag is output, indicating that no further activation function calculation is needed. Thus, when the gating value is lower than the preset threshold, it can be determined that the neuron is not activated, terminating the subsequent loading operation of the weight packet, thereby saving computational and data access resources. If the gating value is greater than or equal to the preset threshold, the neuron is considered activated, and the remaining weight data in the weight packet continues to be loaded, thereby avoiding invalid data access and saving computational resources.

[0061] In some embodiments of this application, the computation method of the neural network model may further include: after the neural network model completes one forward propagation, recording the matching between the predicted set of neurons and the actual set of activated neurons, adjusting the prediction model used to generate the set of neurons according to the matching, and / or adjusting the priority of the read request.

[0062] Specifically, after each forward propagation (i.e., the complete inference process of the neural network model for a single input sample), the actual neuron activation status of all computational layers in that inference has been obtained. By comparing the predicted set of neurons with the actual set of activated neurons, the matching status can be calculated, for example, how many neurons were correctly predicted and which ones, and how many neurons were incorrectly predicted and which ones.

[0063] The matching results reflect the accuracy of the current prediction model. If the matching results are poor, the prediction model needs to be adjusted to reduce invalid or missed prefetches and output more accurate neuron predictions in the next inference process. Adjustments can include, but are not limited to: updating parameters in the prediction model, changing the weights of feature extraction, and modifying the activation threshold.

[0064] At the same time, matching data can also be used to adjust the priority strategy for read requests. For example, if some neurons are frequently predicted to be activated but are not actually activated, the initial priority of their corresponding read requests can be reduced, and storage bandwidth can be allocated to more reliable prediction results. Conversely, if some neurons are frequently predicted not to be activated but are actually activated, their initial priority can be increased, or their waiting time threshold in the request queue can be shortened, so that limited data loading resources can be prioritized for more reliable or more urgent requests.

[0065] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders.

[0066] like Figure 2 The diagram shown is a structural schematic of a computing device 200 for a neural network model provided in an embodiment of this application. The computing device 200 for the neural network model is configured on an electronic device.

[0067] Specifically, the computing device 200 for the neural network model may include: Storage unit 201 is used to divide the weight data of the neural network model into weight packets corresponding to each neuron, and store the weight packets corresponding to each neuron in the storage unit. The prediction unit 202 is used to extract the feature information of the intermediate calculation results of the current calculation layer during the calculation of the current calculation layer, and predict the set of neurons to be activated in the next calculation layer based on the feature information. The prefetching unit 203 is used to read the weight packets corresponding to each neuron in the neuron set from the storage unit before entering the next computation layer, so as to use them for computation in the next computation layer.

[0068] In some embodiments of this application, the storage unit 201 is used to: perform structural analysis on the feedforward network layer of the neural network model, identify the gate weight matrix, the up-projection weight matrix and the down-projection weight matrix in each computational layer; and, taking a single neuron as a unit, aggregate the weight data, quantized weight values ​​and scale information of the neuron in the gate weight matrix, the up-projection weight matrix and the down-projection weight matrix to obtain the weight packet corresponding to the neuron.

[0069] In some embodiments of this application, the storage unit 201 is used to: write multiple weight packets into the storage unit in a preset order so that each weight packet is physically stored continuously; assign a unique physical address identifier to each weight packet and establish a mapping relationship between the neuron index and the physical address.

[0070] In some embodiments of this application, the pre-reading unit 203 is used to: query the target physical address corresponding to each neuron index in the neuron index list according to the neuron index list and mapping relationship corresponding to the neuron set; generate a read request according to the target physical address and add it to the request queue; and initiate a read operation to the storage unit based on the request queue to load the weight packets corresponding to each neuron in the neuron set into the preset cache area for calculation.

[0071] In some embodiments of this application, the pre-reading unit 203 is further configured to: assign an initial priority value to each read request in the request queue; adjust the priority of each read request in the request queue in real time, and sort the read requests in the request queue according to the adjusted priority; the priority adjustment rules include one or more of the following: adjusting the priority of the read request of the preceding computation layer to be higher than the priority of the read request of the following computation layer according to the execution order of each computation layer of the neural network model; raising the priority of the read request to the highest when the computation layer corresponding to the read request is idle due to waiting for weight packets; increasing the priority of each read request according to the waiting time in the request queue; when multiple read requests have the same priority value, adjusting the priority according to the order in which the read requests arrive in the request queue.

[0072] In some embodiments of this application, the pre-reading unit 203 is further configured to: preferentially load the gate weights in the weight package; after completing the gate calculation, determine whether the corresponding neuron is activated based on the gate value; if the neuron is activated, continue loading the remaining weight data in the weight package; if the neuron is not activated, terminate the subsequent loading operation of the weight package.

[0073] In some embodiments of this application, the computing device 200 for the neural network model further includes a feedback adjustment unit for: recording the matching between the predicted set of neurons and the actual set of activated neurons after the neural network model completes one forward propagation; adjusting the prediction model used to generate the set of neurons according to the matching; and / or adjusting the priority of the read request.

[0074] It should be noted that, for the sake of convenience and brevity, the specific working process of the computing device 200 of the above neural network model can be found in [reference needed]. Figure 1 The corresponding process of the method will not be described in detail here.

[0075] like Figure 3 The diagram shown is a schematic representation of an electronic device according to an embodiment of this application. The electronic device 3 may include: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a neural network model calculation program. When the processor 30 executes the computer program 32, it implements the steps in the above-described embodiments of the calculation methods for various neural network models, for example... Figure 1 The steps S101 to S103 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the storage unit 201, prediction unit 202, and pre-fetch unit 203 shown are illustrated.

[0076] The computer program can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in an electronic device.

[0077] For example, a computer program can be divided into: a storage unit, used to divide the weight data of the neural network model into weight packets corresponding to each neuron, and store the weight packets corresponding to each neuron in the storage unit; a prediction unit, used to extract feature information of the intermediate calculation results of the current calculation layer during the calculation of the current calculation layer, and predict the set of neurons to be activated in the next calculation layer based on the feature information; and a prefetch unit, used to read the weight packets corresponding to each neuron in the set of neurons from the storage unit before entering the next calculation layer, for use in the calculation of the next calculation layer.

[0078] The electronic device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0079] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0080] The memory 31 can be an internal storage unit of the electronic device, such as a hard drive or memory. The memory 31 can also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units. The memory 31 is used to store the computer program and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0081] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned electronic device can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0083] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0084] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this application.

[0085] In the embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0089] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calculating a neural network model, characterized in that, The neural network model includes multiple computational layers, each layer comprising neurons, and the computational method of the neural network model includes: The weight data of the neural network model is divided into weight packets corresponding to each neuron, and the weight packets corresponding to each neuron are stored in the storage unit. During the computation of the current computation layer, feature information of the intermediate computation results of the current computation layer is extracted, and the set of neurons to be activated in the next computation layer is predicted based on the feature information. Before entering the next computational layer, the weight packets corresponding to each neuron in the neuron set are read from the storage unit for use in the computation of the next computational layer.

2. The calculation method for the neural network model as described in claim 1, characterized in that, The step of dividing the weight data of the neural network model into weight packets corresponding to each neuron includes: The feedforward network layer of the neural network model is structurally analyzed to identify the gate weight matrix, upper projection weight matrix, and lower projection weight matrix in each computational layer. Taking a single neuron as a unit, the weight data, quantized weight values, and scale information of the neuron in the gating weight matrix, the upper projection weight matrix, and the lower projection weight matrix are aggregated to obtain the weight packet corresponding to the neuron.

3. The calculation method for the neural network model as described in claim 1, characterized in that, The step of storing the weight packets corresponding to each neuron in the storage unit includes: Multiple weight packets are written to the storage unit in a preset order, so that each weight packet is physically stored consecutively. Each weight packet is assigned a unique physical address identifier, and a mapping relationship between the neuron index and the physical address is established.

4. The calculation method for the neural network model as described in claim 3, characterized in that, The step of reading the weight packet corresponding to each neuron in the neuron set from the storage unit includes: Based on the neuron index list corresponding to the neuron set and the mapping relationship, query the target physical address corresponding to each neuron index in the neuron index list; A read request is generated based on the target physical address and added to the request queue; Based on the request queue, a read operation is initiated to the storage unit to load the weight packets corresponding to each neuron in the neuron set into a preset cache area for calculation.

5. The calculation method for the neural network model as described in claim 4, characterized in that, The calculation method of the neural network model also includes: Assign an initial priority value to each read request in the request queue; The priority of each read request in the request queue is adjusted in real time, and the read requests in the request queue are sorted according to the adjusted priority; the rules for priority adjustment include one or more of the following: According to the execution order of each computation layer in the neural network model, the priority of read requests from the preceding computation layer is adjusted to be higher than that of read requests from the following computation layers; when the computation layer corresponding to a read request is idle due to waiting for weight packets, the priority of the read request is increased to the highest level. Each read request is prioritized according to its waiting time in the request queue; when multiple read requests have the same priority value, their priorities are adjusted according to the order in which they arrived in the request queue.

6. The calculation method for the neural network model as described in any one of claims 1-5, characterized in that, The step of reading the weight packet corresponding to each neuron in the neuron set from the storage unit includes: The gated weights in the weight package are loaded first; After completing the gating calculation, the activation status of the corresponding neuron is determined based on the gating value: If the neuron is activated, the remaining weight data in the weight package continues to be loaded; If the neuron is not activated, the subsequent loading of the weight pack is terminated.

7. The calculation method for the neural network model as described in any one of claims 1-5, characterized in that, The calculation method of the neural network model also includes: After the neural network model completes one forward propagation, the matching between the predicted set of neurons and the actual set of activated neurons is recorded. Adjust the prediction model used to generate the set of neurons based on the matching results, and / or adjust the priority of read requests.

8. A computing device for a neural network model, characterized in that, The neural network model includes multiple computational layers, each of which includes neurons. The computational device of the neural network model includes: The storage unit is used to divide the weight data of the neural network model into weight packets corresponding to each neuron, and store the weight packets corresponding to each neuron in the storage unit. The prediction unit is used to extract feature information of the intermediate calculation results of the current calculation layer during the calculation of the current calculation layer, and predict the set of neurons to be activated in the next calculation layer based on the feature information. A prefetching unit is used to read the weight packets corresponding to each neuron in the neuron set from the storage unit before entering the next computational layer, for use in the computation of the next computational layer.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the computation method for the neural network model as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program is run, the computation method of the neural network model as described in any one of claims 1 to 7 is executed.

Images