A method of processing data using a neural network
By scheduling parameters and data processing on an operator-by-operator basis, the problem of limited hardware circuit memory resources is solved, thereby improving the data processing efficiency and memory utilization efficiency of neural networks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MONTAGE TECH KUNSHAN CO LTD
- Filing Date
- 2021-12-14
- Publication Date
- 2026-07-10
AI Technical Summary
Existing hardware circuits have limited memory resources, making it difficult to effectively load neural network processing parameters, which increases hardware resource overhead and reduces data processing efficiency.
By scheduling processing parameters and data on an operator-by-operator basis, the maximum amount of data that each operator can process per batch is determined, and processing parameters are loaded and used in batches to avoid conflicts in the processing capabilities of different operators and improve data processing efficiency.
It improves the data processing efficiency of neural networks, reduces the operating overhead of memory resources, and optimizes the loading and use of processing parameters.
Smart Images
Figure CN116263873B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural networks, and more specifically to a method for processing data using neural networks. Background Technology
[0002] A neural network is a machine learning model that uses one or more layers to process data and compute inferences. Each layer includes one or more operators with a corresponding set of processing parameters, which are used to process the input data received by that layer. The output data (i.e., the inference result) obtained by each layer can be used as the input data for the next layer.
[0003] In the prior art, neural networks can be implemented using specific hardware circuitry. This hardware circuitry may include one or more non-transitory machine-readable storage media (e.g., memory) for storing data input to the neural network and processing parameters required by operators to process that data. However, because hardware circuitry typically includes limited memory resources—for example, integrated "on-chip" memory of a predetermined size—existing neural networks often struggle to efficiently load operator processing parameters, increasing hardware resource overhead.
[0004] Therefore, it is necessary to provide an improved method for processing data using neural networks. Summary of the Invention
[0005] One objective of this application is to provide an improved method for processing data using neural networks to improve the efficiency of loading and using processing parameters.
[0006] In one aspect of this application, a method for processing data using a neural network is provided, the neural network comprising at least one layer; for one or more of the at least one layer, the method comprises: determining a plurality of sets of operators included in the layer, wherein each set of operators has a corresponding set of processing parameters and is allocated memory resources of a predetermined size; determining the maximum amount of data that each set of operators can process per batch based on the set of processing parameters corresponding to each set of operators and the memory resources allocated to it; loading the set of processing parameters corresponding to each set of operators into memory respectively; and processing the input data received by the layer using the plurality of sets of operators according to the maximum amount of data that the plurality of sets of operators can process per batch.
[0007] In some embodiments, the multiple sets of operators process the input data in parallel, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that the multiple sets of operators can process in each batch further includes: each set of operators processes the input data received by the layer separately according to the maximum amount of data that it can process in each batch.
[0008] In some embodiments, the multiple sets of operators process the input data serially, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that the multiple sets of operators can process in each batch further includes: each set of operators sequentially processes the data it receives according to the maximum amount of data that it can process in each batch, and then provides the processed data to the next adjacent set of operators for further processing by the next set of operators.
[0009] In some embodiments, the step of loading a set of processing parameters corresponding to each set of operators into the memory includes: for each set of operators, loading a set of processing parameters corresponding to it into the memory at once.
[0010] In some embodiments, each group of operators processes the same data together.
[0011] In some embodiments, the plurality of operators have different maximum data volumes that can be processed per batch, and the step of processing the input data received by the layer with the plurality of operators according to the maximum data volumes that can be processed per batch by the plurality of operators includes: the plurality of operators processing the input data received by the layer in different numbers of batches.
[0012] In another aspect of this application, a data processing apparatus based on a neural network is also provided. The neural network includes at least one layer. The data processing apparatus includes a non-transitory computer storage medium on which one or more executable instructions are stored. For one or more layers of the at least one layer of the neural network, the one or more executable instructions, after being executed by a processor, perform the following steps: determining multiple sets of operators included in the layer, wherein each set of operators has a corresponding set of processing parameters and is allocated memory resources of a predetermined size; determining the maximum amount of data that each set of operators can process per batch based on the set of processing parameters corresponding to each set of operators and the memory resources allocated to them; loading the set of processing parameters corresponding to each set of operators into memory respectively; and processing the input data received by the layer using the multiple sets of operators according to the maximum amount of data that the multiple sets of operators can process per batch.
[0013] In another aspect of this application, a data processing apparatus based on a neural network is also provided. The data processing apparatus includes: a memory for storing processing parameters corresponding to operators in the neural network and data processed by the neural network; and a controller for controlling the input and storage of data and processing parameters into the memory, and subsequently controlling the provision of these data and processing parameters to a computing array for the computing array to perform computational operations in the neural network. The neural network includes at least one layer, and the operators in each layer are grouped. For each layer of the neural network, the controller is further configured to load a set of processing parameters corresponding to each group of operators into the memory, and process the input data received by the layer using the multiple groups of operators according to the maximum amount of data that each batch of multiple groups of operators can process.
[0014] The above is an overview of this application, and there may be simplifications, generalizations, and omissions of details. Therefore, those skilled in the art should recognize that this section is merely illustrative and not intended to limit the scope of this application in any way. This overview section is neither intended to identify the key or essential features of the claimed subject matter nor to serve as an aid in determining the scope of the claimed subject matter. Attached Figure Description
[0015] The above and other features of this application will become more fully clear through the following description and appended claims, in conjunction with the accompanying drawings. It is understood that these drawings depict only a few embodiments of the application and should not be construed as limiting the scope of the application. The application will be described more clearly and in more detail through the use of the drawings.
[0016] Figure 1 A hardware circuit for implementing a neural network according to an embodiment of this application is shown;
[0017] Figure 2 A schematic diagram of memory resource allocation during the execution of a hardware circuit according to an embodiment of this application is shown;
[0018] Figure 3 A method for processing data using a neural network according to an embodiment of this application is shown;
[0019] Figure 4a A schematic diagram is shown illustrating multiple sets of operators in a layer of a neural network processing data in parallel according to an embodiment of this application;
[0020] Figure 4b This diagram illustrates how multiple sets of operators in a single layer of an existing neural network process data in parallel.
[0021] Figure 5aA schematic diagram is shown illustrating multiple sets of operators in one layer of a neural network sequentially processing data according to an embodiment of this application;
[0022] Figure 5b This diagram illustrates how multiple sets of operators in a single layer of an existing neural network process data sequentially. Detailed Implementation
[0023] In the following detailed description, reference is made to the accompanying drawings, which form a part thereof. In the drawings, similar symbols generally denote similar components unless the context otherwise requires. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments and variations may be employed without departing from the spirit or scope of the subject matter of this application. It will be understood that various different configurations, substitutions, combinations, and designs can be made to the various aspects of the general description and illustrated in the drawings of this application, all of which explicitly form part of the subject matter of this application.
[0024] Figure 1 A hardware circuit 100 for implementing a neural network according to an embodiment of this application is illustrated. In some embodiments, the neural network may include one or more layers, each layer having a corresponding set of processing parameters. In some embodiments, the neural network can compute inferences through multiple layers arranged in a certain order, and each layer has a corresponding set or more sets of processing parameters, each set of processing parameters being constructed as a matrix or tensor. In some embodiments, any layer of the neural network can receive multiple input data and / or generate and output multiple inference computation results. Optionally, the inference computation result output by one layer can be fed back as input to the next layer. Depending on their specific location in the neural network, different layers may have different functions (e.g., input layer, output layer, hidden layer, etc.) and receive different data.
[0025] like Figure 1As shown, the hardware circuit 100 for implementing a neural network may include a memory 102 and a controller 108. The controller 108 may provide control signals 110 indicating various functions and operations to control the operation of other modules of the hardware circuit 100. For example, the control signal 110 may be used to control the input and storage of data in the data memory 104, and subsequently control the data to be provided via an input data bus to a MAC array 107 or similar computing array including multiple multiply-accumulate (MAC) units, so that the MAC array 107 performs computational operations in the neural network; specifically, the MAC array 107 performs calculations on the data. As another example, the control signal 110 may also control the storage of processing parameters corresponding to operators in a parameter memory 106, and subsequently control the loading of these processing parameters into the MAC array 107 during computation. It is understood that the controller 108 may include one or more processing units and / or a memory, and these processing units may call and execute instructions stored in the memory to cause the hardware circuit 100 to perform one or more functions described in the embodiments of this application. The processing unit in controller 108 may be implemented using a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processing Circuit), ASIC (Application-Specific Integrated Circuit), or other similar hardware implementations, or it may be implemented using software or firmware, or a combination of software, hardware, and firmware. The memory in controller 108 may include one or more volatile and / or non-volatile machine-readable storage media, such as solid-state memory, magnetic disk, optical disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other suitable type of storage media.
[0026] As mentioned above, parameter memory 106 and data memory 104 together constitute memory 102. Due to limitations in the actual hardware resources of the hardware circuitry, memory 102, as well as its data memory 104 and parameter memory 106, typically have limited memory resources. It should be noted that, although in Figure 1The data memory 104 and parameter memory 106 are represented as two separate memory modules. However, in some embodiments, the data memory and parameter memory may be integrated into the same memory module, for example, the memory module may have different storage areas to store data and processing parameters respectively. For a given computation cycle, the hardware circuit 100 may need to access the data memory 104 and parameter memory 106 to perform computational operations associated with the inference computation of each layer in the neural network. Specifically, when the hardware circuit 100 receives input data and processing parameters, the controller 108 may perform one or more access operations to the memory 102, such as storing the input data in the data memory 104 or storing the processing parameters in the parameter memory 106; in addition, the controller 108 may also extract these data / processing parameters and provide them to the MAC array 107 for corresponding computations.
[0027] In existing technologies, to reduce access to memory 102, input data or other data to be processed is typically processed in batches, that is, a certain amount of data is divided into batches, and then processed together in subsequent batches. Furthermore, while batching the data, processing parameters are called and loaded on a layer-by-layer basis within the neural network. The inventors of this application have found that, because multiple operators in the same layer need to coordinate with each other (especially when hardware resources are limited), this layer-by-layer parameter calling method reduces the maximum amount of data that can be processed in each batch, thereby reducing the data processing efficiency of the neural network. To solve this problem, the inventors of this application propose a method for scheduling processing parameters and data on an operator-by-operator basis. This method can avoid the impact of conflicts / differences in the processing capabilities of different operators on data processing efficiency, thereby improving data processing efficiency.
[0028] Still referencing Figure 1As shown, in some embodiments, the neural network may have one or more layers, each layer including multiple sets of operators. Each set of operators may include one or more operators that can collectively process the same data. The output data obtained after processing by each layer can be output through an output data bus. Furthermore, these sets of operators may correspond to the same or different computation types. For example, each set of operators may be used to perform one of the following computation types: convolution, element-wise computation, batch normalization (BN), merge computation, etc. In some embodiments, depending on the specific implementation, the multiple operators included in each set of operators may each implement different processing functions and collectively implement one type of computational operation. For example, a set of operators used for convolution computation may include processing functions such as Convolution Direct Memory Access (CDMA), Convolution Sequencer Controller (CSC), Convolution Multiplier And Accumulation (CMAC), and Convolution Accumulator (CACC). Convolution computation can be achieved when these operators process common data. It is understood that the aforementioned division of the set of operators required for convolution computation is merely exemplary; in practical applications, convolution computation can adopt other operator division methods depending on different specific implementations. During convolution computation, the required set of processing parameters (i.e., weight parameters) needs to traverse the matrix of input data and perform vector calculations (dot products) with each submatrix in the matrix during traversal to obtain the cumulative value. For example, for a set of weight parameters, the neural network can multiply each weight parameter with each input data point and sum the products to obtain the cumulative value. It is understood that for each set of operators, since they generally process the same data, they can have a corresponding set of processing parameters. In the embodiments of this application, each group of operators can process data in batches. In other words, each group of operators can process a certain amount of data in a batch and then output the processed data simultaneously; thus, in the next batch, the same group of operators can similarly process another batch of data; until all the data to be processed by the group of operators has been processed.
[0029] In practical applications, limited by the available hardware resources, each set of operators is typically allocated a predetermined amount of memory to store its corresponding set of processing parameters. It can be understood that for each set of operators, the maximum amount of data that can be processed per batch is determined based on the number of processing parameters and the size of the allocated memory resources. This is usually determined empirically by the neural network designers. For example, factors such as network features in the operator (matrix size, number of layers, number of channels, etc.), the data format of the input data in the input operator, and hardware implementation costs are all considerations in determining the maximum amount of data that can be processed per batch. For instance, all other things being equal, the more numerous and complex the network features in the operator, the smaller the maximum amount of data that can be processed per batch. Furthermore, format differences between the input data and the network design of the operator require preprocessing of the input data, which consumes hardware resources, thus reducing the maximum amount of data that can be processed per batch. It is understood that in some cases, the maximum amount of data that each set of operators can process per batch can be calculated through pre-designed rules and algorithms. This application does not limit the determination to be made by designers or other personnel based on experience. It should also be noted that although the maximum amount of data that each set of operators can process per batch can be determined, in actual processing, the amount of data processed by each set of operators per batch may be less than or equal to that maximum amount.
[0030] It should be noted that for each set of operators, the processing parameters corresponding to that set of operators can always be loaded into memory during the process of processing multiple batches of data (e.g., Figure 1 The parameters are stored in the parameter memory 106 shown, without having to repeatedly load this set of processing parameters multiple times. Preferably, after the set of operators has processed the input data stored in the data memory 104 in batches, the memory area occupied by the processing parameters of the set of operators in the parameter memory 106 can be released to store the processing parameters required by other operators.
[0031] It is understandable that in practical applications, in order to realize data processing in a layer of a neural network, each layer may include multiple sets of operators, each with a corresponding set of processing parameters, and each allocated a predetermined amount of memory resources. Figure 2 A schematic diagram illustrating memory resource allocation during the execution of a hardware circuit according to an embodiment of this application is shown. It can be understood that since the neural network implemented by the hardware circuit may include multiple layers, therefore... Figure 2 The memory resource allocation diagram shown is a schematic diagram of a layer in this neural network implemented by hardware circuitry.
[0032] like Figure 2As shown, this layer exemplarily includes three regions: a first memory region 201, a second memory region 203, and a third memory region 205. These regions are used to store a set of processing parameters required by three sets of operators (i.e., the conv operator, the element-wise operator, and the BN operator) when processing data. In some embodiments, since each memory region is used to store the processing parameters of a specific operator, these memory regions are not reused when data is actually processed in each layer. That is, the first memory region 201 does not store the processing parameters corresponding to the element-wise operator or the BN operator, and the second memory region 203 or the third memory region 205 does not store the processing parameters corresponding to the conv operator. It should be noted that the number of operator groups and the functional types included in each layer are merely illustrative. In practical applications, each layer may include 2, 4, 5, or more sets of operators, and may also include operators used to implement functions, such as pooling operators, rectified linearization (ReLU) operators, etc.
[0033] As mentioned earlier, each layer of a neural network can process data in batches. Based on the storage capacity of the first memory region 201, the second memory region 203, and the third memory region 205, and the size of the processing parameters corresponding to the conv operator, BN operator, and element-wise operator, the maximum amount of data that these operators can process per batch can be determined by calculation. For example, in Figure 2 In the example shown, it is assumed that calculations show the conv operator can process a maximum of 10 data points per batch, the BN operator can process a maximum of 20 data points per batch, and the element-wise operator can process a maximum of 30 data points per batch. If input node 1 of this layer receives 60 input data points, then the conv operator needs 6 batches to process all the data, the element-wise operator needs 2 batches, and the BN operator needs 3 batches.
[0034] Based on the above explanation, it is possible to utilize Figure 1 The hardware circuit shown implements a neural network and uses this neural network to process data. Figure 3A method 300 for processing data using a neural network according to an embodiment of this application is illustrated. It should be noted that a neural network typically includes at least one layer, and for one or more of these layers, method 300 can be used to process the data received by each layer. Method 300 only exemplarily illustrates at least a portion of the steps required for one layer to process data; those skilled in the art will understand that other layers of the neural network can process data using the same or similar steps. Furthermore, it is understood that for a neural network with multiple layers, the data processing method of the embodiments of this application can also be applied to some layers of the neural network, while other layers can employ different data processing methods.
[0035] like Figure 3 As shown, in step 302, a layer of the neural network is determined to include multiple sets of operators, wherein each set of operators has a corresponding set of processing parameters and is allocated memory resources of a predetermined size.
[0036] In some embodiments, a set of operators can be multiple operators that collectively process the same data. For example, the aforementioned set of operators for convolution computation; when these operators process common data, convolution computation can be achieved. The set of processing parameters corresponding to each set of operators can be, for example, one or more parameter matrices, which can be arranged according to, for example... Figure 3 The corresponding memory resources are allocated in the manner shown. In some embodiments, different operators can be categorized by providing a preset mapping table. For example, operators related to convolution matrix operations (e.g., conv2d, conv1d, depthwise_conv2d, fully_connected, etc.) and operators related to element-wise operations (add, sub, mul, abs, sin, cos, sqrt, etc.) can be grouped together, while pool operators are grouped separately. In some examples, the controller can determine which operators belong to the same group based on data processing needs and aggregate or group these operators into the same group. Alternatively, operator categorization can also be achieved through other automated algorithms.
[0037] Next, in step 304, based on a set of processing parameters corresponding to each set of operators and the memory resources allocated to them, the maximum amount of data that each set of operators can process per batch is determined.
[0038] For example, calculations show that a current layer in a neural network includes three sets of operators: the conv operator (which will be used here to implement convolution calculation), the BN operator, and the element-wise operator. These operators can process a maximum of 10, 20, and 30 data points per batch, respectively.
[0039] Subsequently, in step 306, a set of processing parameters corresponding to each set of operators is loaded into the memory. As mentioned above, in the embodiments of this application, the processing parameters of operators in each layer of the neural network are loaded on a per-set-operator basis, rather than on a per-layer basis. In other words, for Figure 2 The example shown can load the three sets of processing parameters corresponding to the three sets of operators into their respective allocated memory areas at the same time, or they can be loaded into their respective memory areas sequentially.
[0040] Then, in step 308, the input data received by the layers of the neural network are processed using the multiple sets of operators based on the maximum amount of data that each batch of multiple sets of operators can process.
[0041] In a layer of a neural network, different operators can have different connections, such as parallel connections or serial connections. In parallel connection mode, all operators in a layer perform computations relatively independently and in parallel, with each operator processing the input data received by the layer based on its maximum batch processing capacity. In serial connection mode, all operators in a layer can still perform computations relatively independently, with each group of operators processing its received input data based on its maximum batch processing capacity, and then providing the processed data to the next adjacent group of operators for further processing. Using the data processing method of this application embodiment, the loading and use of processing parameters are more efficient, regardless of whether different groups of operators are connected serially or in parallel.
[0042] Next, steps 306 and 308 of the data processing method of this application embodiment will be specifically described with reference to examples. The examples here assume that the number of input data is 60, and includes three sets of operators: the conv operator, the BN operator, and the element-wise operator. The maximum number of data points that these operators can process per batch are 10, 20, and 30, respectively. For ease of description, the multiple operators required for convolution calculation are collectively referred to as the conv operator, and the other sets of operators are similarly referred to.
[0043] In parallel connection mode, when there are 60 input data points, the conv operator requires 6 batches to process all the input data. In this case, the set of processing parameters required by the conv operator is loaded once and reused for 6 batches. Similarly, the BN operator requires 3 batches to process all the input data, and the set of processing parameters required by the BN operator is loaded once and reused for 3 batches. The element-wise operator requires 2 batches to process all the input data, and the set of processing parameters required is loaded once and reused for 2 batches. After all three sets of operators have completed processing the input data, the processing result can be output by output node 2 and provided to the next layer for further computation.
[0044] Figure 4a A schematic diagram illustrating the parallel processing of data by multiple sets of operators in a layer of a neural network according to an embodiment of this application is shown. Figure 4a As shown, the three sets of processing parameters corresponding to the three sets of operators have been preloaded into memory. Thus, in the first batch (batch 1), the `conv` operator processes 10 data points, the `BN` operator processes 20 data points, and the element-wise operator processes 30 data points. In the second batch (batch 2), the `conv` operator continues to process 10 data points, the `BN` operator continues to process 20 data points, and the element-wise operator continues to process 30 data points (i.e., processing 60 data points in total). At this point, the memory resources occupied by the processing parameters corresponding to the element-wise operator can be released. In the third batch (batch 3), the `conv` operator continues to process 10 data points, and the `BN` operator continues to process 20 data points (i.e., processing 60 data points in total). At this point, the memory resources occupied by the processing parameters corresponding to the `BN` operator can also be released. In the fourth to sixth batches (batch 4 to batch 6), the `conv` operator continues to process 10 data points per batch until all 60 data points have been processed, and the memory resources occupied by the processing parameters corresponding to the `conv` operator can also be released. The processing results of each set of operators can be output by output node 2.
[0045] Figure 4b This diagram illustrates how multiple sets of operators in a single layer of an existing neural network process data in parallel. For example... Figure 4b As shown, the three sets of processing parameters required by the three sets of operators are loaded layer by layer. Therefore, the maximum amount of data that can be processed in each batch depends on the set of operators with the lowest processing capacity. Figure 4bIn the example shown, the `conv` operator is the lowest-capacity operator, capable of processing only 10 data points per batch. Furthermore, due to the accuracy requirements of data processing, in existing technologies, processing parameters cannot be reused when loading them layer by layer. That is, existing technologies process data layer by layer, with each layer processing a fixed amount of data. Only after all operators in a layer have finished processing will the next layer be processed (e.g., processing data from the next layer or loading additional data into the current layer). During layer-by-layer processing, the memory records the parameters of each operator in the processing order. When all operators in a layer have finished processing, the parameters loaded in memory are those of the last operator. Therefore, when new data enters the same layer, the parameters of the first operator need to be reloaded into memory; parameters already loaded in memory cannot be reused. Thus, after each batch of processing, all processing parameters required for the operators of that layer need to be reloaded. For example, as... Figure 4b As shown, in the first batch (batch 1), the `conv` operator processes 10 data points, and the `BN` and `element-wise` operators also process the same 10 data points each. Similarly, in the second batch (batch 2) to the sixth batch (batch 6), the `conv`, `BN`, and `element-wise` operators each process the same 10 data points per batch, until all 60 data points have been processed. The processing results can be further processed by the `concat` operator and then output by output node 2.
[0046] It can be seen that, compared to Figure 4b The existing data processing methods shown herein, compared to the data processing method in this application embodiment, can process data according to the maximum batch size that each set of operators can handle, and can reuse processing parameters without repeatedly loading these parameters. That is, processing is done on an operator-by-operator basis, combining multiple data sets processed at the same level and processing them sequentially, minimizing or avoiding parameter reloading. Therefore, the data processing method of this application can reduce the operating overhead of memory resources.
[0047] Compared to existing data processing methods, loading processing parameters by operator also has advantages for neural networks where multiple operators are connected in series.
[0048] Figure 5a A schematic diagram is shown illustrating how multiple sets of operators in one layer of a neural network process data sequentially according to an embodiment of this application. For example... Figure 5aAs shown, the conv operator receives input data from input node 1. The data processed by the conv operator is then provided to the next-level BN operator, and the data processed by the BN operator is then provided to the next-level element-wise operator. The data processed by the element-wise operator can be output by output node 2.
[0049] Specifically, during the data processing by these three sets of operators, the first-level conv operator can process a maximum of 10 data points per batch. Therefore, after loading a set of processing parameters corresponding to the conv operator, six batches of computation can be performed to process 60 data points. At this time, the processing parameters required by the BN and element-wise operators do not need to be loaded into memory, and memory resources can be used for other computations. After the conv operator completes the processing of 60 data points, the memory resources occupied by the conv operator's processing parameters can be released. At the same time, a set of processing parameters required by the BN operator can be loaded into memory at once, and three batches (20 data points per batch) of computation can be performed to process 60 data points. Similarly, afterwards, the memory resources occupied by the BN operator's processing parameters can be released; at the same time, a set of processing parameters required by the element-wise operator can be loaded into memory at once, and two batches (30 data points per batch) of computation can be performed to process 60 data points.
[0050] Figure 5b This diagram illustrates how multiple sets of operators in a single layer of an existing neural network process data sequentially. For example... Figure 5b As shown, since the three sets of processing parameters required by the three sets of operators at the same layer—the conv operator, the BN operator, and the element-wise operator—are loaded together, and the processing capacity of this serial processing path is limited by the processing capacity of the conv operator with the lowest batch processing capacity (i.e., processing 10 data points per batch), at least six batches are needed to process 60 data points in the existing implementation, with each batch processing 10 data points. Furthermore, since all three sets of processing parameters are loaded together, this method still requires a significant amount of memory resources.
[0051] In some embodiments, this application also provides computer program products including a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium includes computer-executable code for performing... Figure 3 The steps in the illustrated method embodiments. In some embodiments, the computer program product may be stored in a hardware device.
[0052] Embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.
[0053] Those skilled in the art can understand and implement other modifications to the disclosed embodiments by studying the specification, the disclosure, the drawings, and the appended claims. In the claims, the word "comprising" does not exclude other elements and steps, and the words "a" or "an" do not exclude a plurality. In practical applications of this application, a single part may perform the function of multiple technical features referenced in the claims. Any reference numerals in the claims should not be construed as limiting the scope.
Claims
1. A method for processing data using a neural network, characterized in that, The neural network includes at least one layer; for one or more of the at least one layer, the method includes: The layer comprises multiple sets of operators, each set of operators having a corresponding set of processing parameters and being allocated memory resources of a predetermined size; Based on a set of processing parameters corresponding to each set of operators and the memory resources allocated to them, determine the maximum amount of data that each set of operators can process per batch. Load the processing parameters corresponding to each set of operators into memory; and The input data received by the layer is processed using the multiple sets of operators based on the maximum amount of data that each batch can process.
2. The method according to claim 1, characterized in that, The multiple sets of operators process the input data in parallel, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that each batch of operators can process further includes: Each group of operators processes the input data received by the layer according to the maximum amount of data it can process per batch.
3. The method according to claim 1, characterized in that, The multiple sets of operators process the input data serially, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that each batch of operators can process further includes: Each group of operators processes the received data sequentially according to the maximum amount of data it can process in each batch, and then provides the processed data to the next adjacent group of operators for further processing.
4. The method according to claim 1, characterized in that, The step of loading a set of processing parameters corresponding to each set of operators into the memory includes: For each set of operators, a set of processing parameters corresponding to it is loaded into memory at once.
5. The method according to claim 1, characterized in that, Each group of operators processes the same data together.
6. The method according to claim 1, characterized in that, The operators are categorized into multiple groups using a preset mapping table.
7. The method according to claim 1, characterized in that, The multiple sets of operators have different maximum data volumes that can be processed per batch. The step of processing the input data received by the layer using the multiple sets of operators according to the maximum data volumes that can be processed per batch includes: The multiple sets of operators process the input data received by the layer in batches of varying numbers.
8. A data processing device based on a neural network, characterized in that, The neural network includes at least one layer, and the data processing device includes a non-transitory computer storage medium storing one or more executable instructions. For one or more layers of the at least one layer of the neural network, the one or more executable instructions, after being executed by the processor, perform the following steps: The layer comprises multiple sets of operators, each set of operators having a corresponding set of processing parameters and being allocated memory resources of a predetermined size; Based on a set of processing parameters corresponding to each set of operators and the memory resources allocated to them, determine the maximum amount of data that each set of operators can process per batch. Load a set of processing parameters corresponding to each set of operators into memory respectively; as well as The input data received by the layer is processed using the multiple sets of operators based on the maximum amount of data that each batch can process.
9. The apparatus according to claim 8, characterized in that, The multiple sets of operators process the input data in parallel, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that each batch of operators can process further includes: Each group of operators processes the input data received by the layer according to the maximum amount of data it can process per batch.
10. The apparatus according to claim 8, characterized in that, The multiple sets of operators process the input data serially, and the step of processing the input data received by the layer with the multiple sets of operators according to the maximum amount of data that each batch of operators can process further includes: Each group of operators processes the received data sequentially according to the maximum amount of data it can process in each batch, and then provides the processed data to the next adjacent group of operators for further processing.
11. A data processing device based on a neural network, characterized in that, The data processing device includes: A memory, wherein the memory is used to store the processing parameters corresponding to the operators in the neural network and the data processed by the neural network; A controller is configured to control the input and storage of data and processing parameters into the memory, and subsequently control the provision of this data and processing parameters to the computing array so that the computing array can perform computational operations in the neural network. The neural network includes at least one layer, and the operators in each layer are grouped together. For each layer of the neural network, the controller is further configured to load a set of processing parameters corresponding to each group of operators into the memory, and process the input data received by the layer using the multiple groups of operators according to the maximum amount of data that each group of operators can process in each batch.