Method for data processing in a neural network system, neural network system

Abstract

The application provides a method for data processing in a neural network system and a neural network system. The method comprises: inputting training data into the neural network system to obtain first output data, and adjusting weight values stored in at least one memory and calculation unit in a part of neural network arrays in the neural network system according to a deviation between the first output data and target output data. The part of neural network arrays are used to implement calculation of part of neural network layers in the neural network system. The application can improve the performance of the neural network system and the accuracy of recognition.

Description

Technical Field

[0001] This application relates to the field of neural networks, and more specifically, to methods for data processing in neural network systems and neural network systems. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] In the field of AI, deep learning is a learning technique based on deep artificial neural network (ANN) algorithms. The training process of neural networks is a data-centric task that requires computing hardware with high-performance, low-power processing capabilities.

[0004] Neural network systems based on multiple neural network arrays can achieve in-memory computing and handle deep learning tasks. For example, at least one in-memory unit in a neural network array can store the weight values ​​of the corresponding neural network layer. Due to network structure or system architecture design, the processing speeds of different neural network arrays may differ. In such cases, multiple neural network arrays can be used for parallel processing to accelerate the neural network array that is experiencing speed bottlenecks. However, due to some non-ideal characteristics inherent in the in-memory units themselves, such as device fluctuations, conductivity drift, and array yield, the overall performance of the neural network system will be reduced, resulting in lower accuracy. Summary of the Invention

[0005] This application provides a method for data processing in a parallel-accelerated neural network system, and a neural network system that can solve the problems caused by the non-ideal characteristics of devices when using parallel acceleration technology, thereby improving the performance and recognition accuracy of the neural network system.

[0006] In a first aspect, a method for data processing in a neural network system is provided, comprising: in a neural network system employing parallel acceleration, inputting training data into the neural network system to obtain first output data, wherein the neural network system includes multiple neural network arrays, each of the multiple neural network arrays including multiple storage units, each storage unit being used to store weight values ​​of neurons in the corresponding neural network; calculating the deviation between the first output data and target output data; and adjusting the weight values ​​stored in at least one storage unit in a portion of the multiple neural network arrays according to the deviation, wherein the portion of the neural network arrays is used to implement the calculation of a portion of the neural network layers in the neural network system.

[0007] In the above technical solution, the weight values ​​stored in the memory units of some of the neural network arrays can be adjusted and updated according to the deviation between the actual output data and the target output data of the neural network array. This can achieve compatibility with the non-ideal characteristics of the memory units, improve the recognition rate and performance of the system, and thus avoid the performance degradation caused by the non-ideal characteristics of the memory units.

[0008] In one possible implementation of the first aspect, the plurality of neural network arrays include a first neural network array and a second neural network array, wherein the input data of the first neural network array includes the output data of the second neural network array.

[0009] In another possible implementation of the first aspect, the first neural network array includes a neural network array for implementing fully connected layer computations in the neural network.

[0010] In the above technical solution, only the weight values ​​stored in the memory units of the neural network array that implements the fully connected layer computation can be adjusted and updated, which can achieve compatibility with the non-ideal characteristics of the memory units, improve the recognition rate and performance of the system, and has low cost, is effective and easy to implement.

[0011] In another possible implementation of the first aspect, the weight values ​​stored in at least one of the memory units in the first neural network array are adjusted based on the input values ​​of the first neural network array and the deviation.

[0012] In another possible implementation of the first aspect, the plurality of neural network arrays further include a third neural network array, which, together with the second neural network array, is used to perform the computation of convolutional layers in the neural network in parallel.

[0013] In another possible implementation of the first aspect, the weight values ​​stored in at least one of the memory units in the second neural network array are adjusted according to the input data of the second neural network array and the deviation; and the weight values ​​stored in at least one of the memory units in the third neural network array are adjusted according to the input data of the third neural network array and the deviation.

[0014] In the above technical solution, the weight values ​​stored in the storage units of multiple neural network arrays that perform convolutional layer calculations in parallel can also be adjusted and updated to improve the adjustment accuracy, thereby improving the accuracy of the neural network system output.

[0015] In another possible implementation of the first aspect, the deviation is divided into at least two sub-deviations, wherein a first sub-deviation of the at least two sub-deviations corresponds to the output data of the second neural network array, and a second sub-deviation of the at least two sub-deviations corresponds to the output data of the third neural network array; the weight values ​​stored in at least one of the memory units in the second neural network array are adjusted according to the first sub-deviation and the input data of the second neural network array; and the weight values ​​stored in at least one of the memory units in the third neural network array are adjusted according to the second sub-deviation and the input data of the third neural network array.

[0016] In another possible implementation of the first aspect, the number of pulses is determined based on the updated weight values ​​in the memory cell, and the weight values ​​stored in at least one memory cell in the neural network array are rewritten based on the number of pulses.

[0017] Secondly, a neural network system is provided, including:

[0018] The processing module is used to input training data into the neural network system to obtain first output data. The neural network system includes multiple neural network arrays, and each of the multiple neural network arrays includes multiple storage units. Each storage unit is used to store the weight values ​​of the neurons in the corresponding neural network.

[0019] The calculation module is used to calculate the deviation between the first output data and the target output data;

[0020] An adjustment module is used to adjust the weight values ​​stored in at least one of the computing units in a portion of the plurality of neural network arrays according to the deviation, wherein the portion of the neural network array is used to perform calculations on a portion of the neural network layers in the neural network system.

[0021] In one possible implementation of the second aspect, the plurality of neural network arrays includes a first neural network array and a second neural network array, wherein the input data of the first neural network array includes the output data of the second neural network array.

[0022] In another possible implementation of the second aspect, the first neural network array includes a neural network array for implementing fully connected layer computations in the neural network.

[0023] In another possible implementation of the second aspect, the adjustment module is specifically used for:

[0024] Based on the input value of the first neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the first neural network array are adjusted.

[0025] In another possible implementation of the second aspect, the plurality of neural network arrays further include a third neural network array, which, together with the second neural network array, is used to perform the computation of convolutional layers in the neural network in parallel.

[0026] In another possible implementation of the second aspect, the adjustment module is specifically used for:

[0027] Based on the input data of the second neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the second neural network array are adjusted; based on the input data of the third neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the third neural network array are adjusted.

[0028] In another possible implementation of the second aspect, the adjustment module is specifically used for:

[0029] The deviation is divided into at least two sub-deviations, wherein the first sub-deviation of the at least two sub-deviations corresponds to the output data of the second neural network array, and the second sub-deviation of the at least two sub-deviations corresponds to the output data of the third neural network array;

[0030] Based on the first sub-bias and the input data of the second neural network array, the weight values ​​stored in at least one of the memory units in the second neural network array are adjusted;

[0031] Based on the second sub-bias and the input data of the third neural network array, the weight values ​​stored in at least one of the storage units in the third neural network array are adjusted.

[0032] In another possible implementation of the second aspect, the adjustment module is specifically used to: determine the number of pulses based on the updated weight values ​​in the storage unit, and rewrite the weight values ​​stored in at least one of the storage units in the neural network array based on the number of pulses.

[0033] The beneficial effects of the second aspect and any possible implementation of the second aspect correspond to the beneficial effects of the first aspect and any possible implementation of the first aspect, which will not be elaborated further.

[0034] Thirdly, a neural network system is provided, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, such that the neural network system performs the method provided in the first aspect or any possible implementation thereof.

[0035] Optionally, the number of processors is not limited in the specific implementation. The processor can be a general-purpose processor, implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0036] Fourthly, a chip is provided on which a neural network system as described in the second aspect or any possible implementation thereof is disposed.

[0037] The chip includes a processor and a data interface, wherein the processor reads instructions stored in memory through the data interface to execute the method in the first aspect or any possible implementation of the first aspect. In specific implementation, the chip can be implemented in the form of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessor (MPU), a digital signal processor (DSP), a system on chip (SoC), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a programmable logic device (PLD).

[0038] Fifthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0039] Sixthly, a computer-readable storage medium is provided, which stores computer program code that, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Such computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of a neural network system 100 provided in this application.

[0041] Figure 2 A schematic diagram of the structure of another neural network system 200 provided in this application.

[0042] Figure 3 This is a schematic diagram of the mapping relationship between neural networks and neural network arrays.

[0043] Figure 4 This is a schematic diagram of a possible weight matrix provided in this application.

[0044] Figure 5 This is a schematic diagram of a possible neural network model.

[0045] Figure 6 This is a schematic diagram of a neural network system provided in this application.

[0046] Figure 7 This is a schematic diagram of the input and output data structure of the multiple memristor arrays for parallel computing provided in this application.

[0047] Figure 8 (a) is a multi-memristor array provided in this application for accelerating parallel computing of input data.

[0048] Figure 8 (b) is a schematic diagram of a specific data splitting method provided in this application.

[0049] Figure 9 This application provides another type of multi-group memristor array for accelerating parallel computation of input data.

[0050] Figure 10 This is a schematic flowchart of a data processing method in a neural network system provided in this application.

[0051] Figure 11 This is a schematic diagram of a forward and reverse operation process provided in this application.

[0052] Figure 12 This is a schematic diagram provided in this application of updating the weight values ​​stored in the first memristor array that implements fully connected layer calculation in multiple memristor arrays.

[0053] Figure 13 This is a schematic diagram of updating the weight values ​​stored in the first memristor array that performs fully connected layer calculations in multiple memristor arrays, as provided in this application.

[0054] Figure 14 This is a schematic diagram illustrating the updating of weight values ​​stored in multiple memristor arrays used to perform convolutional layer computation.

[0055] Figure 15 This is a schematic diagram illustrating how to update the weight values ​​stored in multiple memristor arrays that perform convolutional layer computations based on the residual values.

[0056] Figure 16 This is another schematic diagram illustrating the updating of weight values ​​stored in multiple memristor arrays used to perform convolutional layer computations.

[0057] Figure 17 This is another schematic diagram illustrating how to update the weight values ​​stored in multiple memristor arrays that perform convolutional layer computations based on the residual values.

[0058] Figure 18 This is a schematic diagram of an application that provides for increasing the weight values ​​stored in at least one memory unit in a neural network array.

[0059] Figure 19 This is a schematic diagram of reducing the weight values ​​stored in at least one memory unit in a neural network array, as provided in this application.

[0060] Figure 20 This is a schematic diagram of an application that provides a method for increasing the weight values ​​stored in at least one storage unit in a neural network array by reading and writing simultaneously.

[0061] Figure 21This is a schematic diagram of a method provided in this application to reduce the weight values ​​stored in at least one memory unit in a neural network array by reading and writing simultaneously.

[0062] Figure 22 This is a schematic flowchart illustrating the training process of a neural network provided in an embodiment of this application.

[0063] Figure 23 This is a schematic structural diagram of a neural network system 2300 provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0065] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0066] In the field of AI, deep learning is a learning technique based on deep artificial neural network (ANN) algorithms. An artificial neural network (ANN), often simply called a neural network (NN) or a neural network-like structure, is a mathematical or computational model in machine learning and cognitive science that mimics the structure and function of biological neural networks (the central nervous system of animals, especially the brain) to estimate or approximate functions. Artificial neural networks can include convolutional neural networks (CNNs), multilayer perceptrons (MLPs), recurrent neural networks (RNNs), and other types of neural networks.

[0067] The training process of a neural network is essentially the process of learning parameter matrices. Its ultimate goal is to obtain the parameter matrix of each neuron in the trained neural network (each layer's parameter matrix includes the weights of each neuron within that layer). The parameter matrices formed by the weights obtained through training can extract pixel information from the user-input image to be reasoned about, thereby helping the neural network to make correct inferences about that image and ensuring that the predicted value output by the trained neural network is as close as possible to the prior knowledge of the training data.

[0068] It should be understood that prior knowledge, also known as ground truth, generally includes the true results corresponding to the training data provided by the user.

[0069] The training process of the aforementioned neural network is a data-centric task that requires computing hardware with high-performance, low-power processing capabilities. Computation based on the traditional von Neumann architecture, due to the separation of storage and computation units, requires extensive data movement and cannot achieve high-energy-efficiency processing.

[0070] The following is combined Figures 1-2 Describe the system architecture diagram of this application.

[0071] Figure 1 This is a schematic diagram of the structure of a neural network system 100 provided in an embodiment of this application. Figure 1 As shown, the neural network system 100 may include a host 105 and a neural network circuit 110.

[0072] The neural network circuit 110 is connected to the host 105 via a host interface. The host interface may include a standard host interface and a network interface. For example, the host interface may include a peripheral component interconnect express (PCIe) interface.

[0073] As an example, such as Figure 1 As shown, the neural network circuit 110 can be connected to the host 105 via the PCIe bus 106. Therefore, data is input to the neural network circuit 110 via the PCIe bus 106, and processed data is received from the neural network circuit 110 via the PCIe bus 106. Furthermore, the host 105 can monitor the operating status of the neural network circuit 110 through the host interface.

[0074] The host 105 may include a processor 1052 and memory 1054. It should be noted that, in addition to... Figure 1In addition to the devices shown, the host 105 may also include a communication interface and other devices such as a disk as external storage, without limitation.

[0075] Processor 1052 is the core of the host 105's processing and control unit. Processor 1052 may include multiple processor cores. Processor 1052 can be a very large-scale integrated circuit. An operating system and other software programs are installed on processor 1052, enabling it to control memory 1054, cache, disk, and peripheral devices (such as…). Figure 1 Access to the neural network circuit in the processor 1052. It is understood that, in the embodiments of this application, the core in the processor 1052 may be, for example, a central processing unit (CPU) or other application-specific integrated circuits (ASICs).

[0076] It should be understood that the processor 1052 in the embodiments of this application can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0077] Memory 1054 is the main memory of host 105. Memory 1054 is connected to processor 1052 via a double data rate (DDR) bus. Memory 1054 is typically used to store various running software, input and output data, and information exchanged with external storage in the operating system. To improve the access speed of processor 1052, memory 1054 needs to have a high access speed. In traditional computer system architectures, dynamic random access memory (DRAM) is typically used as memory 1054. Processor 1052 can access memory via the memory controller (…). Figure 1 (Not shown in the image) High-speed access memory 1054, performing read and write operations on any storage unit in memory 1054.

[0078] It should also be understood that the memory 1054 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0079] Figure 1 The neural network circuit 110 shown can be a chip array composed of multiple neural network chips and multiple routers 120. For ease of description, the neural network chip 115 is simply referred to as chip 115 in this embodiment. Multiple chips 115 are interconnected through routers 120. For example, one chip 115 can be connected to one or more routers 120. Multiple routers 120 can form one or more network topologies. Chips 115 can transmit data and exchange information through the various network topologies.

[0080] Figure 2 This is a schematic diagram of the structure of another neural network system 200 provided in an embodiment of this application. For example... Figure 2 As shown, the neural network system 200 may include a host 105 and a neural network circuit 210.

[0081] The neural network circuit 210 is connected to the host 105 via a host interface, such as... Figure 2As shown, the neural network circuit 210 can be connected to the host 105 via the PCIe bus 106. The host 105 may include a processor 1052 and memory 1054. For a detailed description of the host 105, please refer to [link to documentation]. Figure 2 The explanations in the text will not be repeated here.

[0082] Figure 2 The neural network circuit 210 shown can be a chip array composed of multiple chips 115, which are connected to a PCIe bus 106. The chips 115 exchange data and information with each other via the PCIe bus 106.

[0083] Optionally, Figures 1-2 The architecture of the neural network system described above is merely an example. Those skilled in the art will understand that in practice, the neural network system described above can include more than [other types of systems]. Figure 1 or Figure 2 The neural network system may contain more or fewer units. Alternatively, modules, units, or circuits in the neural network system may be replaced by other modules, units, or circuits with similar functions; this application does not limit this. For example, in other examples, the aforementioned neural network system may also be implemented by a graphics processing unit (GPU) or a field-programmable gate array (FPGA) based on digital computing.

[0084] In some examples, the neural network circuit described above can be implemented using multiple in-memory computing neural network matrices. Each of the multiple neural network matrices can include multiple in-memory computing units, each of which stores the weight values ​​of each neuron in the corresponding layer of the neural network, for the purpose of performing calculations on the neural network layers.

[0085] This application does not specifically limit the memory computing unit, which may include, but is not limited to: memristor, static RAM (SRAM), NOR Flash, magnetic RAM (MRAM), ferroelectric gate field-effect transistors (FeFET), and electrochemical RAM (ECRAM). Among these, memristors may include, but are not limited to: resistive random-access memory (ReRAM), conductive-bridging RAM (CBRAM), and phase-change memory (PCM).

[0086] For example, a neural network matrix is ​​a ReRAM crossbar composed of ReRAMs. The neural network system described above may include multiple ReRAM crossbars.

[0087] In the embodiments of this application, the ReRAM crossbar can also be referred to as a memristor cross array, a ReRAM device, or ReRAM. A chip that includes one or more ReRAM crossbars can be referred to as a ReRAM chip.

[0088] ReRAM crossbar is a novel non-von Neumann computing architecture. This architecture integrates storage and computation functions, features flexible configurability, and utilizes analog computing methods. It promises to achieve faster and more energy-efficient matrix-vector multiplication than traditional computing architectures, and has broad application prospects in neural network computing.

[0089] The following is combined Figure 3 Using ReRAM crossbar as an example, this paper describes in detail the specific implementation process of neural network layer computation through ReRAM crossbar.

[0090] Figure 3 This is a schematic diagram of the mapping relationship between neural networks and neural network arrays. Neural network 110 includes multiple neural network layers.

[0091] In this embodiment, the neural network layer is a logical layer concept; one neural network layer refers to one neural network operation performed. Each neural network computation is implemented by computational nodes (also called neurons). In practical applications, neural network layers may include convolutional layers, pooling layers, fully connected layers, etc.

[0092] Those skilled in the art will understand that when performing neural network computations (e.g., convolution computations), the computational nodes in a neural network system can calculate the input data along with the weights of the corresponding neural network layers. In a neural network system, weights are typically represented by a real-number matrix, where each element represents a weight value. Weights are generally used to indicate the importance of the input data to the output data. Figure 4 As shown, Figure 4 The m-row, n-column weight matrix shown can be a weight of a neural network layer, where each element represents a weight value.

[0093] Since the computation of each layer of the neural network can be implemented using ReRAM crossbars, and ReRAM offers the advantage of in-memory computation, weights can be configured onto multiple ReRAM cells of the ReRAM crossbar before computation. Therefore, matrix multiplication and addition operations between the input data and the configured weights can be performed using the ReRAM crossbars.

[0094] It should be understood that the ReRAM cell in the embodiments of this application can also be called a memristor cell. The configuration of weights in the memristor cell before computation can be understood as storing the weight values ​​of the corresponding neurons in the neural network within the memristor cell. Specifically, the weight values ​​of the neurons in the neural network can be indicated by the resistance or conductance value of the memristor cell.

[0095] It should also be understood that in practical applications, the ReRAM crossbar and the neural network layer can have a one-to-one mapping relationship, or it can also have a one-to-many mapping relationship. A detailed description will follow with accompanying diagrams, and will not be repeated here.

[0096] To illustrate this clearly, the following is a brief description of how the ReRAM crossbar implements matrix multiplication and addition operations.

[0097] It should be noted that, Figure 3 The data processing process is illustrated using the first neural network layer in neural network 110 as an example. In a neural network system, the first neural network layer can be any layer in the system; for ease of description, the first neural network layer can also be simply referred to as "the first layer".

[0098] Figure 3 The ReRAM crossbar 120 shown is an m×n crossbar array, wherein the ReRAM crossbar 120 may include multiple memristor cells (e.g., G...). 1,1 G 1,2(etc.), the bit lines (BL) of each column of memristor cells are connected together, and the source lines (SL) of each row of memristor cells are connected together.

[0099] In this embodiment, the conductance value of a memristor can be used to represent the weight of a neuron in a neural network. Specifically, as an example, the weight of a neuron in a neural network can be represented by the conductance value of a memristor. Figure 4 Each element in the weight matrix shown is represented by the conductance of the memristor located at the intersection of BL and SL. For example, Figure 3 G 1,1 express Figure 4 The weight element W in 0,0 , Figure 3 G 1,2 express Figure 4 The weight element W in 0,1 wait.

[0100] Different conductance values ​​of memristor cells can indicate different weights of neurons in the neural network stored in the memristor cells.

[0101] During neural network computation, the voltage value applied to the memristor BL can represent n input data Vi, for example... Figure 3 In the memristor: V1, V2, V3, ..., Vn. Input data can be represented by voltages, allowing the input data applied to the memristor to be multiplied by the weight values ​​stored in the memristor, resulting in, for example: Figure 3 The m output data shown can be represented by the current of SL, for example... Figure 3 In the form: I1, I2, ..., Im.

[0102] It should be understood that there are various ways to implement the voltage value applied to the memristor, and the embodiments of this application do not specifically limit this. For example, it can be represented by the voltage pulse amplitude, or by the voltage pulse width, or by the voltage pulse number, or by a combination of the voltage pulse number and voltage pulse amplitude.

[0103] It should be noted that the above description, using a single neural network array as an example, details the process by which the array performs the corresponding multiplication and accumulation calculations within the neural network. In practical applications, multiple neural network arrays work together to complete the multiplication and accumulation calculations required for a full neural network.

[0104] One of a plurality of neural network arrays can correspond to a single neural network layer, and that array is used to perform the computation of that layer. Alternatively, multiple neural network arrays can correspond to a single neural network layer, and be used to perform the computation of that single layer. Or, one of a plurality of neural network arrays can correspond to multiple neural network layers, and be used to perform the computation of those multiple layers.

[0105] The following is combined Figures 5-6 The correspondence between neural network arrays and neural network layers is described in detail.

[0106] For ease of description, the following text uses a memristor array as an example of a neural network array.

[0107] Figure 5 This is a schematic diagram of a possible neural network model. This neural network model may include multiple neural network layers.

[0108] In this embodiment, the neural network layer is a logical layer concept, and one neural network layer refers to one neural network operation to be performed. Each neural network computation is implemented by computation nodes. Neural network layers may include convolutional layers, pooling layers, fully connected layers, etc.

[0109] like Figure 5 As shown, a neural network model can include n neural network layers (also known as an n-layer neural network), where n is an integer greater than or equal to 2. Figure 5 This shows a portion of the neural network layers in the neural network model, such as... Figure 5 As shown, the neural network model can include layers 302, 304, 306, 308, 310, and up to layer 312. Layer 302 can perform convolution operations; layer 304 can perform pooling or activation operations on the output data of layer 302; layer 306 can perform convolution operations on the output data of layer 304; layer 308 can perform convolution operations on the output of layer 306; and layer 310 can perform summation operations on the output data of layer 304 and layer 308. Layer 312 can perform fully connected layer operations.

[0110] It should be understood that the pooling or activation operations can be implemented by an external digital circuit module. Specifically, the external digital circuit module can be connected to the neural network circuit 110 via the PCIe bus 106. Figure 1 or Figure 2 (Not shown in the image).

[0111] Understandable Figure 5This is just a simple example and explanation of the neural network layers in a neural network model. It does not restrict the specific operations of each neural network layer. For example, the fourth layer 308 can also be a pooling operation, and the fifth layer 310 can also be a convolution operation or a pooling operation, or other neural network operations.

[0112] Figure 6 This is a schematic diagram of a neural network system provided in an embodiment of this application. Figure 6 As shown, the neural network system may include multiple memristor arrays. For example, a first memristor array, a second memristor array, a third memristor array, a fourth memristor array, etc.

[0113] The first memristor array can be used to perform calculations on fully connected layers in a neural network. Specifically, the weights of the fully connected layers in the neural network can be stored in the first memristor array. The conductance value of each memristor unit in the memristor array can be used to indicate the weights of the fully connected layers and to perform the multiplication and accumulation calculation process of the fully connected layers in the neural network.

[0114] It should be noted that a fully connected layer in a neural network can also correspond to multiple memristor arrays, and the computation of the fully connected layer can be completed by these multiple memristor arrays. This application does not impose specific limitations on this.

[0115] Figure 6 The multiple memristor arrays shown (e.g., the second, third, and fourth memristor arrays) can be used to perform convolutional layer computations in a neural network. For convolutional layer operations, each windowing operation of the convolution kernel introduces a new input, requiring the processing of different inputs for a single complete convolutional layer computation. Therefore, the parallelism of the neural network system can be increased by implementing weights at the same locations in the network using multiple memristor arrays, thereby accelerating parallel processing of different inputs. Specifically, the convolution weights at key locations are implemented using multiple memristor arrays. During computation, each memristor array (e.g., the second, third, and fourth memristor arrays) processes different input data in parallel, working concurrently to improve convolution computation efficiency and system performance.

[0116] It should be understood that a convolutional kernel represents a feature extraction method in the computation process of a neural network. For example, in image processing within a neural network system, given an input image, each pixel in the output image is a weighted average of pixels in a small region of the input image, where the weights are defined by a function called the convolutional kernel. During computation, the convolutional kernel sequentially traverses the input feature map with a certain stride to generate the feature-extracted output data (also called the output feature map). Therefore, the kernel size also indicates the amount of data processed by a computation node in a neural network system in a single computation. Those skilled in the art will recognize that a convolutional kernel can be represented by a real-number matrix, for example... Figure 8 Figure (a) shows a 3x3 convolutional kernel, where each element represents a weight value. In practical applications, a neural network layer can include multiple convolutional kernels. During neural network computation, the input data can be multiplied and added with the convolutional kernels.

[0117] The input data for multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) in parallel computing can include the output data of other memristor arrays or external input data. The output data of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can be used as the input data for a shared first memristor array. In other words, the input data for the first memristor array can include the output data of multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array).

[0118] The input and output data structures of multiple memristor arrays in parallel computing can be varied, and this application does not impose any specific limitations on them.

[0119] The following is combined Figures 7-9 The structure of the input and output data of multiple memristor arrays for parallel computing is described in detail.

[0120] Figure 7 This is a schematic diagram of the input and output data of the multiple memristor arrays for parallel computing provided in the embodiments of this application.

[0121] One possible implementation is, such as Figure 7 In the first method shown, the input data of multiple memristor arrays (e.g., the second memristor array, the third memristor array, and the fourth memristor array) in parallel computing are combined together to form a complete input data, and the output data of the multiple memristor arrays in parallel computing are combined together to form a complete output data.

[0122] For example, the input data for the second memristor array is data 1, the input data for the third memristor array is data 2, and the input data for the fourth memristor array is data 3. For a convolutional layer, a complete input data set includes a combination of data 1, data 2, and data 3. Similarly, the output data for the second memristor array is result 1, the output data for the third memristor array is result 2, and the output data for the fourth memristor array is result 3. For a convolutional layer, a complete output data set includes a combination of result 1, result 2, and result 3.

[0123] For details, see Figure 8 (a) can split an input image into different parts and input them separately into multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The combination of the outputs of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can be used as the complete output data corresponding to the input image.

[0124] Figure 8 (b) illustrates a possible image splitting diagram. Figure 8 (b) splits an image into three parts and feeds them into three parallel, accelerated arrays for computation. The first part is fed into... Figure 8 The second memristor array shown in (a) yields Figure 7 "Result 1" in Method 1 corresponds to the output of the second memristor array in the complete output. The second and third parts can be processed similarly. The overlapping portion between each part is determined based on the kernel size and window stride (e.g., two rows overlap between each part in this example), ensuring that the outputs of the three arrays together constitute the complete output. During training, when the complete residual of the layer is obtained, the residual value of the corresponding neuron and the first part of the input are calculated using the second memristor array according to the correspondence of the forward computation process, and the second memristor array is updated in situ. The updates for the second and third arrays are similar. The specific update process is described below and will not be repeated here.

[0125] Another possible implementation, such as Figure 7 In the second method shown, the input data of the multiple memristor arrays (e.g., the second memristor array, the third memristor array, and the fourth memristor array) in parallel computing is a complete input data, and the output data of the multiple memristor arrays in parallel computing is a complete output data.

[0126] For example, the input data of the second memristor array is data 1. For the convolutional layer, data 1 is a complete input data, and its output data is result 1, which is a complete output data. Similarly, the input data of the third memristor array is data 2. For the convolutional layer, data 2 is a complete input data, and its output data is result 2, which is a complete output data. The input data of the fourth memristor array is data 3. For the convolutional layer, data 3 is a complete input data, and its output data is result 3, which is a complete output data.

[0127] For details, see Figure 9 Multiple different complete input data can be input into multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The output results of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) each correspond to a complete output data.

[0128] If the in-memory units in a neural network array are affected by some non-ideal characteristics, such as device fluctuations, conductivity drift, array yield, etc., the in-memory unit cannot implement weights without loss, which leads to a decrease in the overall performance of the neural network system and a reduction in its recognition rate.

[0129] The technical solutions provided in this application can improve the performance of neural network systems and the accuracy of recognition.

[0130] The following is combined Figures 10-22 The method embodiments provided in this application will be described in detail below.

[0131] It should be noted that the technical solutions of this application embodiment can be applied to various neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) which are widely used in processing natural language and speech, and deep neural networks that combine CNNs and RNNs. The processing of CNNs is similar to that of an animal's visual system, making them highly suitable for image recognition. CNNs can be applied to various image recognition fields such as security, computer vision, and smart city initiatives, as well as speech recognition, search engines, machine translation, and so on. In practical applications, the huge number of parameters and computational demands pose a significant challenge to the application of neural networks in scenarios requiring high real-time performance and low power consumption.

[0132] Figure 10 This is a schematic flowchart illustrating a data processing method in a neural network system provided in an embodiment of this application. Figure 10As shown, the method may include steps 1010-1030, which will be described in detail below.

[0133] Step 1010: Input the training data into the neural network system to obtain the first output data.

[0134] The parallel-accelerated neural network system in this application embodiment may include multiple neural network arrays, and each of the multiple neural network arrays may include multiple storage units, each storage unit being used to store the weight values ​​of neurons in the corresponding neural network.

[0135] Step 1020: Calculate the deviation between the first output data and the target output data.

[0136] The target output data can be the ideal value of the first output data that is actually output.

[0137] The deviation in the embodiments of this application may be the difference between the first output data and the target output data, or it may be the residual between the first output data and the target output data, or it may be other forms of loss function between the first output data and the target output data.

[0138] Step 1030: Adjust the weight values ​​stored in at least one of the memory units in a portion of the neural network arrays in the parallel accelerated neural network system according to the deviation.

[0139] In the embodiments of this application, some neural network arrays can be used to implement the calculation of some neural network layers in the neural network system. That is, the correspondence between neural network arrays and neural network layers can be one-to-one, one-to-many, or many-to-one.

[0140] For example, Figure 6 The first memristor array shown corresponds to the fully connected layer in a neural network and is used to implement the computation of the fully connected layer. For example, Figure 6 The multiple memristor arrays shown (e.g., the second memristor array, the third memristor array, and the fourth memristor array) correspond to the convolutional layers in the neural network and are used to implement the computation of the convolutional layers.

[0141] It should be understood that a neural network layer is a logical layer concept; one neural network layer refers to the execution of one neural network operation. For details, please refer to [link to relevant documentation]. Figure 5 The explanations in the text will not be repeated here.

[0142] The resistance or conductance value in the memory cell can be used to indicate the weight value in the neural network layer. In this embodiment, the resistance or conductance value of at least one memory cell in a portion of the neural network array can be adjusted or rewritten according to the calculation deviation.

[0143] There are several ways to adjust or rewrite the resistance or conductance values ​​of a memory-based computing unit. One possible implementation is to determine the updated resistance or conductance value of the memory-based computing unit based on the deviation, and then apply a fixed number of programming pulses to the memory-based computing unit according to this updated value. Another possible implementation is to determine the updated resistance or conductance value of the memory-based computing unit based on the deviation, and then apply programming pulses to the memory-based computing unit through a read-write operation. Yet another possible implementation is to apply different numbers of programming pulses according to the characteristics of different memory-based computing units to adjust or rewrite their resistance or conductance values. These will be described in detail below with reference to specific embodiments, and will not be elaborated further here.

[0144] It should be noted that, in the embodiments of this application, the resistance or conductance values ​​of the neural network arrays used to implement fully connected layers in multiple neural network arrays can be adjusted by deviation; alternatively, the resistance or conductance values ​​of the neural network arrays used to implement convolutional layers in multiple neural network arrays can be adjusted by deviation; or the resistance or conductance values ​​of both the neural network arrays used to implement fully connected layers and the neural network arrays used to implement convolutional layers can be adjusted by deviation simultaneously. The following will combine... Figures 11-17 A detailed description will not be provided here.

[0145] For ease of description, the following example uses the calculation of the residual between the actual output value and the target output value. The calculation process of the residual will be described in detail first.

[0146] During forward propagation (FP) computation, a training dataset, such as pixel information from an input image, is acquired and fed into the neural network. After passing through the first layer to the last layer, the actual output value is obtained from the output of the last layer.

[0147] In backpropagation (BP) computation, the goal is for the actual output value of the neural network to be as close as possible to the prior knowledge of the training data. This prior knowledge, also known as the ground truth or ideal output value, typically includes predictions based on the training data provided by a human. Therefore, the residual value can be calculated by comparing the current actual output value with the ideal output value and then calculating the deviation between them. Specifically, this can be achieved by calculating the partial derivative of the target loss function. Based on the residual value, the required weights are calculated, and the weight values ​​stored in at least one memory unit in the neural network array can be updated accordingly.

[0148] As an example, the square of the difference between the actual output value and the ideal output value of the neural network can be calculated, and the derivative of that square with respect to the weights in the weight matrix can be used to obtain the residual value.

[0149] Based on the residual values ​​and the corresponding input data of the weight values ​​determined above, the weight size that needs to be updated is determined by formula (1).

[0150]

[0151] Where ΔW represents the weight size that needs to be updated, η represents the learning rate, N represents the number of input data sets, V represents the input data value of this layer, and δ represents the residual value of this layer.

[0152] For details, see Figure 11 ,like Figure 11 In the N x M array shown, SL represents the source line and BL represents the bit line.

[0153] In the forward operation, the voltage is input at BL and the current is output at SL to complete the matrix-vector multiplication calculation of Y = XW (X corresponds to the input voltage V and Y corresponds to the output current I). X is the input calculation data that can be used for forward inference.

[0154] In reverse operation, the input voltage is at SL, and the output current is at BL, i.e., Y = XW. T The calculation (X corresponds to the input voltage V, Y corresponds to the output current I), where X is the residual value, completes the backpropagation calculation of the residual value. The memristor array update operation (also known as in-situ update) can complete the process of weights changing according to the gradient direction.

[0155] Optionally, in some embodiments, for the cumulative updated weight obtained in the m-th row and n-th column of the layer, it can also be determined whether to update the weight value in the m-th row and n-th column of the layer according to the following formula (2).

[0156]

[0157] Among them, threshold represents the preset threshold.

[0158] The cumulative update weight ΔW obtained in the m-th row and n-th column of this layer m,n The threshold update rule shown in formula (2) is adopted, that is, weights that have not reached the threshold requirement are not updated. Specifically, if ΔW m,n If the value is greater than or equal to a preset threshold, the weight value in the m-th row and n-th column of that layer can be updated. If ΔW m,n If the value is less than the preset threshold, the weight value in the m-th row and n-th column of that layer will not be updated.

[0159] The following is combined Figures 12-13 Taking different data organization structures as examples, this paper describes in detail the specific implementation process of updating the weight values ​​stored in the first memristor array for fully connected layer computation.

[0160] Figure 12 This is a schematic diagram illustrating the updating of weight values ​​stored in the first memristor array that performs fully connected layer computation in multiple memristor arrays.

[0161] like Figure 12 As shown, the weights of pre-trained neural network layers can be written into multiple memristor arrays. That is, the weights of corresponding neural network layers are stored in multiple memristor arrays. For example, the first memristor array can implement the calculation of fully connected layers in the neural network. The weights of the fully connected layers in the neural network can be stored in the first memristor array, and the conductance value of each memristor unit in this array can be used to indicate the weights of the fully connected layers, thus implementing the multiplication and accumulation calculation process of the fully connected layers in the neural network. Similarly, multiple sets of memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can implement the calculation of convolutional layers in the neural network. By implementing weights at the same positions in the convolutional layers using multiple sets of memristor arrays, parallel acceleration of different inputs can be achieved.

[0162] like Figure 8 As shown in (a), an input image is split into different parts and fed into multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The output of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can be used as input data to a first memristor array, and the first output data is obtained through the first memristor array.

[0163] In this embodiment, the residual value can be calculated based on the first output data and the ideal output data according to the method described above. The weight values ​​stored in each memristor in the first memristor array that implements the fully connected layer calculation are then updated in situ according to formula (1).

[0164] Figure 13 This is another schematic diagram of updating the weight values ​​stored in the first memristor array that performs fully connected layer calculations in multiple memristor arrays.

[0165] like Figure 9 As shown, multiple different input data are input to multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The output results of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can be used as input data to a first memristor array, and the first output data is obtained through the first memristor array.

[0166] In this embodiment, the residual value can be calculated based on the first output data and the ideal output data according to the method described above. The weight values ​​stored in each memristor in the first memristor array that implements the fully connected layer calculation are then updated in situ according to formula (1).

[0167] The following is combined Figures 14-17 Taking different data organization structures as examples, this paper describes in detail the specific implementation process of updating the weight values ​​stored in multiple memristor arrays that perform convolutional layer computation in parallel.

[0168] Figure 14 This is a schematic diagram illustrating the updating of weight values ​​stored in multiple memristor arrays used to perform convolutional layer computation.

[0169] like Figure 8 As shown in (a), an input image is split into different parts and fed into multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The combination of the outputs of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) can be used as the complete output data corresponding to the input image.

[0170] In this embodiment, the residual value can be calculated based on the output data and the ideal output data according to the method for calculating the residual value described above. The weight values ​​stored in each memristor in the multiple memristor arrays (e.g., the second memristor array, the third memristor array, and the fourth memristor array) that implement convolutional layer computation in parallel are updated in situ according to formula (1). There are various specific implementation methods.

[0171] In one possible implementation, a residual value can be calculated based on the output values ​​of multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) and the corresponding ideal output values, and the weight values ​​stored in each memristor in the multiple memristor arrays that implement the convolutional layer computation in parallel can be updated in situ based on the residual value.

[0172] In another possible implementation, a residual value can be calculated based on the first output value of the first memristor array and the corresponding ideal output value. This residual value is then used to update the weight values ​​stored in each memristor in multiple memristor arrays (e.g., the second, third, and fourth memristor arrays) that perform convolutional layer computations in parallel. The following section combines... Figure 15 Provide a detailed description.

[0173] Figure 15 This is a schematic diagram illustrating how to update the weight values ​​stored in multiple memristor arrays that perform convolutional layer computations based on the residual values.

[0174] like Figure 15 As shown, the complete residual can be the residual value of a shared first memristor array, which is determined based on the output value of the first memristor array and the corresponding ideal output value. In the embodiments of this application, the complete residual can be divided into multiple sub-residuals, for example, residual 1, residual 2, and residual 3.

[0175] Each sub-residual corresponds to the output data of each memristor array in the multiple sets of memristor arrays computed in parallel. For example, residual 1 corresponds to the output data of the second memristor array, residual 2 corresponds to the output data of the third memristor array, and residual 3 corresponds to the output data of the fourth memristor array.

[0176] In this embodiment of the application, the weight values ​​stored in each memristor in the memristor array are updated in situ based on the input data and sub-residual of each memristor array in multiple memristor arrays, combined with formula (2).

[0177] Figure 16 This is another schematic diagram illustrating the updating of weight values ​​stored in multiple memristor arrays used to perform convolutional layer computations.

[0178] According to such Figure 9 The input data structure shown allows multiple different input data to be fed into multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) for parallel computation. The output results of these multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) each correspond to a complete output data.

[0179] In this embodiment, the residual value can be calculated based on the output data and the ideal output data according to the method for calculating the residual value described above. The weight values ​​stored in each memristor in the multiple memristor arrays (e.g., the second memristor array, the third memristor array, and the fourth memristor array) that implement convolutional layer computation in parallel are rewritten according to formula (1). There are various specific implementation methods.

[0180] In one possible implementation, a residual value can be calculated based on the output values ​​of multiple memristor arrays (e.g., a second memristor array, a third memristor array, and a fourth memristor array) and the corresponding ideal output values, and the weight values ​​stored in each memristor in the multiple memristor arrays that implement the convolutional layer computation in parallel can be updated in situ based on the residual value.

[0181] In another possible implementation, a residual value can be calculated based on the first output value of the first memristor array and the corresponding ideal output value. This residual value is then used to update the weight values ​​stored in each memristor in multiple memristor arrays (e.g., the second, third, and fourth memristor arrays) that perform convolutional layer computations in parallel. The following section combines... Figure 17 Provide a detailed description.

[0182] Figure 17 This is another schematic diagram illustrating how to update the weight values ​​stored in multiple memristor arrays that perform convolutional layer computations based on the residual values.

[0183] like Figure 17 As shown, the complete residual can be the residual value of the shared first memristor array, which is determined based on the output value of the first memristor array and the corresponding ideal output value. Since each group of memristor arrays participating in parallel acceleration processes the obtained complete output result, each group of memristor arrays can be updated based on the relevant complete residual data. Assuming that the above complete residual is obtained based on the output result 1 of the second memristor array, the weight values ​​stored in each memristor in the second memristor array can be updated in situ based on the complete residual and the input data 1 of the second memristor array using the above formula (1).

[0184] Optionally, in this embodiment, the weight values ​​stored in the preceding array of multiple memristor arrays that implement convolutional layer computation in parallel can be adjusted, and the residual values ​​of neurons in each layer can be calculated through backpropagation. For details, please refer to the method described above, which will not be repeated here. It should be understood that for the preceding neural network arrays of multiple memristor arrays that implement convolutional layer computation in parallel, the input data of these arrays can be the output data of the preceding memristor arrays, or raw data from external input, such as images, text, or speech; the output data of these arrays serves as the input data of the multiple memristor arrays that implement convolutional layer computation in parallel.

[0185] The above text passed Figures 11-17 The document describes adjusting the weight values ​​stored in the neural network matrix implementing fully connected layer computation, as well as adjusting the weight values ​​stored in multiple neural network matrices implementing convolutional layer computation in parallel. It should be understood that embodiments of this application can also simultaneously adjust the weight values ​​stored in both the neural network matrix implementing fully connected layer computation and the multiple neural network matrices implementing convolutional layer computation in parallel. The methods are similar and will not be repeated here.

[0186] The following is combined Figures 18-21 Taking writing target data to the target memristor cell located at the intersection of BL and SL as an example, the set operation and the reset operation are introduced.

[0187] The set operation is used to adjust the conductance of the memristor cell from low to high, and the reset operation is used to adjust the conductance of the memristor cell from high to low.

[0188] like Figure 18 Assuming the target conductance range of the target memristor cell can represent the target weight W 11 Writing W to the target memristor cell 11 During the process, if the current conductance of the target memristor cell is lower than the lower limit of the target conductance range, the conductance of the target memristor cell can be increased through the set operation. At this time, SL can be used... 11 Apply a gate voltage to the transistor in the target memristor cell to turn it on, thus putting the target memristor cell in a selected state. Simultaneously, ground the SL connected to the target memristor cell and the other BLs in the cross array. Then, apply a set pulse to the BL containing the target memristor cell to adjust the conductance of the target memristor cell.

[0189] like Figure 19 If the current conductance of the target memristor cell is higher than the upper limit of the target conductance range, the conductance of the target memristor cell can be reduced through a reset operation. In this case, a voltage is applied to the gate of the transistor in the target memristor cell that needs adjustment via a switch (SL), putting the target memristor cell in a selected state. Simultaneously, the BL connected to the target memristor cell and the other SLs in the cross array are grounded. Then, a reset pulse is applied to the SL containing the target memristor cell to adjust its conductance.

[0190] There are several ways to adjust the conductance of a target memristor cell. For example, a fixed number of programming pulses can be applied to the target memristor cell. Alternatively, programming pulses can be applied to the target memristor cell through a read-write operation. Furthermore, different numbers of programming pulses can be applied to different memristor cells to adjust their conductance values.

[0191] The following is based on set and reset operations, combined with Figures 20-21 The specific implementation method of applying programming pulses to the target memristor cell by reading and writing simultaneously is described in detail.

[0192] This application embodiment can write target data to the target memristor cell based on the incremental step program pulse (ISPP) strategy. Specifically, the ISPP strategy typically uses a "read verification-correction" method to adjust the conductance of the target memristor cell, so that the conductance of the target memristor cell is eventually adjusted to the target conductance corresponding to the target data.

[0193] See Figure 20 Device 1, Device 2, ... are the target memristor cells in the selected memristor array. First, a read pulse (V) can be applied to the target memristor cell. read The current conductance of the target memristor cell is read. It is compared to the target conductance; if the current conductance is less than the target conductance, a set pulse (Vset) can be applied to the target memristor cell. set This increases the conductance of the target memristor cell. Then, a read pulse (V) is used. read Read the adjusted conductance. If the current conductance is still less than the target conductance, then apply a set pulse (V) to the target memristor cell. set This allows the electrical conductance of the target memristor unit to be adjusted.

[0194] See Figure 21 Device 1, Device 2, ... are the target memristor cells in the selected memristor array. First, a read pulse (V) can be applied to the target memristor cell. read The current conductance of the target memristor cell is read. It is compared to the target conductance; if the current conductance is greater than the target conductance, a reset pulse (V0) can be applied to the target memristor cell. reset This reduces the conductance of the target memristor cell. Then, a read pulse (V) is used. read Read the adjusted conductance. If the current conductance is still greater than the target conductance, then apply a set pulse (V) to the target memristor cell. reset This allows the electrical conductance of the target memristor unit to be adjusted.

[0195] It should be understood that V read It can be a read voltage pulse that is less than the threshold voltage, V set or V reset It can be a read voltage pulse that is greater than the threshold voltage.

[0196] The embodiments of this application can adjust the conductance of the target memristor cell to the target conductance corresponding to the target data through the above-described read-write method. Optionally, the termination condition can be that the conductance increase of all selected devices in the row meets the requirement.

[0197] Figure 22 This is a schematic flowchart illustrating the training process of a neural network provided in an embodiment of this application. Figure 22 As shown, the method may include steps 2210-2255, which will be described in detail below.

[0198] Step 2210: Determine the network layers that need to be accelerated based on the neural network information.

[0199] In this embodiment of the application, the network layer that needs to be accelerated can be determined based on one or more of the following: the number of layers in the neural network, parameter information, the size of the training dataset, etc.

[0200] Step 2215: Determine the initial training weights by performing offline training on an external personal computer (PC).

[0201] The weight parameters on the neurons of a neural network can be trained on an external PC through steps such as forward computation and backward computation to determine the initial training weights.

[0202] Step 2220: Map the initial training weights to the neural network arrays that implement parallel accelerated network layer computation and the neural network arrays that implement non-parallel accelerated network layer computation in the in-memory computing architecture.

[0203] In the embodiments of this application, it can be based on Figure 3 The method shown maps the initial training weights to at least one memory unit of multiple neural network arrays in a memory-computing architecture, thereby enabling matrix multiplication and addition operations between the input data and the configured weights through the neural network array.

[0204] Multiple neural network arrays may include neural network arrays that implement non-parallel accelerated network layer computation and neural network arrays that implement parallel accelerated network layer computation.

[0205] Step 2225: Input a set of training data into multiple neural network arrays of the in-memory computing architecture to obtain the output results of forward computation based on the actual hardware of the in-memory computing architecture.

[0206] Step 2230: Determine whether the accuracy of the neural network system meets the requirements or whether the preset number of training iterations has been reached.

[0207] If the accuracy of the neural network system meets the requirements or the preset number of training iterations is reached, step 2235 can be executed.

[0208] If the accuracy of the neural network system does not meet the requirements or the preset number of training iterations is not reached, step 2240 can be executed.

[0209] Step 2235: Training ends.

[0210] Step 2240: Determine if the training data is the last set of training data.

[0211] If the training data is the last set of training data, you can execute steps 2245 and 2255.

[0212] If the training data is not the last set of training data, you can proceed with steps 2250-2255.

[0213] Step 2245: Reload the training data.

[0214] Step 2250: Based on the training method of the proposed parallel training in-memory computing system, the conductance weights of the parallel acceleration array or other arrays are trained and updated on-chip in situ after backpropagation and other calculations.

[0215] For specific update methods, please refer to the instructions above; they will not be repeated here.

[0216] Step 2255: Load the next set of training data.

[0217] After loading the next set of training data, the operation in step 2225 continues. That is, the loaded training data is input into multiple neural network arrays of the in-memory computing architecture to obtain the output results of the forward computation based on the actual hardware of the in-memory computing architecture.

[0218] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0219] The above text combined Figures 1 to 22 This application describes in detail a data processing method in a neural network system provided by an embodiment of the present application. The following is a detailed description in conjunction with... Figure 23 Embodiments of the apparatus of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0220] It should be noted that the device embodiments are implementations of the solution in this application from the perspective of products and equipment. The device embodiments of this application correspond to or complement parts of the method embodiments described above. They are generally applicable in terms of solution implementation and support for the scope of the claims.

[0221] The following is combined Figure 23 This application describes an apparatus-side embodiment.

[0222] Figure 23 This is a schematic structural diagram of a neural network system 2300 provided in an embodiment of this application. It should be understood that... Figure 23 The neural network system 2300 shown is merely an example; the apparatus in this embodiment may also include other modules or units. It should be understood that the neural network system 2300 is capable of performing… Figures 10-22 To avoid repetition, the steps in the method will not be described in detail here.

[0223] like Figure 23 As shown, the neural network system 2300 may include:

[0224] The processing module 2310 is used to input training data into a neural network system to obtain first output data. The neural network system includes multiple neural network arrays, and each of the multiple neural network arrays includes multiple storage units. Each storage unit is used to store the weight values ​​of the neurons in the corresponding neural network.

[0225] Calculation module 2320 is used to calculate the deviation between the first output data and the target output data;

[0226] The adjustment module 2330 is used to adjust the weight values ​​stored in at least one of the computing units in a portion of the plurality of neural network arrays according to the deviation, wherein the portion of the neural network array is used to implement the calculation of a portion of the neural network layers in the neural network system.

[0227] Optionally, in one possible implementation, the plurality of neural network arrays includes a first neural network array and a second neural network array, wherein the input data of the first neural network array includes the output data of the second neural network array.

[0228] In another possible implementation of the second aspect, the first neural network array includes a neural network array for implementing fully connected layer computations in the neural network.

[0229] Alternatively, in another possible implementation, the adjustment module 2330 is specifically used for:

[0230] Based on the input value of the first neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the first neural network array are adjusted.

[0231] Alternatively, in another possible implementation, the plurality of neural network arrays further include a third neural network array, which and the second neural network array are used to perform the computation of convolutional layers in the neural network in parallel.

[0232] Alternatively, in another possible implementation,

[0233] The adjustment module 2330 is specifically used for:

[0234] Based on the input data of the second neural network array and the deviation, the weight values ​​stored in at least one of the storage units in the second neural network array are adjusted;

[0235] Based on the input data of the third neural network array and the deviation, the weight values ​​stored in at least one of the storage units in the third neural network array are adjusted.

[0236] In another possible implementation of the second aspect,

[0237] The adjustment module 2330 is specifically used for:

[0238] The deviation is divided into at least two sub-deviations, wherein the first sub-deviation of the at least two sub-deviations corresponds to the output data of the second neural network array, and the second sub-deviation of the at least two sub-deviations corresponds to the output data of the third neural network array;

[0239] Based on the first sub-bias and the input data of the second neural network array, the weight values ​​stored in at least one of the storage units in the second neural network array are adjusted;

[0240] Based on the second sub-bias and the input data of the third neural network array, the weight values ​​stored in at least one of the storage units in the third neural network array are adjusted.

[0241] Alternatively, in another possible implementation, the adjustment module 2330 is specifically used to: determine the number of pulses based on the updated weight values ​​in the storage unit, and rewrite the weight values ​​stored in at least one of the storage units in the neural network array based on the number of pulses.

[0242] It should be understood that the neural network system 2300 here is embodied in the form of functional modules. The term "module" here can be implemented in software and / or hardware, without specific limitation. For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. When any of the above modules is implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions to implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it can form a system-on-chip (SoC) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits). Alternatively, it can be integrated as a built-in processor within an application-specific integrated circuit (ASIC). This ASIC with integrated processor can be packaged separately or together with other circuits. In addition to the core for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.

[0243] When the above modules are implemented in hardware circuits, the hardware circuits may be implemented in the form of a general-purpose central processing unit (CPU), microcontroller unit (MCU), microprocessor (MPU), digital signal processor (DSP), system on chip (SoC), application-specific integrated circuit (ASIC), or programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. It can run the necessary software or execute the above method flow independently of the software.

[0244] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0245] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0246] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0247] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0248] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0249] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0251] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0252] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0253] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0254] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for data processing in a neural network system, characterized in that, include: Training data is input into a neural network system to obtain first output data. The neural network system includes multiple neural network arrays, each of which includes multiple storage units. Each storage unit is used to store the weight values ​​of neurons in the corresponding neural network. The multiple neural network arrays include a first neural network array, a second neural network array, and a third neural network array. The input data of the first neural network array includes the output data of the second neural network array. The third neural network array and the second neural network array are used to perform the calculation of convolutional layers in the neural network in parallel. Calculate the deviation between the first output data and the target output data; The weight values ​​stored in at least one of the storage units in a subset of the plurality of neural network arrays are adjusted according to the deviation, wherein the subset of neural network arrays is used to perform calculations on a subset of neural network layers in the neural network system. The step of adjusting the weight values ​​stored in at least one of the memory units in a subset of the plurality of neural network arrays based on the deviation includes: The deviation is divided into at least two sub-deviations, wherein the first sub-deviation of the at least two sub-deviations corresponds to the output data of the second neural network array, and the second sub-deviation of the at least two sub-deviations corresponds to the output data of the third neural network array; Based on the first sub-bias and the input data of the second neural network array, the weight values ​​stored in at least one of the storage units in the second neural network array are adjusted; Based on the second sub-bias and the input data of the third neural network array, the weight values ​​stored in at least one of the storage units in the third neural network array are adjusted.

2. The method according to claim 1, characterized in that, The first neural network array includes a neural network array for implementing fully connected layer computations in the neural network.

3. The method according to claim 2, characterized in that, The step of adjusting the weight values ​​stored in at least one of the memory units in a subset of the plurality of neural network arrays based on the deviation includes: Based on the input value of the first neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the first neural network array are adjusted.

4. A neural network system, characterized in that, include: A processing module is used to input training data into a neural network system to obtain first output data. The neural network system includes multiple neural network arrays, each of which includes multiple storage units. Each storage unit is used to store the weight values ​​of neurons in the corresponding neural network. The multiple neural network arrays include a first neural network array, a second neural network array, and a third neural network array. The input data of the first neural network array includes the output data of the second neural network array. The third neural network array and the second neural network array are used to perform the calculation of convolutional layers in the neural network in parallel. The calculation module is used to calculate the deviation between the first output data and the target output data; An adjustment module is used to adjust the weight values ​​stored in at least one of the storage units in a portion of the plurality of neural network arrays according to the deviation, wherein the portion of the neural network array is used to perform calculations on a portion of the neural network layers in the neural network system. The adjustment module is specifically used for: The deviation is divided into at least two sub-deviations, wherein the first sub-deviation of the at least two sub-deviations corresponds to the output data of the second neural network array, and the second sub-deviation of the at least two sub-deviations corresponds to the output data of the third neural network array; Based on the first sub-bias and the input data of the second neural network array, the weight values ​​stored in at least one of the storage units in the second neural network array are adjusted; Based on the second sub-bias and the input data of the third neural network array, the weight values ​​stored in at least one of the storage units in the third neural network array are adjusted.

5. The neural network system according to claim 4, characterized in that, The first neural network array includes a neural network array for implementing fully connected layer computations in the neural network.

6. The neural network system according to claim 5, characterized in that, The adjustment module is specifically used for: Based on the input value of the first neural network array and the deviation, the weight values ​​stored in at least one of the memory units in the first neural network array are adjusted.

7. A neural network system, characterized in that, include: Memory, used to store programs; A processor for calling and running the program from the memory to perform the method of any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 3.

9. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 3.

Images