Data processing device and data processing method
The integration of storage and computing in a memristor array-based data processing device addresses the computing power demands of neural networks, enabling efficient inference and training in resource-constrained environments.
Patent Information
- Application Number
- CN202111131563.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-26
AI Technical Summary
Existing artificial intelligence technologies based on neural network algorithms require high computing power, making them unsuitable for deployment in resource-constrained environments such as mobile devices and IoT devices due to power consumption and computing efficiency limitations.
A data processing device integrating storage and computing capabilities using a memristor array, with a bidirectional data processing module, control module, parameter management module, and input-output module, enabling both inference and training computing tasks with flexible hardware support.
The solution provides efficient and low-power computing for neural network algorithms, supporting both inference and training tasks in a wide range of scenarios, including resource-constrained environments.
Smart Images

Figure CN113837373B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to a data processing device and a data processing method. Background Art
[0002] At present, artificial intelligence technology based on neural network algorithms has demonstrated powerful capabilities in many application scenarios in daily life, such as speech processing, target recognition and detection, image processing, natural language processing, etc. However, due to the characteristics of the algorithm itself, the algorithm places high demands on the computing power of the hardware. Due to the design characteristics of the separation of storage and computing, traditional processing devices cannot effectively meet the needs of artificial intelligence applications in specific scenarios in terms of power consumption and computing efficiency. At present, large-scale neural network algorithms require computing clusters with powerful computing capabilities to achieve better performance, and thus cannot be effectively deployed in scenarios with limited resources such as volume and power supply, such as mobile electronic devices, Internet of Things devices, and edge devices. Summary of the invention
[0003] Some embodiments of the present disclosure provide a data processing device, including: a bidirectional data processing module, including at least one computing array integrating storage and computing, configured to execute computing tasks, wherein the computing tasks include inference computing tasks and training computing tasks; a control module, configured to switch the working mode of the bidirectional data processing module to an inference working mode to execute inference computing tasks, and to switch the working mode of the bidirectional data processing module to a training working mode to execute training computing tasks; a parameter management module, configured to set weight parameters of the bidirectional data processing module; an input-output module, configured to generate a computing input signal according to the input data of the computing task in response to the control of the control module, and provide the computing input signal to the bidirectional data processing module, receive a computing output signal from the bidirectional data processing module and generate output data according to the computing output signal.
[0004] For example, in a data processing device provided in some embodiments of the present disclosure, the computing array includes a memristor array for realizing the storage and computing integration, and the memristor array includes a plurality of memristors arranged in an array.
[0005] For example, in a data processing device provided by some embodiments of the present disclosure, the parameter management module includes: a weight array write unit, configured to change the conductance value of each memristor in the multiple memristors by using the weight parameters to write the weight parameters into the memristor array; and a weight array read unit, configured to read the conductance value of each memristor in the multiple memristors from the memristor array to complete the reading of the weight parameters.
[0006] For example, in a data processing device provided in some embodiments of the present disclosure, the input-output module includes: a first input submodule, connected to the first connection end side of the bidirectional data processing module to provide an input signal for the first input data of the inference calculation task; a first output submodule, connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference calculation task and generate first output data; a second input submodule, connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task; and a second output submodule, connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
[0007] For example, in a data processing device provided in some embodiments of the present disclosure, the first input submodule includes: a first data buffer unit; a first digital-to-analog signal converter; a first multiplexer, wherein the first data buffer unit is configured to receive the first input data and provide the first input data to the first digital-to-analog signal converter, the first digital-to-analog signal converter is configured to perform digital-to-analog conversion on the first input data and provide the first input signal output by the conversion to the first multiplexer, the first multiplexer is configured to provide the first input signal to the first connection end side of the bidirectional data processing module through a selected channel, the first output submodule includes: a second multiplexer; a first sampling and holding unit; a second analog-to-digital signal converter; a first shift accumulation unit; a second data buffer unit, wherein the second multiplexer is configured to receive the first output signal from the second connection end side of the bidirectional data processing module and provide the first output signal to the first sampling and holding unit through a selected channel, the first sampling and holding unit is configured to sample the first output signal and then provide the sampled first output signal to the first sampling and holding unit The output signal is provided to the second analog-to-digital signal converter, the second analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled first output signal, and provide the first output data of the conversion output to the first shift-accumulation unit, the first shift-accumulation unit is configured to provide the first output data to the second data buffer unit, the second data buffer unit is configured to output the first output data, the second input submodule includes: a third data buffer unit; a third digital-to-analog signal converter; a third multiplexer, wherein the third data buffer unit is configured to receive the second input data and provide the second input data to the third digital-to-analog signal converter, the third digital-to-analog signal converter is configured to perform digital-to-analog conversion on the second input data and provide the second input signal of the conversion output to the third multiplexer, the third multiplexer is configured to provide the second input signal to the second connection end side of the bidirectional data processing module through the selected channel, the second output submodule includes: a fourth multiplexer; a second sampling and holding unit; a fourth analog-to-digital signal converter; a second shift-accumulation unit;A fourth data buffer unit, wherein the fourth multiplexer is configured to receive the second output signal from the first connection end side of the bidirectional data processing module, and provide the second output signal to the second sampling and holding unit through the selected channel, the second sampling and holding unit is configured to sample the second output signal and provide the sampled second output signal to the fourth analog-to-digital signal converter, the fourth analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled second output signal, and provide the converted second output data to the second shift-accumulate unit, the second shift-accumulate unit is configured to provide the second output data to the fourth data buffer unit, and the fourth data buffer unit is configured to output the second output data. ;
[0008] For example, in a data processing device provided in some embodiments of the present disclosure, the control module is configured as follows: in the reasoning working mode, the first input submodule is connected to the first connection end side of the bidirectional data processing module to provide an input signal for the first input data of the reasoning calculation task, and the first output submodule is connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the reasoning calculation task and generate first output data; and, in the training working mode, the second input submodule is connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task, and the second output submodule is connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
[0009] For example, in a data processing device provided in some embodiments of the present disclosure, the input-output module includes: a first input-output sub-module, connected to the first connection end side of the bidirectional data processing module to provide a first input signal based on the first input data of the inference calculation task, and connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data; a second input-output sub-module, connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task, and connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference calculation task and generate first output data.
[0010] For example, in a data processing device provided in some embodiments of the present disclosure, the first input-output submodule includes: a first data buffer unit; a first shift-and-accumulate unit; a first digital-to-analog signal converter; a first analog-to-digital signal converter; a first sampling and holding unit; and a first multiplexer, wherein the first data buffer unit is configured to receive the first input data and provide the first input data to the first digital-to-analog signal converter, the first digital-to-analog signal converter is configured to perform digital-to-analog conversion on the first input data and provide the converted first input signal to the first multiplexer, the first multiplexer is configured to provide the first input signal to the first connection end side of the bidirectional data processing module through a selected channel, and the first multiplexer is configured to receive the first input signal from the first end side of the bidirectional data processing module. The second output signal is received, and the second output signal is provided to the first sampling and holding unit through the selected channel. The first sampling and holding unit is configured to sample the second output signal and provide the sampled second output signal to the first analog-to-digital signal converter. The first analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled second output signal and provide the second output data of the conversion output to the first shift-accumulation unit. The first shift-accumulation unit is configured to provide the second output data to the first data buffer unit. The first data buffer unit is configured to output the second output data. The second input-output submodule includes: a second multiplexer; a second sampling and holding unit; a second digital-to-analog signal converter; a second analog-to-digital signal converter; a second shift-accumulation unit; and a second data buffer unit. Wherein, the second data buffer unit is configured to receive the second input data and provide the second input data to the second digital-to-analog signal converter, the second digital-to-analog signal converter is configured to perform digital-to-analog conversion on the second input data and provide the second input signal output by the conversion to the second multiplexer, the second multiplexer is configured to provide the second input signal to the second connection end side of the bidirectional data processing module through a selected channel, and the second multiplexer is configured to provide the first output signal from the second connection end side of the bidirectional data processing module to the second sampling and holding unit through a selected channel, the second sampling and holding unit is configured to sample the first output signal and provide the sampled first output signal to the second analog-to-digital signal converter, the second analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled first output signal and provide the first output data output by the conversion to the second shift-accumulation unit, the second shift-accumulation unit is configured to provide the first output data to the second data buffer unit, and the second data buffer unit is configured to output the first output data.
[0011] For example, in a data processing device provided in some embodiments of the present disclosure, the control module is configured to: in response to the reasoning working mode, connect the first input-output submodule to the first connection end side of the bidirectional data processing module to provide a first input signal based on the first input data of the reasoning calculation task, and connect the second input-output submodule to the second connection end side of the bidirectional data processing module to receive the calculation result of the reasoning calculation task and generate first output data; and, in response to the training working mode, connect the second input-output submodule to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task, and connect the first input-output submodule to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
[0012] For example, in a data processing device provided in some embodiments of the present disclosure, it also includes: a multiplexing unit selection module, which is configured to, under the control of the control module, in response to the inference working mode, select the first data buffer unit, the first digital-to-analog signal converter, and the first multiplexer for input, and select the second multiplexer, the second sampling and holding unit, the second analog-to-digital signal converter, the second shift and accumulation unit, and the second data buffer unit for output; in response to the training working mode, select the second data buffer unit, the second digital-to-analog signal converter, the second multiplexer for input, and the first multiplexer, the first sampling and holding unit, the first analog-to-digital signal converter, the first shift and accumulation unit, and the first data buffer unit for output.
[0013] For example, in a data processing device provided in some embodiments of the present disclosure, it also includes: a processing unit interface module, which is configured to communicate with an external device outside the data processing device.
[0014] For example, in a data processing device provided in some embodiments of the present disclosure, it also includes: a functional unit configured to provide nonlinear operation to the output data.
[0015] Some embodiments of the present disclosure provide a data processing method for use in any of the above-mentioned data processing devices, comprising: the control module obtains the current working mode and controls the bidirectional data processing module; in response to the working mode being the inference working mode, the bidirectional data processing module uses the inference weight parameters for performing the inference computing task to perform the inference computing task; in response to the working mode being the training working mode, the bidirectional data processing module uses the training weight parameters for performing the training computing task to perform the training computing task.
[0016] For example, in a data processing method provided by some embodiments of the present disclosure, the execution of the inference computing task includes: receiving the first input data and generating a first computing input signal from the first input data; performing a storage and computing integration operation on the first computing input signal, and outputting a first computing output signal; generating the first output data according to the first computing output signal; and the bidirectional data processing module executes the training computing task including: receiving the second input data and generating a second computing input signal from the second input data; performing a storage and computing integration operation on the second computing input signal, and outputting a second computing output signal; generating the second output data according to the second computing output signal. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.
[0018] Figure 1A Schematic diagram of matrix-vector multiplication;
[0019] Figure 1B Schematic diagram of a memristor array used to perform matrix-vector multiplication;
[0020] Figure 2 is a schematic diagram of a data processing device that deploys a neural network algorithm to perform inference calculations;
[0021] Figure 3 yes Figure 2 A flowchart of a data processing method for performing inference calculations by a data processing device shown;
[0022] Figure 4 is a schematic diagram of a data processing device provided by at least one embodiment of the present disclosure;
[0023] Figure 5 is a flow chart of a data processing method provided by at least one embodiment of the present disclosure;
[0024] Figure 6 is a schematic diagram of another data processing device provided by at least one embodiment of the present disclosure;
[0025] Figure 7 is a flowchart of another data processing method provided by at least one embodiment of the present disclosure;
[0026] Figure 8 is a flowchart of another data processing method provided by at least one embodiment of the present disclosure;
[0027] Fig. 9is a schematic diagram of a data scheduling process for a plurality of data processing devices;
[0028] Fig.10 is a schematic diagram of a data processing system provided by at least one embodiment of the present disclosure;
[0029] Fig.11 for Fig.10 The data flow diagram of the data processing system shown in the figure performs an inference computing task;
[0030] Fig.12 for Fig.10 The data flow diagram of the data processing system shown in the figure performs a training computing task; and
[0031] Fig.13 for Fig.10 The data flow diagram of the data processing system shown executes layer-by-layer training computing tasks. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0033] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0034] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, the detailed description of known functions and known components (elements) may be omitted. When any part (element) of the embodiments of the present disclosure appears in more than one figure, the part (element) is represented by the same or similar reference numeral in each figure.
[0035] Currently, the core computational steps of most neural network algorithms consist of a large number of matrix-vector multiplications. Figure 1A is a schematic diagram of matrix-vector multiplication. Figure 1A As shown, the matrix G is multiplied by the column vector V to obtain the column vector I. Each element I1, I2, ..., In of the column vector I is obtained by multiplying the corresponding row element of the matrix G by the column vector V by the vector inner product. Taking the first row of the matrix G multiplied by the column vector V to obtain the first element I1 of the column vector as an example, each element of the n elements G11, G12, ..., G1n of the first row of the matrix G is multiplied by each element of the n elements V1, V2, ..., Vn of the column vector V to obtain the first element I1 corresponding to the column vector I by adding the n products. The calculation method of each element of the other elements I2, ..., In of the column vector I is inferred from the calculation method of the element I1.
[0036] Crossbar arrays based on non-volatile memory devices such as memristor arrays can perform matrix-vector multiplication operations very efficiently. Figure 1B Schematic diagram of an exemplary memristor array for performing matrix-vector multiplication. Figure 1B As shown, the memristor array includes n bit lines (Bit Line, BL) BL1, BL2, ..., BLn that cross but are insulated from each other; n word lines (Word Line, WL) WL1, WL2, ..., WLn and n source lines (Source Line, SL) SL1, SL2, ..., SLn. For example, the intersection of a word line and a bit line intersects with a source line, and a memristor and a transistor are arranged at the intersection, one end of the memristor is connected to the bit line, the other end of the memristor is connected to the drain of the transistor, the gate of the transistor is connected to the word line, and the source of the transistor is connected to the source line. The conductance value of each memristor in the memristor array is set to Figure 1A The value of each element G11~Gnn of the matrix G; Figure 1AThe value of each element V1, V2, …, Vn of the column vector V is mapped to a voltage value and applied to each bit line BL1, BL2, …, BLn of the memristor array; after each bit line WL1, WL2, …, WLn applies the turn-on voltage Vwl1, Vwl2, …, Vwln of each transistor in the corresponding column, according to Ohm's law and Kirchhoff's current law, the output current value of each source line SL1, SL2, …, SLn is the value of the corresponding element I1, I2, …, In in the column vector I. For example, the output current value of the source line SL1 is equal to the voltage value V1, V2, …, Vn applied to the n bit lines BL1, BL2, …, BLn multiplied by the conductance value G11, G12, …, G1 of each corresponding memristor, and then the product is accumulated to obtain the output current value of the source line SL1, which is the value of the element I1 in the column vector I. Therefore, by measuring the output current values of all columns, it can be obtained Figure 1A The result of the matrix-vector multiplication is shown.
[0037] A storage-computing integrated computing device based on a non-volatile storage array such as a memristor array has the characteristics of mutual integration of storage and computing. Compared with a traditional processor computing device, the storage-computing integrated computing device has the advantages of high computing efficiency and low power consumption. Therefore, the storage-computing integrated computing device can provide hardware support for deploying neural network algorithms in a wider range of scenarios.
[0038] Figure 2 It is a schematic diagram of a data processing device that deploys a neural network algorithm to perform inference calculations. Figure 2As shown, the data processing device (or processing unit (PE)) includes an input module, an output module, a computing unit, an array read-write unit, a state control and conversion unit, a special function unit and a processing unit interface module, and these units and modules can be implemented by circuits, such as digital circuits. Among them, the input module includes an input buffer unit, a digital-to-analog converter and a multiplexer; the output module includes a multiplexer, a sampling and holding unit, an analog-to-digital converter, a shift accumulation unit and an output buffer unit; the computing unit can include multiple computing arrays, each of which is based on a memristor array. Under the control of the state control and conversion unit, the input module buffers and converts the received input data, and then inputs it to the computing unit through the bit line end according to the selection channel of the gate for linear calculation processing. The result of the calculation unit processing is output through the source line end and superimposed with the calculation result of the nonlinear operation required by the neural network algorithm, and after being output by the multiplexer, it is sampled and held and converted to analog, and finally shifted, accumulated and buffered to output the result of the inference calculation. Nonlinear operations (such as linear rectification operations) and nonlinear activation function operations are provided by functional function units (such as special function units). The processing unit interface module is used to communicate with external devices outside the data processing device, such as external storage devices, main control units and other data processing devices, for example, to transfer data, instructions, etc., so as to achieve collaborative work between devices.
[0039] Figure 3 is corresponding to Figure 2 Flow chart of a data processing method for performing inference calculation on a data processing device. Figure 3As shown, in the inference calculation process, the data processing device first deploys the inference model. Among them, the deployment process includes model input, compilation optimization, weight deployment and inference mode configuration. After the neural network model algorithm is determined, the various operation units in the neural network model algorithm can be optimized by using technologies such as model compilation to obtain an optimized weight deployment scheme in the data processing device. For example, after the structural data of the neural network model is input, the structural data such as weight data is compiled into a voltage signal that can be written into the memristor array, and the voltage signal is written into the memristor array to change the conductance value of each memristor in the memristor array, thereby completing the weight deployment. The data processing device further configures the input and output modules according to the input model structure data, as well as the special function module for realizing nonlinear operations and the processing unit interface module for communicating with the outside. After the data processing device completes the deployment and configuration of the inference model, it will enter the forward reasoning working mode, for example, it will start to receive external task data and input the task data. According to the existing configuration information, the computing unit of the data processing device starts to execute computing tasks to perform on-chip task calculations. After all computing tasks are completed, the data processing device outputs the results to the outside, thereby completing the forward reasoning process.
[0040] The data processing device may not need to perform data transmission with the main control unit during the above process. When multiple data processing devices work in parallel, they can perform data transmission through their respective processing unit interface modules to perform data synchronization.
[0041] However, the above data processing device is oriented to the inference application of neural network algorithms and cannot provide hardware support for model training of neural network algorithms. However, in order to obtain higher efficiency, the current model training scheme on the processor chip based on memristor array often adopts a deeply customized design, which makes the hardware lack certain flexibility and cannot meet the requirements of inference and training of various neural network algorithms.
[0042] The training method of the neural network algorithm mainly uses the back propagation algorithm (BP). The back propagation algorithm is similar to updating the weight matrix of each layer of the neural network algorithm layer by layer in the opposite direction of the forward propagation algorithm of the inference calculation, and the updated value of the weight matrix is calculated by the error value of each layer. The error value of each layer is obtained by multiplying the transpose of the weight matrix of the next layer adjacent to the layer by the error value of the next layer. Therefore, under the condition of obtaining the error value of the last layer of a neural network algorithm and the weight matrix of the last layer, the updated value of the weight matrix of the last layer can be calculated. At the same time, the error value of the second-to-last layer can be calculated according to the back propagation algorithm, thereby calculating the updated value of the weight matrix of the second-to-last layer, and so on, until all layers of the neural network algorithm are reversely updated. Therefore, at least one embodiment of the present disclosure provides a data processing device that integrates storage and computing and can simultaneously support neural network reasoning and training, such as Figure 4 As shown, the data processing device includes a bidirectional processing module 100 , a control module 200 , a parameter management module 300 , and an input / output module 400 .
[0043] The bidirectional data processing module 100 includes one or more storage and computing integrated computing arrays 110, so the bidirectional data processing module 100 may include a multi-channel input end and a multi-channel output end. The bidirectional data processing module 100 is used to perform computing tasks, and the computing tasks include reasoning computing tasks and training computing tasks. The control module 200 is used to switch the working mode of the bidirectional data processing module to the reasoning working mode to perform the reasoning computing task, and to switch the working mode of the bidirectional data processing module to the training working mode to perform the training computing task. For example, the control module 200 can be implemented as hardware or firmware such as CPU, SoC, FPGA, ASIC, or any combination of hardware or firmware and software. The parameter management module 300 is used to set the weight parameters of the bidirectional data processing module. Under the control of the control module 200, the input and output module 400 generates a computing input signal according to the input data of the computing task, and provides the computing input signal to the bidirectional data processing module, receives the computing output signal from the bidirectional data processing module, and generates output data according to the computing output signal.
[0044] For example, the computing array 110 of the bidirectional processing module 100 may include a memristor array. The memristor array is used to realize storage computing integration. The memristor array may include a plurality of memristors arranged in an array, and each memristor array may adopt Figure 1B The structure shown may also adopt other structures capable of performing matrix multiplication calculations, for example, the memristor units constituting the memristor array do not include a switch circuit, or the memristor units include 2T2R (ie, two switch elements and two memristor units).
[0045] For example, the parameter management module 300 includes a weight array write unit and a weight array read unit. The weight array write unit can change the conductance value of each memristor in the plurality of memristors by using the weight parameter so as to write the weight parameter into the memristor array. Correspondingly, the weight array read unit can read the current conductance value of each memristor in the plurality of memristors from the memristor array so as to complete the reading of the current actual weight parameter, for example, the actual weight parameter read is compared with the preset weight parameter to determine whether the weight parameter needs to be reset.
[0046] For example, in one example, in order to be able to process tasks in both the inference computing tasks and the training computing tasks of the neural network algorithm, the data processing device may be provided with two sets of input modules and two sets of output modules, wherein one set of input modules and one set of output modules are used to process the data input and output of the inference computing tasks of the neural network algorithm, and the other set of input modules and another set of output modules are used to process the data input and output of the training computing tasks of the neural network algorithm. In this case, the input and output modules include an inference computing input module, an inference computing output module, a training computing input module, and a training computing output module. For example, the inference computing input module is equivalent to the first input submodule of the present disclosure, the inference computing output module is equivalent to the first output submodule of the present disclosure, the training computing input module is equivalent to the second input submodule of the present disclosure, and the training computing output module is equivalent to the second output submodule of the present disclosure.
[0047] For example, the inference calculation input module can be connected to the inference calculation input terminal of the bidirectional data processing module 100, and provide an inference input signal for the inference calculation task, and the inference input signal can be an analog signal obtained by processing the inference input data through the inference calculation input module, for example, in the form of a voltage signal applied to the bit line terminal of the memristor array. The inference calculation output module can be connected to the inference calculation output terminal of the bidirectional data processing module 100, receive the calculation result of the inference calculation task, and the calculation structure is output from the source line terminal of the memristor array in the form of a current signal, and the inference calculation output module converts the calculation result into inference output data for output.
[0048] The training calculation input module can be connected to the training calculation input terminal of the bidirectional data processing module 100, and provide a training calculation input signal based on the training calculation task. The training calculation input signal can be an analog signal obtained by processing the training calculation input data through the training calculation input module, for example, in the form of a voltage signal applied to the source line end of the memristor array. The training calculation output module can be connected to the training calculation output terminal of the bidirectional data processing module 100, receive the calculation result of the training calculation task, and the calculation structure is output from the bit line end of the memristor array in the form of a current signal. The data processing module 100 converts the calculation result into training calculation output data for output.
[0049] For example, the inference calculation input end of the bidirectional data processing module 100 corresponds to the first connection end side of the bidirectional data processing module of the present disclosure; the training calculation input end of the bidirectional data processing module 100 corresponds to the second connection end side of the bidirectional data processing module of the present disclosure; the inference input data corresponds to the first input data of the present disclosure; the inference output data corresponds to the first output data of the present disclosure; the training input data corresponds to the second input data of the present disclosure; and the training output data corresponds to the second output data of the present disclosure.
[0050] For example, in another example, the reasoning calculation input module and the training calculation input module are functionally identical and can use the same input module. Any of the input modules in the reasoning calculation input module and the training calculation input module can include an input data buffer unit (buffer), a digital-to-analog signal converter (DAC), and an input multiplexer (MUX). For example, in one example, the input data buffer unit corresponds to the first data buffer unit of the present disclosure, and in another example, it corresponds to the third data buffer unit of the present disclosure; in one example, the digital-to-analog signal converter corresponds to the first digital-to-analog signal converter of the present disclosure, and in another example, it corresponds to the third digital-to-analog signal converter of the present disclosure; in one example, the input multiplexer corresponds to the first multiplexer of the present disclosure, and in another example, it corresponds to the third multiplexer of the present disclosure. Among them, the input data buffer unit can be implemented by various caches (caches), memories (memory), etc. The input data buffer unit is used to receive input data, for example, the input data can be reasoning calculation input data or training calculation input data. Afterwards, the input data buffer unit provides the input data to the input digital-to-analog signal converter, which converts the input data from digital signals to analog signals, and provides the converted analog input signals to the input multiplexer. The input multiplexer can provide the analog input signals to the inference calculation input terminal (e.g., bit line terminal) or the training calculation input terminal (e.g., source line terminal) of the bidirectional data processing module 100 through a switching switch (not shown) through a channel selected by the input multiplexer. The inference calculation input terminal or the training calculation input terminal of the bidirectional data processing module 100 corresponds to a plurality of calculation units 110, and therefore each has a plurality of channels.
[0051] In this other example, similarly, for example, the inference calculation output module and the training calculation output module are also functionally the same, and the same output module can be used. Any output module in the inference calculation output module and the training calculation output module can include an output multiplexer (MUX), a sampling and holding unit, an analog-to-digital signal converter (ADC), a shift accumulation unit, and an output data buffer unit, etc. For example, in one example, the output multiplexer corresponds to the second multiplexer of the present disclosure, and in another example, it corresponds to the fourth multiplexer of the present disclosure; in one example, the sampling and holding unit corresponds to the first sampling and holding unit of the present disclosure, and in another example, it corresponds to the second sampling and holding unit of the present disclosure; in one example, the analog-to-digital signal converter corresponds to the second analog-to-digital signal converter of the present disclosure, and in another example, it corresponds to the fourth analog-to-digital signal converter of the present disclosure; in one example, the shift accumulation unit corresponds to the first shift accumulation unit of the present disclosure, and in another example, it corresponds to the second shift accumulation unit of the present disclosure; in one example, the output data buffer unit corresponds to the second data buffer unit of the present disclosure, and in another example, it corresponds to the fourth data buffer unit of the present disclosure. Among them, through another switching switch (not shown), the output multiplexer can receive multiple output signals, such as the inference calculation output signal or the training calculation output signal, from the inference calculation output terminal or the training calculation output terminal of the bidirectional data processing module 100 through the selected channel. Afterwards, the output multiplexer can provide the output signal to the sampling and holding unit. The sampling and holding unit can be implemented by various samplers and voltage holders, and is used to sample the output signal and provide the sampled output signal to the analog-to-digital signal converter. The analog-to-digital signal converter is used to convert the sampled analog output signal from an analog signal to a digital signal, and provide the digital output data of the converted output to the shift accumulation unit. The shift accumulation unit can be implemented by a shift register, which is used to superimpose the output data and provide it to the output data buffer unit. The output data buffer unit can use the implementation method of the input data buffer unit to match the data rate of the output data with the external data rate. In this example, the above two switching switches are controlled by the control unit, so that the entire data processing device can be switched between the inference working mode and the training working mode. In addition, in this example, the number of input signals and the number of output signals of the calculation array are the same.
[0052] For example, in the case where the data processing device is provided with two sets of input modules and two sets of output modules, the control module 200 can be configured to perform the following operations. In the inference working mode, the control module 200 connects the inference calculation input module with the inference calculation input terminal of the bidirectional data processing module 100 to provide an inference calculation input signal for the inference calculation task, and the inference calculation input signal can be obtained by converting the inference calculation input data through the input and output module 400. The inference calculation output module is connected to the inference calculation output terminal of the bidirectional data processing module 100 to receive the calculation result of the inference calculation task and generate the inference calculation output data. In the training working mode, the control module 200 connects the training calculation input module with the training calculation input terminal of the bidirectional data processing module 100 to provide a training calculation input signal based on the training calculation task, and the training calculation input signal can be obtained by converting the training calculation input data through the input and output module 400. The training calculation output module is connected to the training calculation output terminal of the bidirectional data processing module 100 to receive the calculation result of the training calculation task and generate the training calculation output data.
[0053] For example, in another example, the data processing device can also integrate the input module and the output module of the bit line end of the bidirectional data processing module 100 into a multiplexed input-output submodule, and integrate the input module and the output module of the source line end of the bidirectional data processing module 100 into another multiplexed input-output submodule. Therefore, the two input-output submodules are the same, one of which can be connected to the bit line end of the bidirectional data processing module 100 to provide an inference calculation input signal based on the inference calculation task, and the inference calculation input signal can be converted from the inference calculation input data through the input-output module 400; at the same time, the input-output submodule receives the calculation result of the training calculation task and generates the training calculation output data. The other input-output submodule can be connected to the source line end of the bidirectional data processing module 100 to provide a training calculation input signal based on the training calculation task, and the training calculation input signal can be converted from the training calculation input data through the input-output module 400; at the same time, the input-output submodule receives the calculation result of the inference calculation task and generates the inference calculation output data.
[0054] For example, each of the input-output submodules may include a data buffer unit, a shift-accumulation unit, a digital-to-analog signal converter, an analog-to-digital signal converter, a sampling and holding unit, and a multiplexer. For example, in one example, the data buffer unit corresponds to the first data buffer unit of the present disclosure, and in another example, corresponds to the second data buffer unit of the present disclosure; in one example, the shift-accumulation unit corresponds to the first shift-accumulation unit of the present disclosure, and in another example, corresponds to the second shift-accumulation unit of the present disclosure; in one example, the digital-to-analog signal converter corresponds to the first digital-to-analog signal converter of the present disclosure, and in another example, corresponds to the second digital-to-analog signal converter of the present disclosure; in one example, the analog-to-digital signal converter corresponds to the first analog-to-digital signal converter of the present disclosure, and in another example, corresponds to the second analog-to-digital signal converter of the present disclosure; in one example, the sampling and holding unit corresponds to the first sampling and holding unit of the present disclosure, and in another example, corresponds to the second sampling and holding unit of the present disclosure; in one example, the multiplexer corresponds to the first multiplexer of the present disclosure, and in another example, corresponds to the second multiplexer of the present disclosure. Among them, except for the multiplexed data buffer unit and the multiplexer, the remaining shift accumulation unit, the digital-to-analog signal converter, the analog-to-digital signal converter and the sampling and holding unit are implemented in the same way as the above two sets of input modules and two sets of output modules. Among them, the data buffer unit can be multiplexed, and the data buffer unit can be used to receive the inference calculation input data in addition to outputting the training calculation output data, and provide the inference calculation input data to the digital-to-analog signal converter. The digital-to-analog signal converter is used to perform digital-to-analog conversion on the inference calculation input data and provide the converted inference calculation input signal to the multiplexer. The multiplexer can be bidirectionally multiplexed, and the multiplexer provides the inference calculation input signal to the bit line end of the bidirectional data processing module 100 through the selected channel. At the same time, the multiplexer can also be used to receive the training calculation output signal from the bit line end of the bidirectional data processing module 100, and the multiplexer provides the training calculation output signal to the sampling and holding unit through the selected channel. The sampling and holding unit is used to sample the training calculation output signal and provide the sampled training calculation output signal to the analog-to-digital signal converter. The analog-to-digital signal converter is used to perform analog-to-digital conversion on the sampled training calculation output signal and provide the converted training calculation output data to the shift-accumulation unit. The shift-accumulation unit is used to provide the training calculation output data to the data buffer unit. The data buffer unit can also be used to output the training calculation output data.
[0055] For example, in the case where the data processing device uses a multiplexed input-output submodule, the data processing device may include only two multiplexed input-output submodules. The control module 200 may be configured to perform different operations in the inference working mode and the training working mode. In the inference working mode, the control module 200 may connect an input-output submodule to the bit line terminal of the bidirectional data processing module 100 to provide an inference calculation input signal based on the inference calculation task, and the inference calculation input signal may be converted from the inference calculation input data. At the same time, another input-output submodule may be connected to the source line terminal of the bidirectional data processing module 100 to receive the calculation result of the inference calculation task and generate the inference calculation output data. Correspondingly, in the training working mode, the control module 200 may connect an input-output submodule to the source line terminal of the bidirectional data processing module 100 to provide a training calculation input signal based on the training calculation task, and the training calculation input signal may be converted from the training calculation input data. At the same time, another input-output submodule may be connected to the bit line terminal of the bidirectional data processing module 100 to receive the calculation result of the training calculation task and generate the training calculation output data.
[0056] For example, in the case where the data processing device uses multiplexed input and output submodules, the data processing device may further include a multiplexing unit selection module 500. Under the control of the control module 200, the multiplexing unit selection module 500 may be used to select a data buffer unit, a digital-to-analog signal converter, and a multiplexer of one of the two input and output submodules as an input channel in the inference working mode; and correspondingly select a multiplexer, a sampling and holding unit, an analog-to-digital signal converter, a shift-accumulation unit, and a data buffer unit of the other input and output submodule as an output channel.
[0057] After the configuration of the input channel and the output channel of the inference working mode is completed, in the training working mode, it is only necessary to set the configuration of the input channel and the output channel of the inference working mode in the opposite way. For example, in the training working mode, the multiplexing unit selection module 500 uses the multiplexer, the sampling and holding unit, the analog-to-digital signal converter, the shift accumulation unit and the data buffer unit included in the input and output submodule as the input channel in the inference working mode as the output channel; and correspondingly uses the data buffer unit, the digital-to-analog signal converter and the multiplexer included in the input and output submodule as the output channel in the inference working mode as the input channel.
[0058] For example, the data processing device may further include a processing unit interface module, which is used to communicate with an external device outside the data processing device. For example, the data processing device may transmit data with an external main control module, memory, etc. through the processing unit interface module via an interconnection device to expand the function of the data processing device. The interconnection device may be a bus, an on-chip network, etc.
[0059] For example, the data processing device may further include a function unit, which is used to provide nonlinear operation for the data processed by the bidirectional data processing module 100 and output by the output module. For example, the function unit may perform nonlinear operations such as linear rectification operation (ReLU) and S-curve activation function (SIGMOD) operation in the neural network algorithm.
[0060] At least one embodiment of the present disclosure provides a data processing method, which is used in a data processing device of an embodiment of the present disclosure.
[0061] like Figure 5 As shown, the data processing method can be used to Figure 4 The data processing device shown, the data processing method comprises:
[0062] Step S101, the control module obtains the current working mode and controls the bidirectional data processing module;
[0063] Step S102, when the working mode is the inference working mode, the bidirectional data processing module uses the inference weight parameter for performing the inference computing task to perform the inference computing task;
[0064] Step S103, when the working mode is the training working mode, the bidirectional data processing module uses the training weight parameters for executing the training computing task to execute the training computing task.
[0065] The following will be combined Figure 4 The above three steps are described in detail without limitation.
[0066] In step S101, the control module of the data processing device obtains the current working mode.
[0067] For example, the control module 200 of the data processing device can determine the current working mode according to the user's settings or the input data type. The current working mode includes the reasoning working mode and the training working mode, such as the reasoning working mode of the neural network algorithm and the training working mode of the neural network algorithm. For example, when the input data type is the reasoning calculation input data, the control module 200 can determine the current working mode as the reasoning working mode; when the input data type is the training calculation input data, the control module 200 can determine the current working mode as the training working mode. According to the acquired working mode, the control module can control the bidirectional data processing module to execute the corresponding working mode.
[0068] For step S102, when the working mode is the inference working mode, the bidirectional data processing module uses the inference weight parameters for executing the inference computing task to execute the inference computing task.
[0069] For example, in the inference working mode, the data processing device may set weight parameters for inference before executing the inference computing task, for example, the weight parameters of each layer of the neural network algorithm may be deployed on multiple computing arrays 110 of the bidirectional data processing module 100, each computing array corresponding to a layer of the neural network algorithm. After the data processing device sets the weight parameters for the inference computing task, it may prepare to accept the inference computing input data, and use these weight parameters and the input data to execute the inference computing task.
[0070] For step S103, when the working mode is the training working mode, the bidirectional data processing module uses the training weight parameters for executing the training computing task to execute the training computing task.
[0071] For example, similar to the inference working mode, before the data processing device performs the training computing task, if necessary, it can set the weight parameters for training, or use the weight parameters previously used for other operations (such as inference operations). After the data processing device sets the weight parameters for the training computing task, it can prepare to receive the training computing input data, and use these weight parameters and input data to perform the training computing task.
[0072] For example, when the data processing device performs an inference calculation task, it can first receive the inference calculation input data through the input-output module 400. The bidirectional data processing module 100 of the data processing device is implemented based on a memristor array. The memristor array is used to receive and process analog signals, and the output is also an analog signal. In most cases, the received inference calculation input data is a digital signal. Therefore, the received inference calculation input data cannot be directly transmitted to the bidirectional data processing module 100 for processing, and the digital inference calculation input data must first be converted into an analog inference calculation input signal. For example, a digital-to-analog signal converter can be used to convert the inference calculation input data into an inference calculation input signal.
[0073] Afterwards, the data processing device can use the bidirectional data processing module 100 to perform storage and computing integration operations on the converted reasoning calculation input signal, such as performing a matrix multiplication operation based on a memristor array. After the execution is completed, the bidirectional data processing module 100 outputs the calculated reasoning calculation output signal to the input and output module 400 of the data processing device for subsequent processing. The reasoning calculation output signal can be a classification result after the neural network algorithm reasoning calculation.
[0074] Finally, in order to facilitate subsequent data processing, the data processing device needs to convert the analog signal output by the bidirectional data processing module 100 into a digital signal. For example, the data processing device can convert the analog inference calculation output signal into digital inference calculation output data through the input and output module 400, and output the digital inference calculation output data. For example, the inference calculation input signal corresponds to the first calculation input signal of the present disclosure; the inference calculation output signal corresponds to the first calculation output signal of the present disclosure.
[0075] For example, when the data processing device performs a training computing task, it is similar to performing an inference computing task. The process in which the data processing device receives the training computing input data and generates the training computing input signal from the training computing input data is the same as that of the inference computing task, and will not be repeated here.
[0076] Afterwards, when the bidirectional data processing module 100 of the data processing device performs storage and calculation integration operations on the training calculation input signal, for example, when performing a matrix multiplication operation based on a memristor array, it is necessary to output the calculation results of each layer of the neural network algorithm, and output the calculation results of each layer as a training calculation output signal through the input and output module 400 to the main control unit outside the data processing device, so that the main control unit can perform residual calculation. The external main control unit further calculates the weight update value of each layer of the neural network algorithm based on the calculated residual, and transmits the weight update value back to the data processing device, and the parameter management module 300 of the data processing device updates the weight value of the calculation array 110 of the bidirectional data processing module 100 based on the weight update value. The weight value of the calculation array 110 can correspond to the conductance value of the memristor array. The process of generating training calculation output data according to the training calculation output signal is the same as the inference calculation task, and will not be repeated here. For example, the training calculation input signal corresponds to the second calculation input signal of the present disclosure; the training calculation output signal corresponds to the second calculation output signal of the present disclosure.
[0077] The data processing device of at least one embodiment of the present disclosure can not only schedule data to obtain higher reasoning efficiency under the drive of data flow, but also flexibly configure the data flow path under the scheduling of the control unit to meet the needs of various complex network model algorithm training. At the same time, the data processing device has high energy efficiency and high computing power in reasoning and training capabilities. For example, the data processing device of at least one embodiment of the present disclosure can complete local training, realize incremental training or federated learning, and meet user customized application requirements under the premise of protecting user privacy. The data processing device of at least one embodiment of the present disclosure can increase the stability and reliability of the storage and computing integrated device based on the memristor array through on-chip training or layer-by-layer calibration, so that the storage and computing integrated device can adaptively restore the system accuracy and alleviate the impact of device non-ideal characteristics, other noises, and parasitic parameters on system accuracy.
[0078] A data processing device, a method for the data processing device, and a data processing system including the data processing device proposed in at least one embodiment of the present disclosure will be described below with reference to a specific but non-limiting example.
[0079] For example, Figure 6 is a schematic diagram of another data processing device provided by at least one embodiment of the present disclosure, Figure 6 The data processing device shown is Figure 4 An implementation of a data processing device is shown.
[0080] like Figure 6As shown, the data processing device includes a bidirectional data processing module 100, a control module 200, a parameter management module 300, two input and output modules 400, a multiplexing unit selection module 500, a processing unit interface module 600 and a functional module 700.
[0081] The bidirectional data processing module 100 has a bit line terminal 1001 and a source line terminal 1002, the bit line terminal 1001 can be used to receive and output data; the source line terminal 1002 can also be used to receive and output data, the bidirectional data processing module 100 includes one or more computing arrays, each computing array can be a memristor array, the parameter management module 300 includes a weight array read unit and a weight array write unit, and each input and output module 400 includes a data buffer unit, a shift accumulation unit, an analog-to-digital converter, a digital-to-analog converter, a sampling and holding unit, and a multiplexer. The bidirectional data processing module 100 can complete the matrix multiplication operation of the input data through the memristor array and output the calculation result of the matrix multiplication operation. The control module 200 is used to control the data processing device to perform the calculation task. The parameter management module 300 converts the weight value into a write voltage signal of the memristor array of the bidirectional data processing module 100 through the weight array write unit, thereby changing the conductance value of each memristor unit in the memristor array to complete the writing of the weight value; or reads out the conductance value of each memristor in the memristor array of the bidirectional data processing module 100 as a weight value through the weight array read unit.
[0082] The data processing device is compatible with a forward data path and a reverse data path. The forward data path can be a path for executing the inference calculation task of the neural network algorithm, and the reverse data path can be a path for executing the training calculation task of the neural network algorithm. The input part of the forward data path and the output part of the reverse data path can share the same input-output module 400, and the output part of the forward data path and the input part of the reverse data path can also share the same input-output module 400. In the same input-output module 400, the data buffer unit and the multiplexer can be shared (multiplexed) by the forward data path and the reverse data path. The multiplexing unit selection module 500 is used to configure the data buffer unit and the multiplexer shared by the forward data path and the reverse data path. For example, when the data processing module performs the task of the forward data path, the multiplexing unit selection module 500 configures the data buffer unit and the multiplexer in one of the input-output modules 400 as an input mode, and the input-output module 400 can be used for the input of the forward data path, and configures the data buffer unit and the multiplexer in another input-output module 400 as an output mode, and the input-output module 400 can be used for the input of the reverse data path. Conversely, when the data processing module performs the task of the reverse data path, the multiplexing unit selection module 500 can perform the opposite configuration of the above process. When the data processing device performs the task of the reverse data path, such as performing the training calculation task of the neural network algorithm, the processing unit interface module 600 is used to transmit the error value of the calculation result of each layer in the neural network model to the main control unit outside the data processing device for weight value update calculation, and transmits the calculated weight update value back to the data processing device. The functional function unit 700 is used to provide nonlinear operation calculation functions in the neural network model, such as linear rectification operation, nonlinear activation function operation and other nonlinear operations.
[0083] Figure 7 is a flowchart of another data processing method provided by at least one embodiment of the present disclosure, the data processing method is used to Figure 6 The data processing device shown.
[0084] For example, the process of the data processing device executing the task of the forward data path is the same as the process of the aforementioned inference calculation method, which will not be repeated here. Figure 7 shown. Figure 7In the process, according to the back propagation algorithm (BP), the data processing device first inputs the training set data in batches (Batch), and the training set data includes data items and label values (Lable). According to the inference calculation method, all batches of training set data are inferred and calculated on the data processing device to obtain and record the output results of each batch of training data sets and the intermediate results of the inference calculation process. The inference calculation includes seven steps: model input, compilation optimization, weight deployment, training mode configuration, task data input, on-chip task calculation and forward reasoning. Under the reverse data path, the training mode configuration can be to configure the data processing device according to the training calculation method. For example, the data buffer unit and the multiplexer of the input and output module can be configured to the data direction corresponding to the reverse data path through the multiplexing unit selection module. The task data input can be input from the source line end of the bidirectional data processing module. The model input, compilation optimization, weight deployment, on-chip task calculation and forward reasoning steps are the same as those mentioned above. Figure 3 The corresponding steps shown in are the same and will not be repeated here.
[0085] During the inference calculation task, the result of the inference calculation can be output from the bit line end of the bidirectional data processing module. After the inference calculation task is completed, the data processing device transmits the output result, intermediate result and label value of the inference calculation to the main control unit outside the data processing device through the processing unit interface module. The main control unit obtains the error of the last output layer according to the difference between the label value and the output result, that is, completes the error and calculation, and then calculates the weight update gradient of the last output layer, thereby calculating the weight update value, and transmits the weight update value to the data processing device through the processing unit interface module. The last output layer belongs to the neural network model used for the inference calculation. The parameter management module of the data processing device calculates the conductance value update amount according to the weight update value, converts the conductance value update amount into a voltage value that can be written into the memristor array, and writes the voltage value into the memristor array corresponding to the last output layer through the weight array write unit, thereby updating the weight of the last output layer. Similarly, the remaining layers also follow a similar approach, and the weight gradient of the layer is obtained by the weight value of the previous layer and the error of the previous layer, thereby obtaining the weight update value of the current layer, until all layers are updated. Finally, when all the training set data has been trained and the weights have been updated, the validation set can be used for evaluation to determine whether to terminate the training. If the termination conditions are met, the data processing device outputs the training results; otherwise, the data processing device continues to input training data for a new round of training.
[0086] Figure 8 is a flowchart of another data processing method provided by at least one embodiment of the present disclosure, the data processing method may be a layer-by-layer training method for executing a reverse data path of a neural network algorithm, which may be used Figure 6The data processing device shown.
[0087] For example, the data processing device may use a layer-by-layer neural network model training method. Figure 8 As shown, the data processing device can also meet the needs of neural network reasoning acceleration applications, and update the weight values of each layer of the neural network model in a layer-by-layer training manner, thereby adjusting the conductance value of the memristor array corresponding to each layer of the neural network model. The layer-by-layer training method process is as follows: First, the initialized weights are deployed on the hardware of the data processing device, and forward reasoning calculations are performed. Among them, the reasoning calculation includes six steps: model input, compilation optimization, weight deployment, training mode configuration, task data input, and on-chip task calculation. Figure 7 The corresponding steps shown in are the same and will not be repeated here. The processing interface module of the data processing device outputs the inference results of the convolution layer and the fully connected layer of the neural network algorithm, as well as the inference results of the network algorithm software model with trained weights, to the main control module outside the data processing device. The main control module compares the inference results of the convolution layer and the fully connected layer of the neural network algorithm with the inference results of the network algorithm software model with trained weights, calculates the residual of each layer, and determines whether the residual value of each layer is within the preset threshold range. If the residual value is not within the threshold range, the main control module calculates the change of the weight value according to the residual value and the output result of the previous layer, and outputs the updated amount of the weight value to the data processing device. Thus, the parameter management module of the data processing device generates a memristor array conductance value write voltage signal according to the updated amount of the weight value, and writes it into the memristor array to update the conductance value; if the residual value is within the threshold range, the next layer is calibrated until all convolution layers and fully connected layers have been calibrated and the training results are output.
[0088] By training the data processing device layer by layer, it is possible to resist the influence of non-ideal factors on the accuracy of the finally trained neural network algorithm, so as to greatly improve the accuracy of the neural network algorithm, update the weight value of the neural network algorithm more finely, and calibrate the calculation results of the neural network algorithm more finely.
[0089] Fig. 9 It is a schematic diagram of the data scheduling process of multiple data processing devices. Fig. 9 As shown, the computing core module includes Figure 6The multiple data processing devices shown in the figure transmit information to each other through the processing unit interface module, and the multiple data processing devices transmit information to the main control unit through the processing unit interface module respectively. Under the forward data path task, such as the reasoning working mode of the neural network algorithm, the computing core module accepts external data input and distributes the data input to each data processing device. After receiving the data input, each data processing device performs the reasoning calculation task of the forward data path according to the existing configuration information until all the calculation tasks are completed, and the computing core module outputs the calculation results of each data processing device to the outside. In order to obtain higher execution efficiency, each data processing device may not need to transmit information with the main control unit. In addition, information can also be transmitted between each data processing device through the bus module. Under the reverse data path task, for example, in the training mode of the neural network algorithm, the data processing device needs to obtain the weight update values of the convolution layer and the fully connected layer of the neural network algorithm in addition to performing the above-mentioned reasoning calculation tasks, so as to update the conductance value of the memristor array, so that the data flow is more complicated than the reasoning working mode. Therefore, each data processing device needs to use the main control unit to perform data scheduling, so as to calculate the updated weight values of the convolutional layer and the fully connected layer of the neural network algorithm through the main control unit, and retrieve the updated weight values through the processing unit interface module.
[0090] Fig.10 is a schematic diagram of a data processing system provided by at least one embodiment of the present disclosure. The data processing system includes Figure 6 The data processing device shown can be used to perform reasoning computing tasks and training computing tasks of neural network algorithms.
[0091] like Fig.10As shown, the data processing system includes: a routing module, a computing core module, a main control unit, a bus module, an interface module, a clock module and a power module. The routing module is used for data input and data output between the data processing system and the outside. Data input includes inputting external data into the computing core module through the routing module or transmitting it to the main control unit through the bus module; data output includes outputting data processed by the data processing system to the outside of the data processing system through the routing module. The computing core module is used to implement matrix-vector multiplication, activation, pooling and other operations of the neural network algorithm, and receives data through the routing module or the bus module. The main control unit is used for data scheduling of training computing tasks. For example, the main control unit can transmit data with the computing core module and the routing module through the bus module. The main control unit can be implemented by, but not limited to, an embedded microprocessor, such as an MCU based on the RISC-V architecture or the ARM architecture. The main control module can configure different interface addresses through the bus module to realize control and data transmission of the remaining modules. The bus module is used to provide a data transmission protocol between modules and perform data transmission. For example, the bus module can be an AXI bus. Each module has a different bus interface address, and the data transmission of each module can be completed by configuring the data address information of each module. The interface module is used to expand the capabilities of the data processing system. The interface module can connect different peripherals through interfaces of various protocols. For example, the interface module can be, but is not limited to, a PCIE interface, an SPI interface, etc., to realize the function of data and instruction transmission between the data processing system and more external devices. The clock module is used to provide a working clock for the digital circuits in each module. The power module is used to manage the working power of each module.
[0092] Fig.11 for Fig.10 The data flow diagram of the data processing system shown in FIG. Fig.11 As shown, under the forward data path task, for example, in the inference mode, the data path can be: the routing module receives input data from the outside, and then passes it to the computing core module for inference calculation. When the model parameters are large, the model weights will be deployed in multiple data processing devices of the computing core module. At this time, data processing devices with data dependencies can be transmitted through the bus module. The multiple data processing devices of the computing core module perform inference calculation on the input data according to the configuration until all the input data are calculated. After the calculation is completed, the calculation results will be output to the outside of the system through the routing module.
[0093] Fig.12 for Fig.10 The data flow diagram of the data processing system shown in FIG. 1 is a diagram of a data flow for executing a training computing task. In a reverse data path task, for example, in a training mode, Fig.12As shown, the data path can be: the routing module receives input data from the outside, and then transmits it to the main control unit and the computing core module through the bus module. The residual value of each layer of the neural network algorithm is obtained through forward reasoning calculation, and the weight update value is obtained according to the residual value of each layer and the input calculation corresponding to the layer. The weight update calculation process in the forward reasoning calculation process can be processed and implemented by the main control unit, and the computing core module transmits data with the main control unit through the bus module during this process. After the weight update value of each layer of the neural network algorithm is obtained, the main control unit sends a control signal to configure the corresponding data processing module to update the weight. The entire training process requires the output layer residual of the neural network algorithm to be transmitted in reverse to obtain the residual of each layer, and the cycle is executed until the training update of all layers of the neural network algorithm is completed.
[0094] Fig.13 for Fig.10 The data processing system shown in FIG. 1 performs a data flow diagram of a layer-by-layer training computing task. In a reverse data path task, for example, in a layer-by-layer training mode, as shown in FIG. Fig.13 As shown, the data path can be: the routing module receives input data from the outside, and then transmits it to the main control unit through the bus module. The main control unit will then transmit the data to the computing core module through the bus module to perform training computing tasks. After the convolution layer and the fully connected layer of the neural network algorithm are completed, the calculation results will be transmitted to the main control unit through the bus module, and the main control unit will transmit them to the routing module again through the bus module, so that the calculation results are output to the outside of the data processing system through the routing module. Outside the data processing system, after the calculation results are compared with the calculation results calculated by the neural network algorithm software model to obtain the weight update value, the weight update value is transmitted to the inside of the data processing system through the routing module and transmitted to the main control unit through the bus module. After that, the weight update value is transmitted to the computing core module through the main control unit through the bus module, and the corresponding data processing module is configured to update the weight. This layer-by-layer training calculation process will be executed until the difference between the calculation result of the data processing system and the calculation result of the external neural network algorithm software is within the set threshold. Therefore, by training the neural network algorithm layer by layer, the data processing system can update the weight value of the data processing device more finely, so as to more effectively resist the influence of the non-ideal factors of the data processing system on the final recognition accuracy of the neural network algorithm.
[0095] Therefore, the data processing system can not only perform data scheduling under the drive of data flow to meet the high efficiency requirements of neural network algorithm reasoning operations, but also realize fine-grained scheduling of data flow under the control of the main control unit, support the reasoning and training computing tasks of various neural network algorithms, and adapt to the needs of various application scenarios.
[0096] There are a few points to note about this disclosure:
[0097] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.
[0098] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.
[0099] The above are only specific implementations of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure shall be based on the protection scope of the claims.
Claims
1. A data processing device, comprising: A bidirectional data processing module, comprising at least one computing array integrating storage and computing, configured to execute computing tasks, wherein the computing tasks include reasoning computing tasks and training computing tasks; a control module configured to switch the working mode of the bidirectional data processing module to an inference working mode to perform the inference computing task, and to switch the working mode of the bidirectional data processing module to a training working mode to perform the training computing task; A parameter management module, configured to set weight parameters of the bidirectional data processing module; an input-output module, configured to generate a calculation input signal according to the input data of the calculation task in response to the control of the control module, and provide the calculation input signal to the bidirectional data processing module, receive a calculation output signal from the bidirectional data processing module and generate output data according to the calculation output signal; The computing input signal includes an input signal of first input data for the inference computing task and an input signal of second input data for the training computing task. The computing output signal includes a computing output signal for the inference computing task and a computing output signal for the training computing task; The bidirectional data processing module includes a first connection end side and a second connection end side, The first connection end is connected to the input-output module and is configured to receive an input signal for the first input data and output a calculation output signal of the training calculation task. The second connection end is connected to the input-output module and is configured to receive an input signal for the second input data and output a calculation output signal of the inference calculation task.
2. The data processing apparatus according to claim 1, wherein: The computing array includes a memristor array for realizing the storage computing integration, and the memristor array includes a plurality of memristors arranged in an array.
3. The data processing apparatus according to claim 2, wherein: The parameter management module includes: a weight array writing unit configured to write the weight parameter into the memristor array by changing the conductance value of each memristor in the plurality of memristors using the weight parameter; and The weight array reading unit is configured to read the conductance value of each memristor in the plurality of memristors from the memristor array to complete the reading of the weight parameters.
4. The data processing apparatus according to claim 1, wherein: The input-output module comprises: A first input submodule, connected to the first connection end side of the bidirectional data processing module to provide an input signal of the first input data for the inference calculation task; A first output submodule, connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference calculation task and generate first output data; A second input submodule, connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task; The second output submodule is connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
5. The data processing apparatus according to claim 4, wherein: The first input submodule comprises: A first data buffer unit; a first digital-to-analog signal converter; The first multiplexer, The first data buffer unit is configured to receive the first input data and provide the first input data to the first digital-to-analog signal converter, the first digital-to-analog signal converter is configured to perform digital-to-analog conversion on the first input data and provide the converted first input signal to the first multiplexer, and the first multiplexer is configured to provide the first input signal to the first connection end side of the bidirectional data processing module through a selected channel. The first output submodule comprises: a second multiplexer; a first sample and hold unit; a second analog-to-digital signal converter; a first shift-and-accumulate unit; The second data buffer unit, The second multiplexer is configured to receive a first output signal from the second connection end side of the bidirectional data processing module, and provide the first output signal to the first sampling and holding unit through the selected channel, the first sampling and holding unit is configured to sample the first output signal and provide the sampled first output signal to the second analog-to-digital signal converter, the second analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled first output signal, and provide the converted first output data to the first shift-accumulation unit, the first shift-accumulation unit is configured to provide the first output data to the second data buffer unit, and the second data buffer unit is configured to output the first output data. The second input submodule comprises: A third data buffer unit; a third digital-to-analog signal converter; The third multiplexer, The third data buffer unit is configured to receive the second input data and provide the second input data to the third digital-to-analog signal converter, the third digital-to-analog signal converter is configured to perform digital-to-analog conversion on the second input data and provide the converted output second input signal to the third multiplexer, and the third multiplexer is configured to provide the second input signal to the second connection end side of the bidirectional data processing module through the selected channel. The second output submodule comprises: a fourth multiplexer; a second sample and hold unit; a fourth analog-to-digital signal converter; a second shift-accumulate unit; a fourth data buffer unit, Wherein, the fourth multiplexer is configured to receive a second output signal from the first connection end side of the bidirectional data processing module, and provide the second output signal to the second sampling and holding unit through the selected channel, the second sampling and holding unit is configured to sample the second output signal and provide the sampled second output signal to the fourth analog-to-digital signal converter, the fourth analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled second output signal, and provide the second output data output by the conversion to the second shift-accumulation unit, the second shift-accumulation unit is configured to provide the second output data to the fourth data buffer unit, and the fourth data buffer unit is configured to output the second output data; The first output signal is a calculation output signal used for the inference calculation task, and the second output signal is a calculation output signal used for the training calculation task.
6. The data processing device according to claim 4 or 5, wherein: The control module is configured as follows: In the inference working mode, the first input submodule is connected to the first connection end side of the bidirectional data processing module to provide an input signal for the first input data of the inference computing task, and the first output submodule is connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference computing task and generate first output data; as well as In the training working mode, the second input submodule is connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task, and the second output submodule is connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
7. The data processing apparatus according to claim 1, wherein: The input-output module comprises: a first input-output submodule connected to the first connection end side of the bidirectional data processing module to provide a first input signal based on the first input data of the inference computing task, and connected to the first connection end side of the bidirectional data processing module to receive the computing result of the training computing task and generate second output data; A second input-output submodule is connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training computing task, and is connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference computing task and generate first output data.
8. The data processing apparatus according to claim 7, wherein: The first input-output submodule comprises: A first data buffer unit; a first shift-and-accumulate unit; a first digital-to-analog signal converter; a first analog-to-digital signal converter; a first sample and hold unit; The first multiplexer, wherein the first data buffer unit is configured to receive the first input data and provide the first input data to the first digital-to-analog signal converter, the first digital-to-analog signal converter is configured to perform digital-to-analog conversion on the first input data and provide the converted first input signal to the first multiplexer, the first multiplexer is configured to provide the first input signal to the first connection end side of the bidirectional data processing module through a selected channel, and the first multiplexer is configured to receive a second output signal from the first connection end side of the bidirectional data processing module and provide the second output signal to the first sampling and holding unit through a selected channel, the first sampling and holding unit is configured to sample the second output signal and provide the sampled second output signal to the first analog-to-digital signal converter, the first analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled second output signal and provide the converted second output data to the first shift-accumulate unit, the first shift-accumulate unit is configured to provide the second output data to the first data buffer unit, and the first data buffer unit is configured to output the second output data, The second input-output submodule comprises: a second multiplexer; a second sample and hold unit; a second digital-to-analog signal converter; a second analog-to-digital signal converter; a second shift-accumulate unit; A second data buffer unit; wherein the second data buffer unit is configured to receive the second input data and provide the second input data to the second digital-to-analog signal converter, the second digital-to-analog signal converter is configured to perform digital-to-analog conversion on the second input data and provide the second input signal output by the conversion to the second multiplexer, the second multiplexer is configured to provide the second input signal to the second connection end side of the bidirectional data processing module through a selected channel, and the second multiplexer is configured to output the first signal from the second connection end side of the bidirectional data processing module and provide the first output signal to the second sampling and holding unit through a selected channel, the second sampling and holding unit is configured to sample the first output signal and provide the sampled first output signal to the second analog-to-digital signal converter, the second analog-to-digital signal converter is configured to perform analog-to-digital conversion on the sampled first output signal and provide the first output data output by the conversion to the second shift-accumulate unit, the second shift-accumulate unit is configured to provide the first output data to the second data buffer unit, and the second data buffer unit is configured to output the first output data; The first output signal is a calculation output signal used for the inference calculation task, and the second output signal is a calculation output signal used for the training calculation task.
9. The data processing apparatus according to claim 7 or 8, wherein: The control module is configured as follows: In response to the inference working mode, the first input-output submodule is connected to the first connection end side of the bidirectional data processing module to provide a first input signal based on the first input data of the inference computing task, and the second input-output submodule is connected to the second connection end side of the bidirectional data processing module to receive the calculation result of the inference computing task and generate first output data; as well as In response to the training working mode, the second input-output submodule is connected to the second connection end side of the bidirectional data processing module to provide an input signal based on the second input data of the training calculation task, and the first input-output submodule is connected to the first connection end side of the bidirectional data processing module to receive the calculation result of the training calculation task and generate second output data.
10. The data processing apparatus according to claim 8, further comprising: The multiplexing unit selection module is configured to, under the control of the control module, In response to the inference working mode, the first data buffer unit, the first digital-to-analog signal converter, and the first multiplexer are selected for input, and the second multiplexer, the second sampling and holding unit, the second analog-to-digital signal converter, the second shift-accumulation unit, and the second data buffer unit are selected for output; In response to the training working mode, the second data buffer unit, the second digital-to-analog signal converter, and the second multiplexer are selected for input, and the first multiplexer, the first sampling and holding unit, the first analog-to-digital signal converter, the first shift accumulation unit, and the first data buffer unit are selected for output.
11. The data processing apparatus according to claim 1, further comprising: The processing unit interface module is configured to communicate with external equipment outside the data processing device.
12. The data processing apparatus according to claim 1, further comprising: The functional unit is configured to provide nonlinear operation to the output data.
13. A data processing method, used in the data processing device according to any one of claims 1 to 12, comprising: The control module obtains the current working mode and controls the bidirectional data processing module; In response to the working mode being the inference working mode, the bidirectional data processing module uses the inference weight parameter for performing the inference computing task to perform the inference computing task; In response to the working mode being the training working mode, the bidirectional data processing module uses the training weight parameters for executing the training computing task to execute the training computing task.
14. The data processing method according to claim 13, wherein: The bidirectional data processing module performs the inference calculation task including: receiving the first input data and generating a first calculation input signal from the first input data; Performing a storage-computation integrated operation on the first computing input signal, and outputting a first computing output signal; generating first output data according to the first calculation output signal; and The two-way data processing module performs the training calculation task including: receiving the second input data and generating a second calculation input signal from the second input data; Performing a storage-computation integrated operation on the second calculation input signal, and outputting a second calculation output signal; generating second output data according to the second calculation output signal; The first input data is input data for the inference computing task, and the second input data is input data for the training computing task.
