Data processing method and system, electronic equipment and readable storage medium
By combining the analog computing module and the digital computing module, the analog operation is performed first and then converted into digital operation, the problem of difficulty in taking into account both calculation accuracy and power consumption in the prior art is solved, and efficient neural network data processing is achieved.
Patent Information
- Application Number
- CN202311586702.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-24
- Publication Date
- 2025-05-27
AI Technical Summary
The prior art is difficult to reduce the computing power consumption of processing neural network-related data while ensuring computing accuracy.
Using the combination of analog computing module and digital computing module, the analog computing module first simulates the weight matrix of the neural network to obtain the analog data, and then converts the analog data into digital data through an analog-to-digital converter. Finally, the digital computing module performs digital operations on the neural network topology matrix to obtain the target data.
Without affecting the calculation accuracy, the calculation power consumption is greatly reduced and the speed and efficiency of data processing are improved.
Smart Images
Figure CN120046667A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing, and more specifically, to a data processing method, system, electronic device, and readable storage medium. Background Art
[0002] Currently, as the scale of neural networks gradually increases, the requirements for computing power, energy efficiency, and computing accuracy at the hardware level for processing relevant data of neural networks are getting higher and higher. However, when processing relevant data of neural networks currently, it is difficult to balance the relationship between computing accuracy, computing power, and energy efficiency. Summary of the Invention
[0003] This application provides a data processing method, system, electronic device, and readable storage medium.
[0004] In a first aspect, an embodiment of this application provides a data processing method. The method is applied to a data processing system, and the data processing system includes: an analog computing module, a digital computing module, and an analog-to-digital converter. The analog-to-digital converter is respectively connected to the analog computing module and the digital computing module. The method includes: based on the analog computing module performing an operation on a first matrix according to input data to obtain analog data, where the first matrix includes a matrix corresponding to the weights of a neural network; based on the analog-to-digital converter converting the analog data into digital data; based on the digital computing module performing an operation on a second matrix according to the digital data to obtain target data, where the second matrix includes a matrix for describing the topological structure of the neural network.
[0005] In a second aspect, an embodiment of this application further provides a data processing system. The system includes: an analog computing module, an analog-to-digital converter, and a digital computing module; the analog-to-digital converter is respectively connected to the analog computing module and the digital computing module; the system further includes a processing module, and the processing module is respectively connected to the analog computing module, the analog-to-digital converter, and the digital computing module. The processing module is configured to execute the method described in the first aspect.
[0006] In a third aspect, an embodiment of this application further provides an electronic device, including: one or more processors; a memory; one or more application programs, where the one or more application programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.
[0007] In a fourth aspect, an embodiment of this application further provides a computer-readable storage medium. Program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the method described in the first aspect above.
[0008] The data processing method, system, electronic device and readable storage medium provided by the embodiments of the present application first perform an operation on a first matrix based on an analog computing module according to input data to obtain analog data, where the first matrix includes a matrix corresponding to the weights of a neural network; then convert the analog data into digital data based on an analog-to-digital converter; and then perform an operation on a second matrix based on the digital data by a digital computing module to obtain target data, where the second matrix includes a matrix for describing the topological structure of the neural network. The data processing method of the present application combines analog computing and digital computing, specifically combines the advantage of low power consumption of the analog computing module with the advantage of high computing accuracy of the digital computing module, and greatly reduces the computing power consumption with little impact on the computing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.
[0010] Figure 1 The flowchart of the data processing method provided by the embodiments of the present application is shown;
[0011] Figure 2 The flowchart of the data processing method provided by another embodiment of the present application is shown;
[0012] Figure 3 The schematic diagram of a graph data in the embodiments of the present application is shown;
[0013] Figure 4 The flowchart of the data processing method provided by another embodiment of the present application is shown;
[0014] Figure 5 is shown Figure 3 The flowchart of the data processing method of the graph data in the figure is shown;
[0015] Figure 6 The system block diagram of the data processing system provided by the embodiments of the present application is shown;
[0016] Figure 7 The system block diagram of the data processing system provided by another embodiment of the present application is shown;
[0017] Figure 8 The system block diagram of the data processing system provided by another embodiment of the present application is shown;
[0018] Figure 9Shows a block diagram of a systolic array provided by an embodiment of the present application;
[0019] Figure 10 Shows a block diagram of a processing unit provided by an embodiment of the present application;
[0020] Figure 11 Shows a block diagram of a processing unit provided by another embodiment of the present application;
[0021] Figure 12 Shows a block diagram of the structure of an electronic device provided by an embodiment of the present application;
[0022] Figure 13 Shows a block diagram of the structure of a computer-readable storage medium provided by an embodiment of the present application;
[0023] Figure 14 Shows a block diagram of the structure of a computer program product provided by an embodiment of the present application. Detailed implementation manners
[0024] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.
[0025] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.
[0026] Currently, as the scale of neural networks gradually increases, the requirements for computing power, energy efficiency, and computing accuracy at the hardware level for processing neural network-related data are getting higher and higher. It is difficult for the existing technologies to balance computing accuracy and energy efficiency. How to reduce the computing power consumption on the premise of ensuring computing accuracy is an urgent problem to be solved.
[0027] Please refer to Figure 1 , Figure 1The figure shows a flowchart of a data processing method provided by an embodiment of the present application. The data processing method specifically includes steps S110 to S130. The method can be applied to a data processing system, which includes: an analog computing module, a digital computing module, and an analog-to-digital converter (ADC). The analog-to-digital converter is respectively connected to the analog computing module and the digital computing module.
[0028] Among them, the data processing system can be used to perform computing processing on data. For example, the data can be data related to a neural network. In some embodiments, the data processing system may further include a processing module, which is respectively connected to the analog computing module, the analog-to-digital converter, and the digital computing module. Furthermore, the method steps in the embodiments of the present application can be executed through the processing module.
[0029] Step S110: Based on the analog computing module, perform an operation on a first matrix according to input data to obtain analog data. The first matrix includes a matrix corresponding to the weights of the neural network.
[0030] It should be noted that the analog computing module can be a hardware module for performing analog operations. The analog operations include at least one of addition operations and multiplication operations. Among them, the division operation in the analog operation can be converted into a multiplication operation, and the subtraction operation can be converted into an addition operation.
[0031] In an optional embodiment, the first matrix includes a matrix corresponding to the weights of the neural network. Based on the analog computing module, perform an operation on the first matrix according to input data to obtain analog data, that is, the analog operation of the input data and the weights of the neural network is realized. Specifically, it can be at least one of multiplying and adding the input data and the weights of the neural network.
[0032] Among them, the input data can be one of a numerical value and a matrix. In some embodiments, the input data is an input matrix of the neural network. Exemplarily, in the calculation of a graph convolutional neural network, the input data can be the data in each node of the graph data.
[0033] Among them, the analog data exists in the form of a continuous analog signal, such as voltage or current. The analog data can be one of a numerical value and a matrix.
[0034] Among them, the neural network can be a convolutional neural network, a recurrent neural network, or a graph neural network. The embodiments of the present application do not make specific limitations.
[0035] Exemplarily, the first matrix includes the weight matrix of a convolutional neural network. Based on the analog computing module operating on the first matrix according to input data, analog data is obtained, which realizes the analog operation between the input data and the weight matrix of the convolutional neural network. Specifically, it can be at least one of multiplying and adding the input data and the weight matrix of the convolutional neural network.
[0036] It can be understood that since the first matrix includes the weight matrix of a convolutional neural network, and the data volume of the weight matrix of a neural network is generally large, while the analog computing module has the characteristics of fast processing speed and high energy efficiency. Therefore, by calculating the first matrix and the input data through the analog computing module, high data processing speed and energy efficiency can be obtained.
[0037] In some embodiments, it can be by storing the elements in the first matrix into the analog computing module, and then operating according to the input data and the elements in the first matrix to obtain analog data. For a detailed introduction, refer to the subsequent embodiments.
[0038] Step S120: Based on the analog-to-digital converter, convert the analog data into digital data.
[0039] For subsequent operations on the second matrix, where the second matrix includes a matrix for describing the neural network topology structure, it can be understood that the second matrix has a high requirement for calculation accuracy. Although the analog computing module has high data processing speed and energy efficiency, its data processing accuracy is lower than that of the digital computing module. Therefore, if directly calculating the second matrix through the analog computing module, the resulting accuracy will be low, and thus it is impossible to obtain high data processing speed and energy efficiency while ensuring the calculation accuracy. Therefore, before operating on the second matrix, the analog data obtained in the previous step can be first converted into digital data, so that subsequently, based on the digital computing module, the second matrix can be operated according to the digital data. For a detailed introduction, refer to the subsequent steps.
[0040] It should be noted that analog data can be represented by analog signals, and different analog signals represent different analog data. Digital data can be represented by digital signals, and different digital signals represent different digital data. The analog-to-digital converter is used to convert analog signals into digital signals, that is, the analog data can be converted into digital data through the analog-to-digital converter.
[0041] Step S130: Based on the digital computing module, operate on the second matrix according to the digital data to obtain target data, where the second matrix includes a matrix for describing the neural network topology structure.
[0042] It should be noted that the digital calculation module can be a hardware module for performing digital operations, and the digital operations include at least one of addition operation, subtraction operation, multiplication operation, and division operation.
[0043] In an optional embodiment, the second matrix includes a matrix for describing the neural network topology. Based on the digital calculation module operating on the second matrix according to the digital data, target data is obtained, that is, the operation on the neural network topology according to the digital data is realized. Specifically, it can be at least one of addition and addition of the digital data and the neural network topology matrix.
[0044] In an optional embodiment, the first matrix includes a matrix corresponding to the weights of the neural network, and the second matrix includes a matrix for describing the neural network topology. It can be understood that the first matrix has a higher tolerance to noise than the second matrix, the influence of the calculation accuracy of the first matrix on the overall calculation accuracy is less than the influence of the calculation accuracy of the second matrix on the overall calculation accuracy, and the calculation amount related to the first matrix is greater than the calculation amount related to the second matrix. For the same calculation, the power consumption of analog calculation is lower than that of digital calculation, and the calculation accuracy of analog calculation is lower than that of digital calculation. In this application, the first matrix is subjected to analog calculation, at the cost of a small loss of accuracy, greatly reducing the calculation power consumption and improving the calculation speed, and then the second matrix is subjected to digital calculation to ensure the calculation accuracy of the second matrix, so as to achieve a high data processing speed and energy efficiency without significantly degrading the overall calculation accuracy.
[0045] In some embodiments, it can be by separately storing each element in the second matrix into the digital calculation module, and then operating according to the digital data and the elements in the second matrix to obtain the target data. For a detailed introduction, please refer to the subsequent embodiments.
[0046] The data processing method provided by the embodiments of this application is based on an analog calculation module operating on the first matrix according to input data to obtain analog data, where the first matrix includes a matrix corresponding to the weights of the neural network; based on an analog-to-digital converter to convert the analog data into digital data; based on a digital calculation module operating on the second matrix according to the digital data to obtain target data, where the second matrix includes a matrix for describing the neural network topology. The data processing method of this application combines analog calculation and digital calculation, combines the advantage of low power consumption of the analog calculation module with the advantage of high calculation accuracy of the digital calculation module, and realizes a significant reduction in calculation power consumption without affecting the calculation accuracy.
[0047] Please refer to Figure 2 , Figure 2The figure shows a flowchart of a data processing method provided by an embodiment of the present application. The data processing method specifically includes steps S210 to S250.
[0048] Step S210: Store each element in the first matrix into the analog computing module respectively.
[0049] It should be noted that storing each element in the first matrix into the device corresponding to the element of the first matrix in the analog computing module. Specifically, the elements in the first matrix can be characterized by current signals or voltage signals, and different current signals or voltage signals represent different element values.
[0050] Step S220: Input the input data into the analog computing module, and based on the analog computing module, calculate the analog product of the input data multiplied by the first matrix as the analog data.
[0051] In some embodiments, input a numerical value into the analog computing module, and based on the analog computing module, calculate the multiplication of the numerical value and the first matrix, and obtain the multiplied data as the analog data. The analog data is voltage or current.
[0052] Step S230: Convert the analog data into digital data based on the analog-to-digital converter.
[0053] It should be noted that the number of analog-to-digital converters is not less than one. The number of analog-to-digital converters can correspond to the number of output terminals of the analog computing module. Each analog-to-digital converter converts the received analog signal and transmits the converted digital signal to the corresponding input terminal of the digital computing module respectively.
[0054] For example, the analog computing module includes a first output port and a second output port, so there can be two analog-to-digital converters. The first analog-to-digital converter is connected to the first output port of the analog computing module, and the second analog-to-digital converter is connected to the second output port of the analog computing module.
[0055] For some embodiments, one analog-to-digital converter can also include multiple conversion channels, so that the conversion of multiple output ports of the analog computing module can be completed through one analog-to-digital converter. Continuing with the previous example where the analog computing module includes a first output port and a second output port, for example, the analog-to-digital converter includes a first channel and a second channel. The first channel of the analog-to-digital converter is connected to the first output port of the analog computing module, and the second channel of the analog-to-digital converter is connected to the second output port of the analog computing module.
[0056] Step S240: Store each element in the second matrix into the digital computing module respectively.
[0057] It is understandable that the digital computing module may include registers, so that each element in the second matrix can be separately stored in the registers in the digital computing module that have a corresponding relationship with the elements of the second matrix.
[0058] In an optional embodiment, the digital computing module may include a systolic array, and the systolic array includes a plurality of processing units. The register of each processing unit is used to store the elements in the second matrix that have a second mapping relationship with the corresponding processing unit. The second mapping relationship is the mapping relationship between the processing unit and the elements in the second matrix. For a detailed introduction, reference can be made to the subsequent embodiments.
[0059] Step S250: Input the digital data into the digital computing module, and calculate the digital product of the digital data and the second matrix based on the digital computing module as the target data.
[0060] It should be noted that the target data is the data obtained after data processing. That is to say, the target data in this embodiment is the result of multiplying the digital data by the second matrix, and the digital data is the result of multiplying the input data by the first matrix.
[0061] Here, it should be noted that in principle, step S240 can be before step S210 or after step S210, and the embodiments of the present application do not make specific limitations.
[0062] In one embodiment, for the data processing method of a graph convolutional layer, please refer to Figure 3 , Figure 3 shows a kind of graph data; among them, the Graph Convolutional Network (GCN) is a deep learning model for processing graph-structured data. Traditional deep learning models are usually used to process vector data, while the graph convolutional network extends this idea to graph data. In graph data, each node represents an entity of the data, and each edge represents the relationship between entities. The goal of the graph convolutional network is to capture the structure and relationship between nodes in the graph by learning the representation of nodes. The core idea of the graph convolutional network is to update the representation of nodes by aggregating the neighbor information of nodes. Specifically, for each node, the graph convolutional network will aggregate the features of the node and its neighbor nodes, and update the representation of the node through a neural network layer. Such operations can be iterated multiple times to obtain a richer representation of the nodes.
[0063] Figure 3Shows a graph data with 7 nodes that the current graph convolutional layer needs to process. The forward calculation of each layer in the graph convolutional neural network can be expressed as: f(H, A) = σ(AHW). The H matrix stores the data of each node in the current graph data. f(H, A) is used as the data of each node in the graph data of the next graph convolutional layer when input to the next graph convolutional layer. The A matrix stores the connection relationships within the current graph data. Specifically, the A matrix represents the adjacency matrix of the graph data. W represents the trainable parameter matrix of the current graph convolutional layer
[0064] When calculating the forward calculation of each layer in the graph convolutional neural network, it is necessary to first calculate AHW. It can be understood that in the embodiments of the present application, the first matrix is the W matrix, the second matrix is the A matrix, and the input matrix can be the H matrix. Each element in the W matrix is respectively stored in the analog computing module, the H matrix is input into the analog computing module, and based on the analog computing module, the analog product HW of the multiplication of the H matrix and the W matrix is calculated as the analog data; based on the analog-to-digital converter, the analog data is converted into digital data; each element in the A matrix is respectively stored in the digital computing module, the digital data is input into the digital computing module, and based on the digital computing module, the digital product AHW of the multiplication of the digital data and the A matrix is calculated as the target data, and then the calculation of σ(AHW) is realized through the vector processor.
[0065] The data processing method provided by the embodiments of the present application stores each element in the first matrix in the analog computing module respectively, and stores each element in the second matrix in the digital computing module respectively. The input data is input into the analog computing module, and based on the analog computing module, the analog product of the multiplication of the input data and the first matrix is calculated as the analog data; based on the analog-to-digital converter, the analog data is converted into digital data; the digital data is input into the digital computing module, and based on the digital computing module, the digital product of the multiplication of the digital data and the second matrix is calculated as the target data. The present application combines the analog computing module to realize the calculation of the multiplication of the input data and the first matrix to obtain the analog data, then converts the analog data into digital data through the analog-to-digital converter, and finally realizes the operation of the multiplication of the digital data and the second matrix through the digital computing module to obtain the target data. The present application deploys the overall operation to the analog computing module and the digital computing module respectively, reduces the overall computing power consumption through the analog computing module, and ensures the overall computing accuracy through the digital computing module.
[0066] Please refer to Figure 4 , Figure 4The figure shows a flowchart of a data processing method provided by an embodiment of the present application. The method can be applied to a data processing system. For example, the processing module of the data processing system can be used as the execution entity of the data processing method. The data processing method specifically includes steps S410 to S450.
[0067] Step S410: Based on the analog computing module, perform an operation on a first matrix according to input data to obtain analog data. The first matrix includes a matrix corresponding to the weights of a neural network.
[0068] Among them, step S410 has been introduced in detail in the foregoing embodiments and will not be elaborated here.
[0069] Step S420: Convert the second matrix into a form of multiplying a scaling matrix and a fourth matrix. The scaling matrix is a diagonal matrix, and the fourth matrix is used as the second matrix.
[0070] It can be understood that the second matrix includes a matrix for describing the topology of the neural network. During the training process, problems such as gradient disappearance or gradient explosion may occur. Especially in a deep network, the problems of gradient disappearance or gradient explosion may lead to unstable training. In the present application, the second matrix is normalized by converting it into a form of multiplying a scaling matrix and a fourth matrix, so as to avoid the problem of unstable training.
[0071] It should be noted that converting the second matrix into a form of multiplying a scaling matrix and a fourth matrix includes one of the following: the second matrix = the fourth matrix * the scaling matrix, the second matrix = the scaling matrix * the fourth matrix, and the second matrix = the scaling matrix * the fourth matrix * the scaling matrix.
[0072] Step S430: Adjust the range of the analog-to-digital converter based on the diagonal elements of the scaling matrix to obtain a target analog-to-digital converter.
[0073] Furthermore, the full range of the analog-to-digital converter can be adjusted based on the ratio between the reciprocals of the diagonal elements of the scaling matrix to obtain a target analog-to-digital converter. Exemplarily, if the scaling matrix has two diagonal elements corresponding to two analog-to-digital converters, and the ratio between the reciprocals of the two diagonal elements is 1:2, then the ratio of the full ranges of the two analog-to-digital converters is adjusted to 1:2. The analog-to-digital converter after adjusting the range is the analog-to-digital converter.
[0074] If the range of the analog-to-digital converter is non-adjustable, after the analog data is converted into digital data through analog-to-digital conversion, the digital data needs to go through an additional operation of multiplying the digital data by a scaling matrix by a vector processor before being input into the digital calculation module. Then, the result after multiplication is input into the digital calculation module. Therefore, there is also a step of data storage and reading between the digital data after analog-to-digital conversion and the input of the digital calculation module, which greatly reduces the calculation speed and increases the calculation power consumption. Therefore, in the embodiment provided in this application, the range of the analog-to-digital converter can also be adjustable.
[0075] Exemplarily, the range of the analog-to-digital converter can be adjusted by a scaling matrix. Specifically, the range of the analog-to-digital converter can be adjusted by the elements on the diagonal of the scaling matrix. For example, adjusting the range of the analog-to-digital converter can include: adjusting the external reference voltage, adjusting the internal reference voltage, gain setting, or resolution setting, etc.
[0076] Exemplarily, convert the second matrix into the form of multiplying the fourth matrix by a scaling matrix, that is, the second matrix = the fourth matrix * the scaling matrix, and the scaling matrix is a diagonal matrix. For example, the scaling matrix is If the diagonal elements of the scaling matrix are d1, d2, d3, d4, d5, d6, d7 respectively, then the ratio of the full-scale ranges of the 7 analog-to-digital converters corresponding to the output end of the analog calculation module can be adjusted to 1 / d1, 1 / d2, 1 / d3, 1 / d4, 1 / d5, 1 / d6, 1 / d7.
[0077] Among them, adjusting the external reference voltage: The analog-to-digital converter usually has a reference voltage input pin, and the range can be adjusted by an external voltage reference source. Continuing with the above-mentioned scaling matrix as an example, if the diagonal elements of the scaling matrix are d1, d2, d3, d4, d5, d6, d7 respectively, then the ratio of the external voltage reference sources of the 7 analog-to-digital converters is adjusted to 1 / d1, 1 / d2, 1 / d3, 1 / d4, 1 / d5, 1 / d6, 1 / d7, and the specific magnitude of the external voltage reference source is selected according to actual requirements.
[0078] Adjusting the internal reference voltage: Some analog-to-digital converters have an internal reference voltage source, and different reference voltage values can be selected by setting registers or switch pins. By selecting different internal reference voltages, the range of the analog-to-digital converter can be adjusted. Continuing with the above-mentioned scaling matrix as an example, if the diagonal elements of the scaling matrix are d1, d2, d3, d4, d5, d6, d7 respectively, then the ratio of the internal reference voltages of the 7 analog-to-digital converters is adjusted to 1 / d1, 1 / d2, 1 / d3, 1 / d4, 1 / d5, 1 / d6, 1 / d7, and the specific magnitude of the internal reference voltage is selected according to actual requirements.
[0079] It should be noted that the method for adjusting the full scale of the analog-to-digital converter can be selected accordingly according to the type of the analog-to-digital converter.
[0080] Step S440: Convert the analog data based on the target analog-to-digital converter to obtain the digital data.
[0081] Furthermore, adjust the range of the analog-to-digital converter based on the diagonal elements of the scaling matrix, transmit the analog data to the analog-to-digital converter with the adjusted range, and obtain the digital data through the conversion of the analog-to-digital converter with the adjusted range. That is, the analog-to-digital conversion realizes the operation of multiplying the analog data by the scaling matrix.
[0082] Step S450: Based on the digital calculation module, perform an operation on the second matrix according to the digital data to obtain the target data, where the second matrix includes the matrix for describing the neural network topology structure.
[0083] It should be noted that the second matrix includes the matrix for describing the neural network topology structure. During the training process, problems such as gradient disappearance or gradient explosion may occur. Especially in deep networks, the problems of gradient disappearance or gradient explosion may lead to unstable training. In this application, the second matrix is transformed into the form of the product of a scaling matrix and a fourth matrix to perform normalization processing on the second matrix, avoiding the problem of unstable training.
[0084] Please refer to the introduction in the previous embodiment. The first matrix is the W matrix, and the second matrix is the A matrix. During the training process, problems such as gradient disappearance or gradient explosion may occur. Therefore, in this embodiment, the A matrix is normalized to obtain Then the forward calculation of each layer in the graph convolutional neural network can be expressed as: where D is Figure 3 the degree matrix of the graph data in In this embodiment, when calculating the fourth matrix is matrix, D is the scaling matrix, each element in the W matrix is respectively stored in the analog calculation module, the H matrix is input into the analog calculation module, and the analog product HW of the multiplication of the H matrix and the W matrix is calculated based on the analog calculation module as the analog data; the A matrix is transformed into the form of the product of the scaling matrix and the fourth matrix and the fourth matrix matrix is used as the second matrix, and based on the scaling matrix The diagonal elements of adjust the range of the analog-to-digital converter to obtain a target analog-to-digital converter, and based on the target analog-to-digital converter, convert the analog data HW to obtain the digital data Based on the digital calculation module according to the digital data DHW for the second matrix Perform operations to obtain target data Then it is implemented through a vector processor Calculation of.
[0085] Among them, the vector processor supports floating-point multiplication of vectors and non-linear activation functions, such as the ReLU function.
[0086] The data processing method provided by the embodiments of the present application, by normalizing the second matrix, converting the second matrix into the product of a scaling matrix and a fourth matrix, ensures that all eigenvectors in the second matrix are on the same scale, thereby making the network training more stable. After the analog data is converted by the analog-to-digital converter with adjusted range, the step of multiplying the analog data by the scaling matrix has been realized, reducing the step of storing the scaling matrix in the cache and reading data from the cache during digital calculation, greatly improving the calculation speed.
[0087] In one embodiment Figure 3 Shows a graph data with 7 nodes that needs to be processed by the current graph convolutional layer. To implement the forward calculation of each layer in the graph convolutional neural network The processing method of the data processing system is as Figure 5 Shown, the input data of the graph convolutional layer includes the first matrix as the W matrix, the input matrix as the H matrix, and the fourth matrix as Matrix, the scaling matrix is This data processing method specifically includes steps S510 to S5110. The input data of the graph convolutional layer includes the first matrix W matrix, the input matrix H matrix, and the fourth matrix D matrix.
[0088] Step S510: Obtain the input data of the graph convolutional layer, where the input data of the graph convolutional layer includes the first matrix, the input matrix, and the fourth matrix.
[0089] It should be noted that before executing the method of this instance, the first matrix W matrix has been deployed in the analog calculation module, that is, the elements of the first matrix have been stored in the memristor array.
[0090] Step S520: Store the scaling matrix and the fourth matrix in the on-chip cache.
[0091] Step S530: Load the scaling matrix in the cache into the analog-to-digital converter, and adjust the range of the analog-to-digital converter based on the scaling matrix.
[0092] Specifically, load the scaling matrix in the cache into an analog-to-digital converter with an adjustable range, and adjust the range of the analog-to-digital converter based on the diagonal elements of the scaling matrix.
[0093] Step S540: Input the input matrix into the analog computing module to convert it into a voltage signal.
[0094] It should be noted that different element values correspond to different voltage signals.
[0095] Step S550: Transmit the voltage signal to the memristor array for calculation to obtain analog data.
[0096] It should be noted that the analog data is HW.
[0097] Step S560: Transmit the analog data to the analog-to-digital converter with an adjusted range to convert it into digital data Y1, and store Y1 in the on-chip cache.
[0098] It should be noted that the digital matrix
[0099] Step S570: Load the digital data Y1 in the on-chip cache to the input end of the systolic array of the digital computing module.
[0100] Step S580: Load the fourth matrix in the on-chip cache into the systolic array of the digital computing module.
[0101] It should be noted that the systolic array includes multiple processing units. The register of each processing unit is used to store the elements in the fourth matrix that have a second mapping relationship with the corresponding processing unit. The second mapping relationship is the mapping relationship between the processing unit and the elements in the fourth matrix. Exemplarily, the register of the processing unit in the e-th row and f-th column of the systolic array is used to store the element in the e-th row and f-th column of the fourth matrix, where the value range of e is from 1 to E, E represents the total number of rows of the fourth matrix, and the value range of f is from 1 to F, F represents the total number of columns of the fourth matrix.
[0102] Step S590: Perform digital calculation in the systolic array to obtain digital data Y2, and output the digital data Y2 to the vector processor.
[0103] It should be noted that the digital matrix
[0104] Step S5100: Load the scaling matrix in the on-chip cache into the vector processor as an input.
[0105] Step S5110: Perform the multiplication operation of the digital data Y2 and the scaling matrix in the vector processor, and then perform the operation of the activation function to obtain the output matrix Y3.
[0106] It should be noted that the digital matrix Y3 is the output of the convolutional layer and also the output of the data processing system.
[0107] In the embodiments of the present application, the forward calculation of one layer in the graph convolutional neural network is realized. If multi-layer graph convolution calculation is to be realized, the digital matrix is used as the input matrix H matrix, and then the calculation is performed, so as to realize the training of the graph convolutional network. Without significantly affecting the calculation accuracy, the training speed is greatly improved and the power consumption is reduced.
[0108] Please refer to Figure 6 , Figure 6 FIG. shows a data processing system 600 provided by an embodiment of the present application. The data processing system 600 includes: an analog computing module 610, an analog-to-digital converter 620, and a digital computing module 630.
[0109] The analog-to-digital converter 620 is respectively connected to the analog computing module 610 and the digital computing module 630.
[0110] It should be noted that the analog-to-digital converter 620 is an analog-digital converter. An analog-digital converter is a type of device used to convert a continuous signal in analog form into a discrete signal in digital form. The analog-digital converter converts an analog signal into a digital signal representing a certain proportional voltage value, and the digital signal output may use different coding structures.
[0111] Exemplarily, the digital signal is represented in binary. Binary is a number system used in computers. It only uses two digits, 0 and 1, to represent numerical and logical states. In binary, the value of each bit is a power of 2, increasing sequentially from right to left. For example, the weight of the first bit is 2 0 , and the weight of the second bit is 2 1 , and so on. By combining the values on different bits, any numerical value and data can be represented. In a computer, all data and instructions are stored and processed in binary form.
[0112] Exemplarily, the digital signal is represented in Gray code. Gray code is a binary coding method in which only one bit is different between two adjacent numbers.
[0113] It should be noted that the analog-to-digital converter of the present application is used to convert analog signals into digital signals, which can be a current analog-to-digital converter or a voltage analog-to-digital converter, and the model of the analog-to-digital converter is not limited.
[0114] It should be noted that the analog calculation module 610 is an analog circuit capable of performing analog operations, and the analog operations include at least one of addition operation and multiplication operation. Specifically, the analog calculation module is a calculation module for processing analog data. Among them, the analog data is continuous. One example is that the analog data is one of pressure, temperature, voltage, speed, and weight, and the analog calculation module stores continuous physical quantities. The calculation elements in the analog calculation module are used to implement analog calculations. One example is that the calculation elements in the analog calculation module include at least one of a function generator, an integrator, a comparator, and a multiplier.
[0115] It should be noted that the digital calculation module 630 is a hardware module for performing digital operations, and the digital operations include at least one of addition operation, subtraction operation, multiplication operation, and division operation. Specifically, the digital calculation module is a calculation module for processing digital data. Among them, the digital data is discrete. One example is that the digital data is binary numbers 0 and 1, and the digital calculation module stores discrete physical quantities. The calculation elements in the digital calculation module are used to implement digital calculations. One example is that the calculation elements in the digital calculation module include at least one of an encoder, a decoder, a data distributor, and a data selector.
[0116] It should be noted that in the analog calculation module, the signal almost completely represents the real signal in the form of voltage or current according to a ratio. Compared with the digital calculation module affected by noise, the analog calculation module is more affected by noise. A small deviation in the signal will be quite significant, resulting in information loss. In contrast, the digital calculation module only depends on high and low levels. If an error in information transmission is to occur, the deviation of the signal must be at least half of the high level, and the specific size varies according to different circuit specifications. Therefore, the digital calculation module for quantifying information has a stronger resistance to noise than the analog calculation module. As long as the deviation does not exceed a certain specified value, the information will not be lost. In the digital calculation module, the noise can be reduced at each logic gate. Although the analog calculation module has a lower calculation accuracy than the digital calculation module, the power consumption of the analog calculation module is also lower than that of the digital calculation module.
[0117] Therefore, for the same calculation, the power consumption of the analog calculation module is lower than that of the digital calculation module, and the calculation accuracy of the analog calculation module is lower than that of the digital calculation module. The influence of the calculation accuracy of the first matrix on the overall calculation accuracy is less than that of the second matrix on the overall calculation accuracy, and the amount of calculation related to the first matrix is greater than that related to the second matrix. In this application, the first matrix is calculated analogously, sacrificing a part of the accuracy, greatly reducing its calculation power consumption and improving the computing power, and then the second matrix is calculated digitally to ensure that the overall calculation accuracy does not decrease significantly.
[0118] The data processing system 600 further includes a processing module 640, which is respectively connected to the analog calculation module 610, the analog-to-digital converter 620, and the digital calculation module 630. The processing module is used to execute any method described in the above method embodiments, which will not be elaborated here.
[0119] The data processing system provided by the embodiments of this application combines the analog calculation module and the digital calculation module, combines the advantages of low power consumption of the analog calculation module and high calculation accuracy of the digital calculation module, and realizes a significant reduction in calculation power consumption without significantly affecting the calculation accuracy, improving the processing ability.
[0120] Please refer to Figure 7 , Figure 7 FIG. shows a data processing system 700 provided by the embodiments of this application. The analog calculation module 610 includes a memristor array 710, and the memristor array 710 includes a plurality of memristors.
[0121] Each memristor is used to store an element in the first matrix that has a first mapping relationship with the corresponding memristor.
[0122] It should be noted that the memristor, also known as the memory resistor, is a passive electronic component. Like a resistor, a memristor can generate and maintain a safe current through a device. However, the difference from a resistor is that a memristor can "remember" the amount of charge passed previously after the power is turned off. The resistive switching characteristics and non-volatility of the memristor make it have great advantages in new computing modes such as in-memory computing.
[0123] Among them, the first mapping relationship is the mapping relationship between the memristor and the elements in the first matrix. Exemplarily, the element in the i-th row and j-th column of the first matrix containing at least one element is stored on the memristor in the i-th row and j-th column of the memristor array 710, where the value range of i is from 1 to M, M represents the total number of rows of the first matrix, and the value range of j is from 1 to N, N represents the total number of columns of the first matrix.
[0124] Furthermore, the memristor array 710 includes a 1T1R architecture and a 2T2R architecture. Among them, in the 1T1R architecture, there is one transistor and one memristor in a basic unit, and the 1T1R architecture can store each unsigned element. Among them, in the 2T2R architecture, there are two memristors, one positive and one negative, in a basic unit, and the 2T2R architecture can store each signed element.
[0125] The analog computing module 610 stores the elements of the first matrix in one of the 1T1R method and the 2T2R method. After each element in the first matrix is respectively stored in the analog computing module, the analog computing module calculates the product of the input data and the first matrix to obtain an analog product as the analog data.
[0126] The data processing system provided by the embodiment of the present application realizes analog computing through a memristor array, greatly reduces the number of memory accesses through the memristor, improves the ability to parallel process matrix-vector operations, improves the computing speed, and reduces the computing power consumption.
[0127] Please refer to Figure 8 , Figure 8 , which shows a data processing system 800 provided by the embodiment of the present application. The digital computing module includes a systolic array 810, and the systolic array includes a plurality of processing units, and adjacent processing units are connected to each other.
[0128] Each processing unit is used to store the element in the second matrix that has a second mapping relationship with the corresponding processing unit.
[0129] It should be noted that the second mapping relationship is the mapping relationship between the processing unit and the element in the second matrix. Exemplarily, the register of the processing unit in the e-th row and f-th column of the systolic array is used to store the element in the e-th row and f-th column of the second matrix, where the value range of e is from 1 to E, and E represents the total number of rows of the second matrix, and the value range of f is from 1 to F, and F represents the total number of columns of the second matrix.
[0130] The systolic array is used to calculate the product of the digital data and the second matrix as the target data.
[0131] It should be noted that the systolic array is a parallel computing architecture. The design inspiration of the systolic array comes from the pulsation phenomenon in biology, that is, a periodic change similar to the beating of the heart. The design goal of the systolic array is to improve the computing efficiency and performance by distributing computing tasks to multiple processing units and executing the computing in a highly parallel manner. The systolic array realizes parallel computing by dividing data into multiple subtasks and assigning them to different processing units. Each processing unit independently computes the task assigned to it and exchanges data and collaborates in computing with other processing units through a communication mechanism. The systolic array can be used to solve intensive computing problems, including matrix multiplication, vector addition, convolution operations, etc.
[0132] The systolic array is a network composed of multiple identical processing elements (PEs) according to a certain interconnection rule. The systolic array can be one of a one-dimensional linear shape, a two-dimensional triangular shape, a two-dimensional rectangular shape, a two-dimensional hexagonal shape, and a two-dimensional binary tree shape. The characteristics of the systolic array are as follows.
[0133] First, each processing unit of the systolic array is the same.
[0134] Second, each processing unit only communicates with its adjacent processing units. That is to say, the communication between processing units has locality and the communication is regular. If the communication is not local and irregular, the connection relationship of each processing unit in the network will be very chaotic, and it will also be difficult to perform layout and wiring in hardware.
[0135] Finally, each processing unit has its local memory, that is, some edges of the processing unit have delays, and the delays correspond to registers in hardware. In other words, the data storage of the systolic array has locality, and it is also a necessary condition for pipelined operation.
[0136] Thus, it can be seen that due to the high degree of pipelining and regularization between the processing units of the systolic array, the throughput rate of the systolic array is very large and it is easy to implement in a digital processing system.
[0137] In an alternative embodiment, please refer to Figure 9 , Figure 9 which shows a schematic diagram of the systolic array 810 provided by the embodiment of the present application. The systolic array 810 is a two-dimensional rectangle. Figure 9The shown systolic array 810 has 3×3 processing units 811. Each processing unit 811 in the systolic array 810 is the same, and each processing unit 811 communicates with its adjacent processing units 811. The processing unit in the first row and the first column only communicates with the processing unit in the second row and the first column and the processing unit in the first row and the second column. The specific data flow is that the processing unit in the first row and the first column receives input data and transmits the calculated data to the processing unit in the second row and the first column and the processing unit in the first row and the second column respectively. The processing unit in the second row and the second column only communicates with the processing unit in the first row and the second column, the processing unit in the third row and the second column, the processing unit in the second row and the first column, and the processing unit in the second row and the third column. The specific data flow is that the processing unit in the second row and the second column receives the data transmitted by the processing unit in the first row and the second column and the data transmitted by the processing unit in the second row and the first column respectively. The processing unit in the second row and the second column transmits the calculated data to the processing unit in the third row and the second column and the processing unit in the second row and the third column respectively.
[0138] It should be noted that Figure 9 The shown systolic array 810 is only an example. The systolic array in actual applications may also include other numbers of processing units, which are not specifically limited in the embodiments of the present application.
[0139] In an alternative embodiment, please refer to Figure 10 , Figure 10 shows the processing unit 811 provided by the embodiment of the present application. The processing unit 811 includes: an adder 910; a register 920; a multiplier 930. The register 920 is connected to the multiplier 930, and the multiplier 930 is connected to the adder 910. The register is used to store the elements in the second matrix that have a second mapping relationship with the corresponding processing unit. When the input data enters the processing unit, the input data is transmitted to the multiplier, and the input data is multiplied by the value in the register, and the multiplied result is transmitted to the adder. The adder sums the received data and takes the sum result as the result data of the processing unit. The processing unit 811 has two data receiving ends and two data output ends. The two data receiving ends are used to receive the data output by the two adjacent processing units respectively, and the two data output ends transmit the result data to the two adjacent processing units respectively.
[0140] The data processing system provided by the embodiment of the present application. The digital computing module includes a systolic array. The systolic array distributes the computing tasks to multiple processing units and executes the computing in a highly parallel manner, improving the computing efficiency and performance of the digital computing module.
[0141] Please refer to Figure 11 , Figure 11The processing unit 811 provided by an embodiment of the present application is shown. The processing unit 811 includes: an adder 910; a register 920; a data selector 940. The register 920 is connected to the control end of the data selector 940, and the output end of the data selector 940 is connected to the adder 910.
[0142] The register is used to store the elements in the second matrix that have a second mapping relationship with the corresponding processing unit, and the elements of the second matrix are 0 or 1. When the input data enters the processing unit, the input data is transmitted to the data terminal of the data selector 940, and the output of the data selector 940 is controlled by the signal stored in the register. The output of the data selector 940 transmits the output data to the adder, and the adder sums the received data and takes the sum result as the result data of the processing unit. The processing unit 811 has two data receiving ends and two data output ends. The two data receiving ends are used to receive the data output by the two adjacent processing units respectively, and the two data output ends transmit the result data to the two adjacent processing units respectively.
[0143] It should be noted that the selection signal of the data selector is provided by the register of each processing unit. The register stores the elements of the second matrix. The two inputs of the data selector are the input data and 0 respectively. Through the selection output of the data selector, the multiplication operation of the input data and the corresponding elements of the second matrix is realized. The calculation result of a single processing unit in the systolic array is added to the result of the previous processing unit in the vertical direction to generate a partial sum, which is propagated to the next processing unit in the vertical direction until the overall matrix multiplication calculation result of the systolic array is obtained.
[0144] In an alternative embodiment, in principle, the data selector can be a data selector of any model. Here, the only decisive criterion is that the second data input terminal of the data selector is connected to an input terminal of the processing unit, and the first data input terminal of the data selector is connected to a low level.
[0145] Preferably, the data selector is a MUX21 data selector, and the memory is a single-bit register.
[0146] The data processing method provided by the embodiment of the present application, the elements of the second matrix are 0 or 1, and each of the processing units of the systolic array includes: an adder; a register; a data selector, the control end of the data selector is connected to the register, and the data output end of the data selector is connected to the adder. The multiplier can be replaced by a data selector. On the one hand, the area of the data selector is much smaller than that of the multiplier. Secondly, the calculation delay of the data selector is less than that of the multiplier. Finally, the power consumption of the data selector is much less than that of the multiplier. For this reason, the present application replaces the multiplier in the processing unit of the systolic array with a data selector, realizing the saving of the area of the systolic array, reducing the calculation delay, and effectively reducing the power consumption and area of the digital calculation module.
[0147] In an alternative embodiment, a data processing system provided by the embodiment of the present application may further include a storage unit, which is used to store intermediate data in the data processing process. The intermediate data includes at least one of a first matrix, a second matrix, input data, a scaling matrix, a fourth matrix, analog data, and digital data. The storage unit may be at least one of on-chip cache or off-chip cache, which is not limited herein.
[0148] Please refer to Figure 12 , Figure 12 shows a structural block diagram of an electronic device 1200 provided by the embodiment of the present application. The electronic device 1200 may be a computing system. The electronic device 1200 in the present application may include one or more of the following components: a processor 1211, a memory 1212, and one or more application programs, wherein the processor 1211 is electrically connected to the memory 1212, and one or more programs are configured to execute the methods described in the embodiments of the foregoing test methods.
[0149] The processor 1211 may include one or more processing cores. The processor 1211 connects various parts within the entire electronic device 1200 through various interfaces and lines, and executes various functions of the electronic device 1200 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1212, and by invoking data stored in the memory 1212. Optionally, the processor 1211 may be implemented in at least one hardware form of a Field-Programmable Gate Array (FPGA) or a Programmable Logic Array (PLA). The processor 1211 may integrate a combination of one or more of a Central Processing Unit (CPU) and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, computer programs, etc.; the GPU is responsible for rendering and drawing the displayed content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1211 and may be implemented separately through a communication chip. Specifically, the above-described method may be executed by one or more processors 1211.
[0150] For some embodiments, the memory 1212 may include a Random Access Memory (RAM), and may also include a Read-Only Memory. The memory 1212 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1212 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created during the use of the electronic device 1200.
[0151] Please refer to Figure 13 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 1300, and the program code can be called by a processor to execute the method described in the above method embodiments.
[0152] The computer-readable storage medium 1300 can be an electronic memory such as a flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1300 includes a non-transitory computer-readable medium. The computer-readable storage medium 1300 has a storage space for program code 1310 that executes any of the method steps in the above-described methods. These program codes can be read from or written to one or more computer program products. The program code 1310 can be compressed in a suitable form, for example.
[0153] Please refer to Figure 14 , which shows a structural block diagram of a computer program product 1400 provided by an embodiment of the present application. The computer program product 1400 includes a computer program / instructions 1410, and when the computer program / instructions 1410 are executed by a processor, the steps of the above-described method are implemented.
[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data processing method, characterized in that, the method is applied to a data processing system, the data processing system includes: an analog computing module, a digital computing module, and an analog-to-digital converter, the analog-to-digital converter is respectively connected to the analog computing module and the digital computing module, and the method includes: Based on the analog computing module operating on a first matrix according to input data to obtain analog data, the first matrix includes a matrix corresponding to the weights of a neural network; Based on the analog-to-digital converter converting the analog data into digital data; Based on the digital computing module operating on a second matrix according to the digital data to obtain target data, the second matrix includes a matrix for describing the topological structure of the neural network.
2. The method according to claim 1, characterized in that, the step of based on the analog computing module operating on a first matrix according to input data to obtain analog data includes: Storing each element in the first matrix into the analog computing module respectively; Inputting the input data into the analog computing module, and based on the analog computing module, calculating an analog product of the input data multiplied by the first matrix as the analog data.
3. The method according to claim 1, characterized in that, the step of based on the digital computing module operating on a second matrix according to the digital data to obtain target data includes: Storing each element in the second matrix into the digital computing module respectively; Inputting the digital data into the digital computing module, and based on the digital computing module, calculating a digital product of the digital data multiplied by the second matrix as the target data.
4. The method according to claim 1, characterized in that, the range of the analog-to-digital converter is adjustable, and the step of based on the analog-to-digital converter converting the analog data into digital data includes: Converting the second matrix into a form of a scaling matrix multiplied by a fourth matrix, the scaling matrix is a diagonal matrix, and using the fourth matrix as the second matrix; Adjusting the range of the analog-to-digital converter based on the diagonal elements of the scaling matrix to obtain a target analog-to-digital converter; Converting the analog data based on the target analog-to-digital converter to obtain the digital data.
5. A data processing system, characterized in that, the system includes: an analog computing module, an analog-to-digital converter, and a digital computing module; the analog-to-digital converter is respectively connected to the analog computing module and the digital computing module; the system further includes a processing module, the processing module is respectively connected to the analog computing module, the analog-to-digital converter, and the digital computing module, and the processing module is used to execute the method according to any one of claims 1-4.
6. The system according to claim 5, characterized in that, the analog computing module includes a memristor array, and the memristor array includes a plurality of memristors; each memristor is used to store an element in the first matrix that has a first mapping relationship with the corresponding memristor.
7. The system according to claim 5, characterized in that, The digital calculation module includes a systolic array, and the systolic array includes a plurality of processing units, and adjacent processing units are connected to each other; Each of the processing units is configured to store elements in the second matrix that have a second mapping relationship with the corresponding processing unit; The systolic array is configured to calculate the product of the digital data and the second matrix as target data.
8. The system according to claim 7, wherein, the elements of the second matrix are 0 or 1; Each of the processing units includes: an adder; a register; a data selector, a control end of the data selector is connected to the register, and a data output end of the data selector is connected to the adder.
9. An electronic device, wherein, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory, the one or more applications are configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-4.
10. A computer-readable storage medium, wherein, the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1-4.