A computational method for a stored-source system of grouped forward gradient regression

Abstract

This invention discloses a computational method for a stored-source computational system for grouped forward gradient regression. The stored-source computational system for grouped forward gradient regression includes a variable grouping module, a non-volatile memory module, and an iteration module. The variable grouping module is used to group, orthogonalize, and map variable data. The non-volatile memory module stores variable data and calculates the cosine similarity between the residual vector and the variable matrix. The iteration module updates the regression coefficients and residual vector based on the calculated cosine similarity. In this invention, the non-volatile memory module uses low-precision, high-energy-efficiency analog circuits to perform computationally expensive operations, while the variable grouping module and the iteration module use high-precision digital computing systems to control the accuracy of the algorithm. This allows the system to efficiently solve the grouped forward gradient regression problem while effectively ensuring algorithm accuracy.

Description

Technical Field

[0001] This invention belongs to the field of analog circuit technology, and more specifically, relates to a calculation method for a stored-value system of grouped forward gradient regression. Background Technology

[0002] Group-Structured Forward Stagewise Regression (GSLR) is a commonly used algorithm in statistics and computer vision. It can be used to solve various data-intensive sparse model problems involving variable grouping, such as feature selection, sparse coding, sparse estimation, anomaly detection, and feature extraction. GSLR can select suitable grouping variables to enhance the accuracy and interpretability of statistical or machine learning models, achieving similar results to group lasso regression. However, GSLR involves large-scale matrix-vector multiplication operations, severely limiting its computational efficiency.

[0003] Novel in-memory computing systems based on non-volatile memory are an emerging and highly efficient system for processing data-intensive tasks, significantly reducing computational overhead and increasing processing speed. This system typically performs operations directly in non-volatile memory, thereby reducing data transfer and power consumption during computation. Furthermore, by utilizing Kirchhoff's laws and Ohm's law, the system performs matrix-vector multiplication operations in analog circuits with a time complexity of O(1), achieving computational acceleration. Currently, in-memory computing systems have gained attention in the fields of machine learning and microelectronics, occupying an important position in next-generation computing technologies.

[0004] However, in-memory computing systems face certain challenges in terms of computational accuracy when accelerating data-intensive tasks. For example, the precision of memory is limited, or the precision of digital-to-analog converters (DACs) and analog-to-digital converters (ADCs) is insufficient to represent the data. Therefore, how to improve the computational efficiency of grouped forward gradient regression while ensuring its computational accuracy using in-memory computing systems is a pressing issue that needs to be addressed. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a computational method for a stored-value system of grouped forward gradient regression, which can effectively ensure its computational accuracy while improving the computational efficiency of grouped forward gradient regression.

[0006] To achieve the above objectives, this invention provides a computational method for a stored-in computing system of grouped forward gradient regression, used to solve data-intensive sparse model problems. The stored-in computing system includes a non-volatile memory module and a variable grouping module and an iteration module whose operations are performed by a full-precision digital computing system. The non-volatile memory module includes a digital-to-analog converter, a non-volatile memory array, a transimpedance amplifier, and an analog-to-digital converter. The computational method includes the following steps:

[0007] (1) Based on the correlation between the variables in the dataset of the problem to be solved, the variable grouping module divides the variables in the dataset into several groups, then performs orthogonalization and normalization on the variables in each group, then maps the processed variable data to the finite conductance values ​​of the non-volatile memory array, and transmits the grouping information and the processed variable data to the iteration module; wherein, the iteration module is used to perform iterative calculation of the residual vector and regression coefficients during the calculation of the grouped forward gradient regression;

[0008] (2) The digital-to-analog converter first performs data mapping on the residual vector output by the iteration module and then converts it into a voltage signal input to the non-volatile memory array, so that the current signal output by the non-volatile memory array is the cosine similarity between the residual vector and each variable. Then the transimpedance amplifier converts the current signal into a voltage signal, and finally the analog-to-digital converter converts the voltage signal into a digital signal and transmits it to the iteration module.

[0009] (3) The iterative module calculates the L2 norm of the cosine similarity of each group of variables, then finds the variable group most related to the residual vector, then iteratively updates the regression coefficients of the variables in the group, and finally iteratively updates the residual vector based on the iterative update of the regression coefficients.

[0010] (4) Determine whether the number of iterations has reached the upper limit or whether the L2 norm of the residual vector is lower than the threshold, and determine whether to terminate the iteration loop;

[0011] (5) Repeat steps (2) to (4) until the number of iterations reaches the upper limit or the L2 norm of the residual vector is lower than the threshold, then output the regression coefficients to complete the problem solution.

[0012] In one embodiment, the problem to be solved includes feature selection, sparse coding, sparse estimation, anomaly detection, and feature extraction problems involving variable grouping.

[0013] In one embodiment, in step (1), the numerical mapping of the processed variable data is to set the mapping range based on the average of the absolute values ​​of the data, and to map the data to a limited precision range according to a certain ratio.

[0014] In one embodiment, in step (1), the orthogonalization of the groups is performed using Schmidt orthogonalization.

[0015] In one embodiment, in step (2), the numerical mapping of the residual vector is to set the mapping interval based on the average value of the absolute value of the data, and to map the data to a limited precision interval according to a certain ratio.

[0016] In one embodiment, the regression coefficients are initialized to 0; the residual vector is a vector of the difference between the true value and the estimated value, the estimated value vector is initialized to 0, and the residual vector is initialized to the true value vector.

[0017] In one embodiment, in step (3), the variables most relevant to the residual vector are grouped together by the group with the largest L2 norm of cosine similarity.

[0018] In one embodiment, in step (3), the regression coefficients of the variables in the most relevant variable group are updated based on the cosine similarity between the variable and the residual vector and the L2 norm.

[0019] In one embodiment, when performing the task of solving grouped forward gradient regression, the variable grouping module only needs to be called once to save the processed variable data to a non-volatile memory array.

[0020] In one embodiment, the non-volatile memory in the non-volatile memory array is a resistive switching memory, a phase-change memory, a NOR-FLASH, a spin-transfer torque memory, or a ferroelectric field-effect transistor.

[0021] The computation method of the in-memory computing system for grouped forward gradient regression provided by this invention has the following beneficial effects: (1) This in-memory computing system provides a method for accelerating the solution of grouped forward gradient regression using in-memory computing technology. Since grouped forward gradient regression has a wide range of applications in solving sparse model problems involving variable grouping, this in-memory computing system has high reusability; (2) When mapping high-precision data to low-precision voltage and conductance, this in-memory computing system will truncate some outlier values ​​and use limited precision to reflect the distribution of values ​​as much as possible. This method can overcome the impact of limited hardware precision to a certain extent; (3) This in-memory computing system makes full use of the advantages of mixed analog and digital computing. It uses low-precision, high-energy-efficiency analog computing to perform the computation with the largest computational overhead and uses a high-precision digital computing system to perform the computation that controls the precision of the algorithm. Therefore, this in-memory computing system can accelerate the solution of grouped forward gradient regression while having good computational accuracy. Attached Figure Description

[0022] Figure 1This is a block diagram of a storage system for grouped forward gradient regression provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a calculation method for a storage system for grouped forward gradient regression provided in an embodiment of the present invention;

[0024] Figure 3 This is a circuit schematic diagram of a non-volatile memory module provided in an embodiment of the present invention;

[0025] Figure 4 This is a circuit schematic diagram of a non-volatile memory for storing data according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the electrical conductance of a non-volatile memory for storing data, provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] To address the issue of low computational accuracy in traditional in-memory computing systems when accelerating data-intensive tasks, this invention provides a computational method for in-memory computing systems using grouped forward gradient regression. This method can be used to solve data-intensive sparse model problems, such as feature selection, sparse coding, sparse estimation, anomaly detection, and feature extraction involving variable grouping. Figure 1 As shown, the in-memory computing system provided by the present invention includes a variable grouping module, a non-volatile memory module, and an iteration module. The non-volatile memory module includes a digital-to-analog converter, a non-volatile memory array, a transimpedance amplifier, and an analog-to-digital converter.

[0029] In this embodiment, the operations in the variable grouping module are performed using a full-precision digital computing system. This module is used to group, orthogonalize, and map the variables in the dataset containing the problem to be solved. Variable grouping refers to grouping related variables into a single group; sometimes the same variable may be grouped into multiple groups simultaneously. Group orthogonalization involves orthogonalizing the variable data within a group, such as using Schmidt orthogonalization, ensuring that variables within each group are mutually orthogonal. Data mapping maps the variable data to a finite numerical precision for storage in non-volatile memory. Preferably, when performing the grouped forward gradient regression solution task, the variable grouping module only needs to be called once to save the data to the non-volatile memory array.

[0030] In the non-volatile memory module, the non-volatile memory array uses a finite conductance value to store the mapped data. Specifically, the difference in conductance between two memories can be used to store the data. It is also used to perform matrix-vector multiplication operations, i.e., to calculate the cosine similarity between the residual vector and the variable. During the grouped forward gradient regression operation, the non-volatile memory module uses a digital-to-analog converter to convert the residual vector data transmitted by the iteration module into a voltage signal. A transimpedance amplifier is used to convert the output current of the non-volatile memory array into an output voltage. An analog-to-digital converter is then used to convert this output voltage into a digital signal and transmit it to the iteration module. In this embodiment, the non-volatile memory in the non-volatile memory array can be various types of non-volatile memory, such as resistive random access memory (RRAM), phase-change memory (PCM), NOR-FLASH, spin-transfer torque memory (STT-MRAM), and ferroelectric field-effect transistor (FeFET).

[0031] The iteration module performs iterative calculations of the residual vector and regression coefficients during the grouped forward gradient regression operation. Specifically, it is used for grouped L2 norm calculation, maximum value determination, regression coefficient update, residual vector update, and data mapping. Specifically, the grouped L2 norm calculation calculates the L2 norm of the cosine similarity between each group's variable and the residual vector, based on the variable grouping module's results. Maximum value determination identifies the maximum L2 norm for each group, and the group corresponding to this maximum value is the variable group most relevant to the residual vector. Regression coefficient update updates the regression coefficients of the variables in the most relevant variable group, based on the cosine similarity and L2 norm between the variable and the residual vector. Residual vector update updates the residual vector accordingly based on the changes in the regression coefficient values. Data mapping maps the residual vector values ​​to a finite precision so that the digital-to-analog converter in the non-volatile memory module can convert them into voltage signals.

[0032] Specifically, the calculation method of the in-memory computing system provided in this embodiment includes steps S10 to S50, which are detailed below:

[0033] S10, based on the correlation between the variables in the dataset of the problem to be solved, the variable grouping module divides the variables in the dataset into several groups, then performs orthogonalization and normalization on the variables in each group, then maps the processed variable data to a finite number of values, and finally transmits the data to the non-volatile memory module, which uses the conductance value of the non-volatile memory to store the data, and transmits the grouping information and the processed variable data to the iteration module.

[0034] Preferably, the data only needs to be stored once during the grouped forward gradient regression operation.

[0035] S20, the digital-to-analog converter first performs data mapping on the residual vector output by the iteration module and then converts it into a voltage signal input to the non-volatile memory array, so that the current signal output by the non-volatile memory array is the cosine similarity between the residual vector and each variable. Then, the transimpedance amplifier converts the current signal into a voltage signal, and finally the analog-to-digital converter converts the voltage signal into a digital signal and transmits it to the iteration module.

[0036] Preferably, the residual vector is the difference between the true value vector and the estimated value vector. The estimated value vector is initialized to 0, and the residual vector is initialized to the true value vector.

[0037] S30, the iterative module calculates the L2 norm of the cosine similarity of each group of variables, then finds the variable group most relevant to the residual vector, then iteratively updates the regression coefficients of the variables in that group, and finally iteratively updates the residual vector based on the iterative update of the regression coefficients.

[0038] Preferably, when updating the regression coefficients of a variable, it is necessary to refer to the cosine similarity between the variable and the residual vector, as well as the L2 norm of the group in which it belongs.

[0039] S40, determine whether the number of iterations has reached the upper limit or whether the L2 norm of the residual vector is lower than the threshold, and determine whether to terminate the iteration loop.

[0040] Preferably, the judgment operation in step S40 can be performed every two or more iterations to determine whether the L2 norm of the residual vector is lower than the threshold.

[0041] S50. Repeat steps S20 to S40 until the number of iterations reaches the upper limit or the L2 norm of the residual vector is lower than the threshold. Then output the regression coefficients to complete the problem solution.

[0042] The computation method of the in-memory computing system for grouped forward gradient regression provided in this embodiment has the following beneficial effects: (1) This in-memory computing system provides a method for accelerating the solution of grouped forward gradient regression using in-memory computing technology. Since grouped forward gradient regression has a wide range of applications in solving sparse model problems involving variable grouping, this in-memory computing system has high reusability; (2) When mapping high-precision data to low-precision voltage and conductance, this in-memory computing system will truncate some outlier values ​​and use limited precision to reflect the distribution of values ​​as much as possible. This method can overcome the impact of limited hardware precision to a certain extent; (3) This in-memory computing system makes full use of the advantages of mixed analog and digital computing. It uses low-precision, high-energy-efficiency analog computing to perform the computation with the largest computational overhead and uses a high-precision digital computing system to perform the computation that controls the precision of the algorithm. Therefore, this in-memory computing system can accelerate the solution of grouped forward gradient regression while having good computational accuracy.

[0043] The technical solution of the present invention will be fully described below with reference to embodiments and illustrations.

[0044] Example 1

[0045] A stored-value system for grouped forward gradient regression includes: a variable grouping module, a non-volatile memory module, and an iteration module.

[0046] like Figure 1 As shown, this in-memory computing system comprises a digital system and an analog system. The digital system includes a variable grouping module and an iteration module; the analog system includes a non-volatile memory module.

[0047] The variable grouping module is used to group variables in a dataset and perform orthogonalization, normalization, and data mapping. The variable grouping module includes: variable grouping, group orthogonalization, and data mapping. Variable grouping divides related variables into several groups, with the same variable potentially belonging to different groups. Group orthogonalization orthogonals the variables within a group, for example, using Schmidt orthogonalization. Data mapping maps the numerical values ​​of the variables to finite conductance values ​​in non-volatile memory.

[0048] The non-volatile memory module stores variable data and performs matrix-vector multiplication operations based on Ohm's law and Kirchhoff's laws for analog circuits. Specifically, matrix data is stored as the conductance value G of the non-volatile memory, vector data is converted into the voltage V of the digital-to-analog converter (DAC), and the output current I is the value of the matrix-vector multiplication, calculated by the formula I = V1 × G1 + V2 × G2 + ... + V n ×G n The current, which is the result of the calculation, is first converted into a voltage by a transimpedance amplifier, and then sampled by an analog-to-digital converter to obtain a digital signal, which is the calculation result. The calculation result will be transmitted to the iteration module.

[0049] The iterative module updates the regression coefficients and residual vectors during the grouped forward gradient regression operation and transmits the residual vectors to the non-volatile memory module for further processing. Specifically, grouping L2 norm calculation involves calculating the L2 norm of the cosine similarity of the variables in each group; maximum value determination involves identifying the group with the highest L2 norm, which is the most relevant group to the residual vector; regression coefficient update refers to updating the regression coefficients of the most relevant variable group based on cosine similarity and L2 norm; residual vector update refers to updating the residual vectors corresponding to the updated regression coefficients; and data mapping maps the residual vector data to the finite voltage values ​​of the digital-to-analog converter and transmits the results to the non-volatile memory module.

[0050] Example 2

[0051] The operation method of the storage system for grouped forward gradient regression in Example 1.

[0052] The in-memory computing system for grouped forward gradient regression provided in this embodiment fully utilizes the advantages of mixed-signal computing. Specifically, it employs high-energy-efficiency, low-precision analog computation to perform the most expensive operations, while simultaneously using high-precision digital computation to perform operations that control the algorithm's precision. The flowchart for a dataset with only four variables is as follows: Figure 2 As shown.

[0053] Preferably, in the flowchart, y is a vector of true values. It is the residual vector. Here, V is the estimated value vector, X is the cosine similarity vector calculated by the non-volatile memory module, l1 is the L2 norm of the cosine similarity of the first group, l2 is the L2 norm of the cosine similarity of the second group, and l... j The L2 norm is the group with the largest L2 norm, β1, β2, β3, β4 are the regression coefficients of the four variables, ε is the step size, k is the number of iterations, and σ is the maximum L2 norm. 2 It is the threshold of the residual vector. Furthermore, in Figure 3 The current I is the cosine similarity calculated by the non-volatile memory module, and the sampled V is also the calculated cosine similarity.

[0054] More preferably, the flowchart of the calculation process is shown below:

[0055] S1, the variable grouping module first groups the variables. For example, if there are 4 variables X1, X2, X3, and X4 in the dataset, X1 and X2 are divided into the first group, and X3 and X4 are divided into the second group. Then, the variables within the group are subjected to Schmitt orthogonalization and normalization according to the group. Next, the processed data is mapped to the 8 conductance states of non-volatile memory. Finally, the processed data and grouping information are transmitted to the iteration module.

[0056] Preferably, the regression coefficients β1, β2, β3, and β4 are initialized to 0, and the estimated value vector... Initialized to 0, residual vector Initialize it as the true value vector y.

[0057] Preferably, when mapping the processed variable data to eight conductance states, the data is first mapped to 15 integers in the range [-7, 7] according to a certain ratio. For example... Figure 4 As shown, the final result will be the difference in conductance between the two non-volatile memories (G). + -G - ) to store the data of variable matrix X.

[0058] More preferably, such as Figure 5 As shown, the non-volatile memory has eight conductance states: 60μS, 80μS, ..., 200μS. When G... + Take 60 μS, G - When G is 80 μS, it represents -1; when G + Take 160 μS, G - A value of 60 μS represents 5.

[0059] S2 uses a non-volatile memory array to calculate the cosine similarity between the residual vector and the variable. First, the residual vector is converted into a voltage input to the non-volatile memory array via a digital-to-analog converter. Then, the output current value is the calculated cosine similarity. Next, the current passes through a transimpedance amplifier and an analog-to-digital converter to become a digital signal. Finally, this signal is transmitted to the iteration module as the calculated cosine similarity.

[0060] Preferably, when mapping the residual vector to voltage, (1) calculate the average value of the absolute values ​​of the vector elements in the residual vector, (2) set the mapping interval according to the average value and truncate the values ​​of some elements, (3) map the data to 15 integers in [-7,7] according to a certain ratio, and (4) convert the values ​​to voltage according to a certain ratio using a digital-to-analog converter.

[0061] Preferably, the non-volatile memory module calculates the cosine similarity based on Ohm's law and Kirchhoff's current law. For example... Figure 3 As shown, the output for each column is This refers to the cosine similarity between the residual vector and the variable.

[0062] S3, the iterative module first calculates the L2 norm of the cosine similarity of each group variable. Next, determine the largest L2 norm l j Let's assume it's l1; then the regression coefficients β1, β2 of the grouping variables X1, X2 will be determined based on the ratio of the cosine similarity V1, V2 to the L2 norm l1. The regression coefficients are then updated; finally, the residual vector is updated based on the changes in the regression coefficients.

[0063] S4, the iteration module determines whether the number of iterations k has reached the upper limit or whether the L2 norm of the residual vector is lower than the threshold, and determines whether to terminate the iteration loop.

[0064] S5. Repeat steps S2 to S4. When the iteration loop terminates, output the regression coefficients to complete the problem solution.

[0065] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A computing method of a compute-in-memory system for packet forward gradient regression, for solving data-intensive sparse model solving problems, characterized in that, The in-memory computing system includes a non-volatile memory module and a variable grouping module and an iteration module for operations performed by a full-precision digital computing system. The non-volatile memory module includes a digital-to-analog converter, a non-volatile memory array, a transimpedance amplifier, and an analog-to-digital converter. The computing method includes the following steps: (1) Based on the correlation between the variables in the dataset of the problem to be solved, the variable grouping module divides the variables in the dataset into several groups, then performs orthogonalization and normalization on the variables in each group, then maps the processed variable data to the finite conductance values ​​of the non-volatile memory array, and transmits the grouping information and the processed variable data to the iteration module; wherein, the iteration module is used to perform iterative calculation of the residual vector and regression coefficients during the calculation of the grouped forward gradient regression; (2) The digital-to-analog converter first performs data mapping on the residual vector output by the iteration module and then converts it into a voltage signal input to the non-volatile memory array, so that the current signal output by the non-volatile memory array is the cosine similarity between the residual vector and each variable. Then the transimpedance amplifier converts the current signal into a voltage signal, and finally the analog-to-digital converter converts the voltage signal into a digital signal and transmits it to the iteration module. (3) The iterative module calculates the L2 norm of the cosine similarity of each group of variables, then finds the variable group most related to the residual vector, then iteratively updates the regression coefficients of the variables in the group, and finally iteratively updates the residual vector based on the iterative update of the regression coefficients. (4) Determine whether the number of iterations has reached the upper limit or whether the L2 norm of the residual vector is lower than the threshold, and determine whether to terminate the iteration loop; (5) Repeat steps (2) to (4) until the number of iterations reaches the upper limit or the L2 norm of the residual vector is lower than the threshold, then output the regression coefficients to complete the problem solution.

2. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 1, characterized in that, The problems to be solved include feature selection, sparse coding, sparse estimation, anomaly detection, and feature extraction involving variable grouping.

3. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 1, characterized in that, In step (1), the numerical mapping of the processed variable data is based on the average value of the absolute value of the data to set the mapping range, and the data is mapped to a limited precision range according to a certain ratio.

4. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 1, characterized in that, In step (1), the orthogonalization of the groups is performed using Schmidt orthogonalization.

5. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 1, characterized in that, In step (2), the numerical mapping of the residual vector is based on the average value of the absolute value of the data to set the mapping interval, and the data is mapped to a limited precision interval according to a certain ratio.

6. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 1, characterized in that, The regression coefficients are initialized to 0; the residual vector is the vector of the difference between the true value and the estimated value, the estimated value vector is initialized to 0, and the residual vector is initialized to the true value vector.

7. The calculation method for the in-store computing system of grouped forward gradient regression according to claim 6, characterized in that, In step (3), the variables most relevant to the residual vector are grouped together by the group with the largest L2 norm of cosine similarity.

8. The calculation method for the in-store system of grouped forward gradient regression according to claim 7, characterized in that, In step (3), the regression coefficients of the variables in the most relevant variable group are updated based on the cosine similarity between the variable and the residual vector and the L2 norm.

9. The calculation method for the in-store system of grouped forward gradient regression according to claim 1, characterized in that, When performing the task of solving grouped forward gradient regression, the variable grouping module only needs to be called once to save the processed variable data to a non-volatile memory array.

10. The calculation method for the in-memory system of grouped forward gradient regression according to claim 1, characterized in that, The non-volatile memory in the non-volatile memory array adopts resistive switching memory, phase change memory, NOR-FLASH, spin-transfer torque memory or ferroelectric field-effect transistor.

Images