Model processing method, chip assembly and equipment

Abstract

The invention discloses a model processing method, a chip assembly and equipment. The method comprises the following steps: an in-memory computing element performs first operation processing on input data to obtain first processing data; wherein the first operation processing comprises fixed weight calculation of an operand; the near memory computing element performs second operation processing on the first processing data to obtain second processing data; wherein the second operation processing at least comprises attention calculation of operand change. Thus, heterogeneous deployment of model parameters of the model can be realized based on cooperation of the in-memory computing element and the near-memory computing element, the in-memory computing element can realize weight calculation of fixed operands, and the near-memory computing element can realize attention calculation of operand changes. The characteristics of high computing power efficiency and low power consumption of the in-memory computing element can be utilized to the greatest extent, the erasing operation on the in-memory computing element can be reduced by means of the near-memory computing element, and the application performance of a model deployed on the equipment side can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model processing method, chip component and device. Background Art

[0002] With the lightweight development of big model technology and the improvement of computing power resources on the terminal side, big models are expected to be deployed on the terminal side to achieve low latency, high reliability, personalization, low cost, and privacy protection, improve user experience, and enrich terminal product functions and services.

[0003] When the high-computing chip on the front end adopts the CPU (Central Processing Unit) + GPU (Graphics Processing Unit) + NPU (Neural Processor Unit) architecture, the CPU is mainly responsible for task control, the GPU is mainly responsible for signal processing, and the NPU is mainly responsible for large model calculations. The main calculations of large models are matrix-vector multiplication and addition calculations (accounting for more than 95%). The NPU achieves higher computing power through dedicated hardware structures such as matrix computing units and vector computing units, and reduces the overhead of data handling through data flow and memory optimization, which is suitable for parallel computing tasks such as neural networks.

[0004] Although the neural network computing module used in the NPU increases computing power by increasing the number of operations within a time step, it still uses the von Neumann architecture with separate storage modules and computing modules, and the storage wall and power consumption wall problems still exist. For large model inference calculations, due to the large number of model parameters and low parallelism, weight data needs to be repeatedly moved between storage and computing units. The CPU and memory bandwidth limits the speed of data movement, the NPU computing power cannot be fully utilized, and the energy consumption during the transfer process is relatively large, making it difficult to further improve the computing speed and energy efficiency. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a model processing method, a chip component and a device, aiming to improve the application performance of the model deployed on the terminal-side device.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a model processing method, including:

[0008] The in-memory computing element performs a first operation processing on the input data to obtain first processed data; wherein the first operation processing includes a weight calculation with fixed operands;

[0009] The near-memory computing element performs a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

[0010] In the above scheme, the method further includes:

[0011] Deploy the first model parameter of the model to be deployed to the in-memory computing element, so that the in-memory computing element performs the first operation processing;

[0012] Deploy the second model parameter of the to-be-deployed model to the near-storage computing element, so that the near-storage computing element performs the second operation processing;

[0013] The first model parameter corresponds to a processing block of important weights of the model to be deployed, and the second model parameter corresponds to a processing block of unimportant weights of the model to be deployed.

[0014] In the above scheme, the method further includes:

[0015] A weight importance evaluation is performed on each layer processing block of the model to be deployed to determine the first model parameter and the second model parameter.

[0016] In the above solution, the step of evaluating the weight importance of each layer processing block of the model to be deployed to determine the first model parameter and the second model parameter includes:

[0017] Evaluate the similarity between the output data of each layer processing block and the label data based on the test data set;

[0018] Based on the similarity and the set similarity threshold, determine the target processing block, and use the target processing block and the previous processing blocks in the model to be deployed as processing blocks with important weights, and the processing blocks after the target processing block as processing blocks with non-important weights; wherein the target processing block is the first processing block in the processing network of the model to be deployed whose similarity is greater than or equal to the set similarity threshold;

[0019] The first model parameters are generated based on the network parameters of the processing blocks with the important weights, and the second model parameters are generated based on the network parameters of the processing blocks with the unimportant weights.

[0020] In the above scheme, the method further includes:

[0021] The near-memory computing element generates an output token as next input data of the in-memory computing unit based on the first processed data and the second processed data.

[0022] In the above scheme, the method further includes:

[0023] The near-storage computing element determines whether to exit the second operation process corresponding to the current output token based on the termination classifier.

[0024] In the above solution, the near-storage computing element determines whether to exit the second operation processing corresponding to the output token based on the termination classifier, including:

[0025] The near-memory computing element, based on the feature data obtained from the current processing and the termination classifier, exits the second operation processing corresponding to the current output token if a termination mark is detected or the processing block of the model is determined to be completed.

[0026] In the above scheme, the method further includes:

[0027] A feature extraction module for extracting the feature data and the pre-trained termination classifier are deployed to the near-memory computing element.

[0028] In the above scheme, the method further includes:

[0029] Based on the test data set, the feature extraction module is used to extract feature data for the processing blocks of the non-important weights of the model to be deployed, and the model parameters of the termination classifier are trained based on the extracted feature data to obtain a trained termination classifier.

[0030] In a second aspect, an embodiment of the present application provides a chip component, including:

[0031] An in-memory computing element, the in-memory computing element being configured to perform a first operation process on the input data to obtain first processed data; wherein the first operation process comprises a weight calculation with fixed operands;

[0032] A near-memory computing element, wherein the near-memory computing element is configured to perform a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

[0033] In the above solution, the near-storage computing element is further configured as:

[0034] An output token is generated as next input data of the in-memory computing unit based on the first processed data and the second processed data.

[0035] In the above solution, the near-storage computing element is further configured as:

[0036] Determine whether to exit the second operation process corresponding to the current output token based on the termination classifier.

[0037] In the above solution, the near-storage computing element is specifically configured as follows:

[0038] Based on the feature data obtained from the current processing and the termination classifier, if a termination mark is detected or it is determined that the processing block of the model has been completed, the second operation processing corresponding to the current output token is exited.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising the chip assembly described in the second aspect of the embodiment of the present application.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: an in-memory computing element and a near-memory computing element, wherein the electronic device is configured to execute the steps of the method described in the first aspect of the embodiment of the present application when running a computer program.

[0041] The technical solution provided by the embodiment of the present application is that the in-memory computing element performs a first operation on the input data to obtain first processed data; wherein the first operation includes a weight calculation with fixed operands; and the near-memory computing element performs a second operation on the first processed data to obtain second processed data; wherein the second operation includes at least an attention calculation with varying operands. In this way, the heterogeneous deployment of the model parameters of the model can be achieved based on the coordinated cooperation of the in-memory computing element and the near-memory computing element, wherein the in-memory computing element can implement the weight calculation with fixed operands, and the near-memory computing element can implement the attention calculation with varying operands, which can maximize the use of the high efficiency and low power consumption characteristics of the in-memory computing element, and can also reduce the erase and write operations on the in-memory computing element with the help of the near-memory computing element, which can effectively improve the application performance of the model deployed on the device side. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a flow chart of a model processing method according to an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of the chip assembly of the application embodiment of the present application;

[0044] Figure 3 This is a flow chart of the model deployment method in the application embodiment of the present application;

[0045] Figure 4 This is a flow chart of the model reasoning method in the application embodiment of this application. DETAILED DESCRIPTION

[0046] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0048] In related technologies, the computing-in-memory (CIM) architecture implements computing inside the storage unit, completely solving the data transfer problem, and can significantly improve the inference efficiency and energy efficiency for memory-intensive computing. Due to the low integration of the storage and computing array, the storage unit cannot deploy all model parameters. The current storage-computing solution needs to use a small array multiple transfer method to achieve the calculation of large models on the end side.

[0049] It should be noted that the performance of the multiple transfer method to achieve large model calculations on the end side is still limited by bandwidth and cannot give full play to the advantages of the CIM architecture. In addition, for non-volatile memory devices such as FLASH (flash memory), RRAM (Resistive Random Access Memory), MRAM (Magnetoresistive Random Access Memory), PCM (Phase Change Memory), FeRAM (Ferroelectric RAM), etc., the device erase and write life is limited. Frequent updates of device parameters will limit the number of uses, making it difficult to achieve intermediate result calculations and repeated transfer calculations.

[0050] Based on this, in various embodiments of the present application, an architecture system that is not restricted by bandwidth is designed to target the large number of parameters and low parallelism of large models on the edge. The model parameters are heterogeneously deployed according to the importance of model weights, providing a low-latency, high-computing-power, and low-cost computing power solution for large models on the edge.

[0051] Here, the big models on the end side can be understood as machine learning models with a large number of parameters and complex computing structures. These models are usually built by deep neural networks and have a large number of parameters. The big models learn complex patterns and features by training massive data, have stronger generalization capabilities, and can handle more complex tasks and data, such as natural language processing, computer vision, speech recognition, and recommendation systems.

[0052] Exemplarily, the large model on the terminal side may be a Transformer-based model, for example, a GPT (Generative Pre-Trained Transformer) model.

[0053] The present application embodiment provides a model processing method, such as Figure 1 As shown, the method includes:

[0054] Step 101: the in-memory computing element performs a first operation on input data to obtain first processed data; wherein the first operation includes a weight calculation with fixed operands.

[0055] Step 102: the near-storage computing element performs a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

[0056] It can be understood that the electronic device applying the model processing method of the embodiment of the present application includes a computing-in-memory (CIM) element and a computing-near-memory (CNM) element, and the heterogeneous deployment of the model parameters of the model can be realized based on the coordinated cooperation of the computing-in-memory element and the computing-near-memory element, wherein the computing-in-memory element can realize the weight calculation of the fixed operand, and the computing-near-memory element can realize the attention calculation of the variable operand. Since the CIM element is responsible for the weight calculation of the fixed operand, which accounts for most of the calculation, and can achieve high efficiency and low energy consumption, the CNM element is responsible for the attention calculation of the variable operand, handles the high erase and write operations that CIM is not good at, and accounts for a small part of the calculation, it can maximize the use of the high efficiency and low power consumption of the computing power of the computing-in-memory element, and can reduce the erase and write operations of the computing-in-memory element with the help of the computing-near-memory element, which can effectively improve the application performance of the model deployed on the device side.

[0057] Exemplarily, the method further includes:

[0058] Deploy the first model parameter of the model to be deployed to the in-memory computing element, so that the in-memory computing element performs the first operation processing;

[0059] Deploy the second model parameter of the to-be-deployed model to the near-storage computing element, so that the near-storage computing element performs the second operation processing;

[0060] The first model parameter corresponds to a processing block of important weights of the model to be deployed, and the second model parameter corresponds to a processing block of unimportant weights of the model to be deployed.

[0061] Here, considering that each processing block in the large model structure has different effects on the output results, based on this, all layers of blocks can be divided into processing blocks with important weights and processing blocks with unimportant weights, where the processing blocks with important weights are necessary blocks in the reasoning process, and the processing blocks with unimportant weights are optional blocks in the reasoning process. The first model parameters corresponding to the processing blocks with important weights are deployed to in-memory computing elements, and the second model parameters corresponding to the processing blocks with unimportant weights are deployed to near-memory computing elements. In this way, the CIM integration scale requirements can be effectively reduced, and the reasoning requirements of large models with large parameters and low parallelism can be better met.

[0062] Exemplarily, the method further includes:

[0063] A weight importance evaluation is performed on each layer processing block of the model to be deployed to determine the first model parameter and the second model parameter.

[0064] Here, the electronic device may pre-evaluate the weight importance of each layer processing block of the model to be deployed, determine the first model parameter and the second model parameter, and further provide a basis for heterogeneous deployment of model parameters.

[0065] Exemplarily, performing weight importance evaluation on each layer processing block of the to-be-deployed model to determine the first model parameter and the second model parameter includes:

[0066] Evaluate the similarity between the output data of each layer processing block and the label data based on the test data set;

[0067] Based on the similarity and the set similarity threshold, determine the target processing block, and use the target processing block and the previous processing blocks in the model to be deployed as processing blocks with important weights, and the processing blocks after the target processing block as processing blocks with non-important weights; wherein the target processing block is the first processing block in the processing network of the model to be deployed whose similarity is greater than or equal to the set similarity threshold;

[0068] The first model parameters are generated based on the network parameters of the processing blocks with the important weights, and the second model parameters are generated based on the network parameters of the processing blocks with the unimportant weights.

[0069] Here, before the large model is deployed, all processing blocks of the large model can be tested based on the test data set to evaluate the consistency (i.e. similarity) between the output data of each processing block and the final output (i.e. label data). The processing blocks with significantly improved consistency are classified as processing blocks with important weights, otherwise they are classified as processing blocks with unimportant weights. For example, a similarity percentage k can be set, and the similarity between the output of the i-th layer of the large model and the final output can be compared in sequence starting from the first layer of the large model. When the consistency of the output of the m-th layer and after the m-th layer is higher than k, the block weights of the 1-m layers are important weights, and the block weights after the m-th layer are unimportant weights.

[0070] Exemplarily, the method further includes:

[0071] The near-memory computing element generates an output token as next input data of the in-memory computing unit based on the first processed data and the second processed data.

[0072] It should be noted that in the decoding stage of the large model, each autoregression will generate an output token, and the inferred output token and the input tokens will be concatenated together as the next input data. This process is repeated to realize the output inference until the end token is encountered or the output length reaches the set value.

[0073] Here, token (symbol or logo) refers to the smallest unit of information processed by the model, and through the Self-Attention mechanism, the model can learn the dependencies between tokens to better understand the input data. For example, in NLP (natural language processing), tokens can be words, characters, or subwords, while in CV (machine vision), tokens are patches cut from images.

[0074] Exemplarily, the method further includes:

[0075] The near-storage computing element determines whether to exit the second operation process corresponding to the current output token based on the termination classifier.

[0076] It should be noted that there is a redundant structure in the large model structure. For simple concepts, the latter few layers of blocks often do not work, but the large model reasoning calculation on the front end side adopts a complete calculation method, and redundant calculation increases the reasoning time and energy consumption. Based on this, in the embodiment of the present application, when the CNM element performs reasoning on a block with non-important weights, it can determine whether to exit the second operation processing corresponding to the current output token based on the termination classifier, so as to adaptively predict the reasoning termination time for the reasoning of simple concepts, reduce redundant layer calculations, and reduce CNM calculation amount, calculation delay and calculation energy consumption.

[0077] Exemplarily, the near-storage computing element determines whether to exit the second operation processing corresponding to the output token based on the termination classifier, including:

[0078] The near-memory computing element, based on the feature data obtained from the current processing and the termination classifier, exits the second operation processing corresponding to the current output token if a termination mark is detected or the processing block of the model is determined to be completed.

[0079] It can be understood that the near-memory computing element can obtain the feature data obtained by the current processing of conceptual reasoning, input the feature data into the termination classifier, and determine whether a termination mark is obtained. If not, continue to execute the next layer of blocks until the termination mark is detected or all blocks are completed, thereby ending the second operation processing corresponding to the output token.

[0080] Exemplarily, the method further includes:

[0081] A feature extraction module for extracting the feature data and the pre-trained termination classifier are deployed to the near-memory computing element.

[0082] It should be noted that the electronic device needs to pre-deploy the model's feature extraction module and termination classifier on the near-storage computing element, so as to achieve the adaptive prediction of the inference termination time in the above-mentioned model inference process and reduce the effect of redundant layer calculation.

[0083] Exemplarily, the method further includes:

[0084] Based on the test data set, the feature extraction module is used to extract feature data for the processing blocks of the non-important weights of the model to be deployed, and the model parameters of the termination classifier are trained based on the extracted feature data to obtain a trained termination classifier.

[0085] It should be noted that for processing blocks with non-important weights, feature data can be extracted based on test data using a feature extraction module, and the model parameters of the termination classifier can be trained based on the extracted feature data, thereby obtaining a trained termination classifier, so that the termination classifier has the ability to automatically determine whether the current processor block is a terminated processing block. Among them, the feature extraction module can select at least one of the number of block layers corresponding to the current processing block, the attention activation value of the current block, and the output data of the previous layer and the current layer; the termination classifier can be constructed using classic machine learning models such as SVM (Support Vector Machine), CRF (conditional random field), and RNN (Recurrent Neural Network), which is not limited in the embodiments of the present application.

[0086] Illustratively, an embodiment of the present application provides a chip assembly, including:

[0087] An in-memory computing element, the in-memory computing element being configured to perform a first operation process on the input data to obtain first processed data; wherein the first operation process comprises a weight calculation with fixed operands;

[0088] A near-memory computing element, wherein the near-memory computing element is configured to perform a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

[0089] It is understandable that since the CIM component is responsible for the weight calculation of fixed operands, accounting for most of the calculations, and can achieve high efficiency and low energy consumption, the CNM component is responsible for the attention calculation of the operand changes, processing the high erase and write operations that CIM is not good at, accounting for a small part of the calculations. Based on this, the chip component of the embodiment of the present application can maximize the high efficiency and low power consumption of the computing power of the in-memory computing element, and can reduce the erase and write operations on the in-memory computing element with the help of the near-memory computing element, which can effectively improve the application performance of the model deployed on the device side.

[0090] Exemplarily, the near-storage computing element is further configured as:

[0091] An output token is generated as next input data of the in-memory computing unit based on the first processed data and the second processed data.

[0092] Here, the near-memory computing element can concatenate the inferred output token and the input tokens together as the next input data, and repeat this process to implement output reasoning until the end token is encountered or the output length reaches the set value.

[0093] Exemplarily, the near-storage computing element is further configured as:

[0094] Determine whether to exit the second operation process corresponding to the current output token based on the termination classifier.

[0095] Here, when the CNM element performs reasoning on a block with non-important weights, it can determine whether to exit the second operation processing corresponding to the current output token based on the termination classifier, thereby adaptively predicting the reasoning termination time for the reasoning of simple concepts, reducing redundant layer calculations, and reducing CNM calculation amount, calculation delay, and calculation energy consumption.

[0096] Exemplarily, the near-storage computing element is specifically configured as follows:

[0097] Based on the feature data obtained from the current processing and the termination classifier, if a termination mark is detected or it is determined that the processing block of the model has been completed, the second operation processing corresponding to the current output token is exited.

[0098] It can be understood that the near-memory computing element can obtain the feature data obtained by the current processing of conceptual reasoning, input the feature data into the termination classifier, and determine whether a termination mark is obtained. If not, continue to execute the next layer of blocks until the termination mark is detected or all blocks are completed, thereby ending the second operation processing corresponding to the output token.

[0099] The present application is further described in detail below in conjunction with application examples.

[0100] Figure 2 FIG. 1 shows a schematic diagram of the structure of the chip assembly of this application embodiment. Figure 2 As shown, the chip components include a CIM computing module (corresponding to the aforementioned in-memory computing element), a CNM computing module (corresponding to the aforementioned near-memory computing element), a shared memory and controller (Shared Memory & Controller), a data transmission module (Pre / Post Process), a clock module (Clock Management Unit, CMU), etc., an input and output (I / O) module, etc.

[0101] The CIM computing module includes a CIM array (CIM Macro), a control module (Ctrl), a read / write module (R / W), a transmission module (Driver), and a cache module (Cache). The CIM array can use a digital storage and computing circuit or an analog storage and computing circuit. The storage and computing unit can use SRAM, FLASH, RRAM, MRAM, PCM, FeRAM and other devices. The CIM array deploys and computes the important weight layer of the large model on the end side.

[0102] The CNM computing module includes a storage unit, a computing unit, a termination judgment module, a control module, a read / write module, a transmission module, and a cache module. The CNM array deploys and computes the intermediate results (KV matrix) generated by the non-important weight layer and attention of the large model on the end side; the termination judgment module determines whether the current token reasoning ends early.

[0103] Reference Figure 3 In this application embodiment, the model deployment method includes the following steps:

[0104] Step 301, evaluate the importance of weights of each layer of the large model.

[0105] Before deploying the large model, all blocks of the large model are tested. The consistency between the output of each block and the final output is evaluated through the test set. The blocks with significantly improved consistency are recorded as important weights, otherwise they are recorded as unimportant weights. For example, set the percentage k, and compare the similarity between the output of the i-th layer of the large model and the final output in sequence starting from the first layer of the large model. When the consistency of the output of the m-th layer and after the m-th layer is higher than k, the weight of the block of the 1-m layers is important weight, and the weight of the block after the m-th layer is unimportant weight.

[0106] Step 302, terminate model building.

[0107] The termination model is used to determine whether the output result of the current block layer can be used as the final result for the current token reasoning, and dynamically decides whether the current token reasoning is terminated. The termination model includes a feature extraction module and a termination classifier.

[0108] Step 302 may specifically include:

[0109] Step 3021, the model terminates feature screening;

[0110] For example, the model termination feature can select the current layer attention activation value, the number of block layers, the output of the previous layer and the current layer for statistical analysis. For example, the logits of the output probability vector are selected to construct features, and the difference between the argmax probability value and the highest and second highest probabilities is determined.

[0111] Step 3022, terminating the classifier training;

[0112] For non-important block layers, classic machine learning models such as SVM, CRF, and RNN are used to build a termination classifier. The classifier input is the feature in step 3021 of the current block layer, and the classifier output is whether the output of the current block layer is consistent with the output of the final layer. The parameters of the termination classifier are trained according to the test set data to obtain a trained termination classifier.

[0113] Step 303, model deployment, deploying important weights on the CIM array, and deploying non-important weights and termination classifiers on the CNM array.

[0114] Here, the large model can be deployed to CIM and CNM. The CIM module deploys the important parameter calculation with fixed operands, and the CNM deploys the non-important parameter calculation and intermediate result calculation. For example, the important block layer weights in step 301 are deployed in CIM, which can include W of the important block layer. K , W Q , W V Matrix parameters, feedforward neural network W 1 , W 2 Matrix parameters; non-important block weights are deployed in CNM, and can include non-important block layers W K , W Q , W V Matrix parameters, feedforward neural network W 1 , W 2 Matrix parameters. Each block layer includes an attention module and a feedforward neural network. The attention module first calculates the intermediate result QKV vector of each token, and then multiplies the QKV vectors of different tokens to obtain the attention result, where Q is the query, K is the key, V is the value, and W is the K That is, the matrix parameters of the key matrix, W Q That is, the matrix parameters of the query matrix, W V That is, the matrix parameter of the value matrix, W 1 , W 2 They are the weight matrix parameters of the two input signals of the feedforward neural network respectively. The intermediate results KV calculated by CIM and CNM are deployed in CNM (including the KV of the input token and the KV of the output token).

[0115] It should be pointed out that feature extraction modules and termination classifiers are also deployed in the CNM array, which can reduce the reasoning process of simple tokens, adaptively predict the reasoning termination time, reduce redundant layer calculations, and reduce CNM calculation amount, calculation delay and calculation energy consumption.

[0116] Reference Figure 4 In this application embodiment, the model reasoning method includes the following steps:

[0117] Step 401, input processing.

[0118] For input processing of large models, weight-sharing parallel computing can be used to improve computing power utilization. All blocks in CIM and CNM participate in the calculation. The CIM module calculates the KV matrix of the important layer, and the CNM module calculates the KV matrix of the unimportant layer. All KV matrices are stored in CNM. After the calculation is completed, CNM obtains the first token.

[0119] Step 402, important block calculation.

[0120] Exemplarily, important block calculations specifically include:

[0121] 1) Input the newly output token into the CIM module and perform the first important block layer calculation according to W K , W Q , W V Matrix parameters, calculate the QKV vector of the new token and send it to the CNM module;

[0122] 2) The CNM module calculates the attention result based on the QKV vector of the new token and the stored KV matrix, and sends it back to the CIM module;

[0123] 3) The CIM module receives the attention calculation result and performs the feedforward neural network (FNN) calculation, repeats the calculation of the next important block layer, until all block calculations in the CIM module are completed, and sends the result to the CNM module.

[0124] Step 403: non-important block calculation.

[0125] Here, the CNM module inputs the results sent by the CIM module into the termination judgment model. If the classification result is termination, the reasoning stops; if the classification result is not termination, the CNM module calculates the non-important block layers one by one, and the calculation results of each block layer are sent to the termination judgment model until the termination mark appears or all non-important block layers are calculated. If the reasoning is terminated early, the KV value of the subsequent block layer copies the KV value of the termination layer and is stored in the CNM module.

[0126] Step 404, output processing.

[0127] Determine whether the current token is an end token. If it is not an end token, output the current token and repeat steps 402 to 404 until an end token is obtained or the output length reaches the set value.

[0128] It can be understood that this application embodiment is a heterogeneous reasoning system for large models on the edge, and adopts two new storage and computing integrated architectures, CIM+CNM, to collaboratively implement large model reasoning on the edge. CIM is responsible for weight calculation of fixed operands, which accounts for most of the calculations and achieves high efficiency and low energy consumption. CNM is responsible for attention calculation of changing operands, which accounts for a small part of the calculations and handles high erase and write operations that CIM is not good at.

[0129] This application embodiment adopts a weight evaluation method to divide block weights into important weights and unimportant weights, fixes the parameters of blocks with important weights in the CIM array, takes advantage of storage and computing integration, deploys the parameters of blocks with unimportant weights in the CNM, terminates reasoning in advance according to the termination model, and improves reasoning speed and energy efficiency.

[0130] Illustratively, an embodiment of the present application provides an electronic device, comprising the aforementioned chip assembly of the embodiment of the present application.

[0131] Illustratively, an embodiment of the present application provides an electronic device, comprising: an in-memory computing element and a near-memory computing element, wherein the electronic device is configured to execute the steps of the method described in the embodiment of the present application when running a computer program.

[0132] It is understandable that the chip assembly of the embodiment of the present application can provide a high-performance, low-energy computing infrastructure for the terminal, support the terminal to run large-model applications, and enhance product differentiation. The electronic device of the embodiment of the present application may include but is not limited to at least one of the following:

[0133] Smart speakers / smart central control screens: The large end-side model can help smart speakers realize applications such as voice interaction recognition, service content optimization, and smart home control. The CIM+CNM hybrid architecture chip can achieve a fast and stable application experience.

[0134] Smart cockpit: The large model on the terminal side can help the smart cockpit achieve applications such as improved safety, complex voice interaction, and autonomous driving assistance. The CIM+CNM hybrid architecture chip can reduce response delays and meet real-time driving requirements.

[0135] Home robots: Large models on the edge can help robots achieve real-time interaction and decision-making, complex task processing, and diverse application expansion. CIM+CNM hybrid architecture chips can improve the versatility of computing chips, support larger and more complex models, and reduce energy consumption on the edge.

[0136] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0137] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model processing method, characterized in that: include: The in-memory computing element performs a first operation processing on the input data to obtain first processed data; wherein the first operation processing includes a weight calculation with fixed operands; The near-memory computing element performs a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

2. The method according to claim 1, characterized in that The method further comprises: Deploy the first model parameter of the model to be deployed to the in-memory computing element, so that the in-memory computing element performs the first operation processing; Deploy the second model parameter of the to-be-deployed model to the near-storage computing element, so that the near-storage computing element performs the second operation processing; The first model parameter corresponds to a processing block of important weights of the model to be deployed, and the second model parameter corresponds to a processing block of unimportant weights of the model to be deployed.

3. The method according to claim 2, characterized in that The method further comprises: A weight importance evaluation is performed on each layer processing block of the model to be deployed to determine the first model parameter and the second model parameter.

4. The method according to claim 3, characterized in that The step of performing weight importance evaluation on each layer processing block of the model to be deployed to determine the first model parameter and the second model parameter includes: Evaluate the similarity between the output data of each layer processing block and the label data based on the test data set; Based on the similarity and the set similarity threshold, determine the target processing block, and use the target processing block and the previous processing blocks in the model to be deployed as processing blocks with important weights, and the processing blocks after the target processing block as processing blocks with non-important weights; wherein the target processing block is the first processing block in the processing network of the model to be deployed whose similarity is greater than or equal to the set similarity threshold; The first model parameters are generated based on the network parameters of the processing blocks with the important weights, and the second model parameters are generated based on the network parameters of the processing blocks with the unimportant weights.

5. The method according to claim 1, characterized in that The method further comprises: The near-memory computing element generates an output token as next input data of the in-memory computing unit based on the first processed data and the second processed data.

6. The method according to claim 5, characterized in that The method further comprises: The near-storage computing element determines whether to exit the second operation process corresponding to the current output token based on the termination classifier.

7. The method according to claim 6, characterized in that The near-storage computing element determines whether to exit the second operation process corresponding to the output token based on the termination classifier, including: The near-memory computing element, based on the feature data obtained from the current processing and the termination classifier, exits the second operation processing corresponding to the current output token if a termination mark is detected or the processing block of the model is determined to be completed.

8. The method according to claim 6, characterized in that The method further comprises: A feature extraction module for extracting the feature data and the pre-trained termination classifier are deployed to the near-memory computing element.

9. The method according to claim 8, characterized in that The method further comprises: Based on the test data set, the feature extraction module is used to extract feature data for the processing blocks of the non-important weights of the model to be deployed, and the model parameters of the termination classifier are trained based on the extracted feature data to obtain a trained termination classifier.

10. A chip assembly, characterized in that: include: An in-memory computing element, the in-memory computing element being configured to perform a first operation process on the input data to obtain first processed data; wherein the first operation process comprises a weight calculation with fixed operands; A near-memory computing element, wherein the near-memory computing element is configured to perform a second operation on the first processed data to obtain second processed data; wherein the second operation at least includes an attention calculation of operand changes.

11. The chip assembly according to claim 10, characterized in that: The near-storage computing element is further configured as: An output token is generated as next input data of the in-memory computing unit based on the first processed data and the second processed data.

12. The chip assembly according to claim 11, characterized in that: The near-storage computing element is further configured as: Determine whether to exit the second operation process corresponding to the current output token based on the termination classifier.

13. The chip assembly according to claim 12, characterized in that: The near-storage computing element is specifically configured as follows: Based on the feature data obtained from the current processing and the termination classifier, if a termination mark is detected or it is determined that the processing block of the model has been completed, the second operation processing corresponding to the current output token is exited.

14. An electronic device, characterized in that: Comprising the chip assembly according to any one of claims 10 to 13.

15. An electronic device, characterized in that: include: The electronic device comprises an in-memory computing element and a near-memory computing element, and the electronic device is configured to execute the steps of the method according to any one of claims 1 to 9 when running a computer program.

Images