A multi-dimensional joint control method of a storage-computation integrated end-side inference chip

Abstract

A multi-dimensional joint control method of a storage-computation integrated end-side inference chip. The method comprises: collecting current running state parameters; generating a data residence score and determining the residence position of a data block in an on-chip cache, an on-chip local storage or an external storage; determining the quantization bit width of each network layer or output channel; jointly determining a control configuration, the control configuration at least comprising the working state, working voltage, working frequency and ADC conversion configuration of each storage-computation array bank; generating and issuing a control word to execute a current inference window; collecting inference feedback parameters and updating the next window control parameters. The present application realizes multi-dimensional closed-loop joint control of data residence, array gating, voltage frequency, quantization bit width and ADC conversion, and can effectively reduce the energy consumption of the end-side inference chip under the premise of meeting the inference accuracy and delay constraints.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computing chip technology, and in particular to a multi-dimensional joint control method for an in-memory computing edge inference chip. Background Technology

[0002] As AI inference tasks migrate to edge devices, smartwatches, wireless earphones, mobile terminals, and edge sensing nodes need to perform inference tasks such as voice wake-up, heart rate anomaly detection, image classification, and biometric recognition under limited battery capacity and heat dissipation conditions. These edge AI inference tasks typically involve a large number of convolution operations, matrix multiplications, fully connected computations, and frequent handling of intermediate feature maps, placing stringent demands on chip power consumption, inference latency, and inference accuracy.

[0003] In-memory computing architecture is considered one of the key technological paths to solve the energy efficiency bottleneck of edge inference chips. This architecture effectively reduces the long-distance transport of model weights between storage and computing units by performing multiply-accumulate operations directly inside or near the storage array, thereby alleviating the memory wall problem inherent in the traditional von Neumann architecture to some extent and reducing data transport power consumption.

[0004] However, while in-memory computing architectures can reduce data movement, chip operation faces uncertainties such as input data changes, temperature fluctuations, and task load switching. This necessitates dynamic coordination of multiple coupled dimensions, including data residency, array activation, voltage frequency, quantization bit width, and ADC conversion. In existing chips, these dimensions are typically managed independently by the cache controller, power management unit, quantization configuration module, and ADC controller. Cache scheduling lacks integration with access prediction and data movement energy consumption; quantization bit width and ADC conversion bit width lack linkage; and operational feedback such as actual power consumption, latency, ADC saturation rate, and output confidence is not used in a closed-loop manner for subsequent decisions. This open-loop, separate control approach prevents the chip from adaptively adjusting the configuration of each dimension according to the operating state, easily leading to problems such as precision loss in critical layers, energy redundancy in non-critical layers, or array resource idleness. This makes it difficult to continuously approach optimal energy efficiency in dynamic edge environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a multi-dimensional joint control method for in-memory computing edge inference chips. Under the constraints of accuracy and latency, multi-dimensional joint control reduces the power consumption of in-memory computing edge inference chips.

[0006] To achieve the above objectives, the present invention provides a multi-dimensional joint control method for in-memory computing edge inference chips, comprising: Within the inference window, collect current running status parameters; A data residency score is generated based on the historical access frequency of the data block to be accessed, the predicted probability of the data block being accessed in the next inference window, the energy cost of moving the data block from the current storage level, the size of the data block, and the predicted distance of the data block from the next use. The residency location of the data block in on-chip cache, on-chip local storage, or external storage is determined based on the data residency score. Based on the network layer sensitivity parameters, the quantization bit width is determined for each network layer or output channel while satisfying accuracy and delay constraints. Based on the data dwell location, the quantization bit width, the memory array utilization, the chip temperature, the power consumption budget, and the delay constraint, the control configuration of the current inference window is jointly determined. The control configuration includes at least the operating state, operating voltage, operating frequency, and ADC conversion configuration corresponding to each memory array bank or output channel. Control words are generated according to the control configuration and sent to the corresponding execution units to configure and execute the current inference window; After the current inference window ends, inference feedback parameters are collected, and the control parameters of the next inference window are updated based on the inference feedback parameters.

[0007] Furthermore, the operating status parameters include data access status, network layer status, array status, chip status, inference constraints, and ADC output status.

[0008] Furthermore, the data access status includes one or more of the following: historical access count, most recent access time, data size, current storage level, and expected reuse distance for each data block; The network layer state includes one or more of the following: current network layer identifier, network layer type, input tensor size, output tensor size, and network layer sensitivity parameter; The array status includes one or more of the following: storage array utilization, number of idle banks, number of activated banks, and number of dormant banks; The chip status includes one or more of the following: chip temperature, power supply status, current operating voltage, current operating frequency, and power consumption budget; The inference constraints include the maximum permissible inference delay and the minimum permissible inference accuracy or output confidence. The ADC output state includes one or more of the following: ADC output saturation rate, output dynamic range, quantization error estimate, and output confidence level.

[0009] Furthermore, the data residency score is calculated using the following formula: R_i(k) = a1 × F_i(k) + a2 × P_i(k) + a3 × C_i - a4 × M_i - a5 ×D_i(k); Where F_i(k) is the access frequency of data block i in the historical inference window, P_i(k) is the predicted probability of data block i being accessed in the next inference window, C_i is the energy cost of moving data block i from the current storage level to the vicinity of the in-memory array, M_i is the size of data block i, D_i(k) is the predicted distance of data block i from the next use, and a1, a2, a3, a4 and a5 are weighting coefficients.

[0010] Furthermore, the specific steps for determining the residence location of the data block in on-chip cache, on-chip local storage, or external storage based on the data residence score include: The residence location of the data block is determined based on the comparison results of the data residence score with the first threshold and the second threshold. When the data residence score is greater than or equal to the first threshold, the data block i is configured as hot data and resides in the on-chip cache or the cache near the memory array. When the second threshold is less than or equal to the data residence score and less than the first threshold, the data block i is configured as warm data and resides in the on-chip local storage or the array bank to be activated. When the data residence score is less than the second threshold, the data block i is configured as cold data and resides in the external storage, and prefetching is performed before expected use.

[0011] Furthermore, the predicted probability is output by the access prediction model after predicting the access probability of the data block in future inference windows. The input of the access prediction model includes the identifier of the data block, the access sequence of the most recent multiple inference windows, the network layer to which the data block belongs, the size of the data block, and the current storage level of the data block.

[0012] Furthermore, the network layer sensitivity parameter represents the accuracy loss estimate corresponding to the network layer under different quantization bit widths, and the quantization bit width is selected from any one of a plurality of optional quantization bit widths; the network layer sensitivity parameter is generated by calibration dataset in the offline stage, or corrected in the runtime stage based on output confidence, output entropy, key layer output distribution drift or quantization error estimate.

[0013] Furthermore, when selecting the quantization bit width for each network layer or output channel, the optimization objective is to minimize the sum of the computing power consumption and ADC conversion power consumption of each network layer or output channel, with the constraints being that the total accuracy loss does not exceed the allowable accuracy loss threshold and the inference delay does not exceed the maximum allowable inference delay. The formula is as follows: minimize: Σ E_layer(l, b_l) + Σ E_adc(l, b_l) subject to: Σ S_l(b_l) <= E_acc L_total <= L_max b_l ∈ B Where E_layer(l, b_l) is the memory power consumption of network layer l with quantization bit width b_l, E_adc(l,b_l) is the ADC conversion power consumption of network layer l with quantization bit width b_l, E_acc is the allowable accuracy loss threshold, L_total is the estimated inference delay, L_max is the maximum allowable inference delay, and B is the optional quantization bit width set.

[0014] Furthermore, when the output confidence is lower than the confidence threshold, the output entropy is higher than the entropy threshold, the key layer output distribution drift exceeds the drift threshold, the ADC saturation rate exceeds the saturation threshold, or the quantization error estimate exceeds the error threshold, the quantization bit width of the corresponding network layer or output channel is increased. When the output confidence is stable and the power consumption pressure is higher than the power consumption pressure threshold, reduce the quantization bit width of non-critical network layers or non-critical output channels.

[0015] Furthermore, the ADC conversion configuration also includes an ADC quantization threshold or quantization boundary, as well as a data prefetch start time and a prefetch data block identifier; the working state of each memory array bank includes an active, hold, or sleep state.

[0016] Furthermore, the control configuration is obtained by looking up a configuration table, solving an energy consumption model, or a combination of both. The formula for selecting the configuration target is as follows: minimize: E_move + E_compute + E_leak + E_adc subject to: A_est >= A_min L_est <= L_max T_est <= T_limit Where E_move is the data moving energy consumption, E_compute is the in-memory array computing energy consumption, E_leak is the idle array leakage energy consumption, E_adc is the ADC conversion energy consumption, A_est is the accuracy estimate or output confidence estimate, A_min is the minimum accuracy or confidence requirement, L_est is the inference delay estimate, L_max is the maximum allowable inference delay, T_est is the temperature estimate, and T_limit is the upper limit of temperature.

[0017] Furthermore, the control word includes: Cache control words are used to indicate data prefetching, data replacement, or data retention. Array gating control words are used to instruct each memory array bank to enter an active, hold, or sleep state; Voltage and frequency control word, used to indicate the operating voltage and operating frequency; The quantization control word is used to indicate the quantization bit width of the network layer or output channel; ADC control word, used to indicate ADC conversion bit width, quantization threshold or quantization boundary.

[0018] Furthermore, the inference feedback parameters include at least two of the following: actual inference delay, actual power consumption or current integral value, temperature change, data prediction hit rate, ADC saturation rate, quantization error estimate, output confidence, and array bank idle time.

[0019] Furthermore, the specific steps for collecting inference feedback parameters and updating the control parameters of the next inference window based on the inference feedback parameters after the current inference window ends include: When the data prediction hit rate is lower than the hit rate threshold, the weight coefficients used in the data residency score are updated. When the ADC saturation rate is higher than the saturation threshold, increase the ADC conversion bit width of the corresponding network layer or output channel, or adjust the corresponding quantization boundary. When the quantization error estimate is higher than the error threshold, increase the quantization bit width of the corresponding network layer or output channel; When the chip temperature approaches the upper temperature limit, reduce the number of active banks, reduce the operating frequency, or increase the data prefetch lead time. When the actual inference latency is lower than the latency limit and the output confidence is higher than the confidence threshold, reduce the quantization bit width of non-critical network layers or reduce the number of activation banks. When the actual inference latency approaches the latency limit, increase the residency priority of critical data blocks or increase the number of active banks. To achieve the above objectives, the present invention also provides an in-memory computing edge inference chip, comprising: Multiple independently gated in-memory array banks, on-chip cache, on-chip local storage, external storage interface, array gating unit, voltage and frequency control unit, quantization control unit and configurable ADC module; And a controller, which is configured to perform the method as described above.

[0020] Furthermore, the controller includes: The status acquisition module is used to collect the current running status parameters of the inference window; The data residency decision module, connected to the status acquisition module, is used to generate a data residency score and determine the residency location of the data block; The precision configuration module, connected to the data dwell decision module, is used to determine the quantization bit width of the network layer or output channel; A conversion configuration module, connected to the accuracy configuration module, is used to determine the ADC conversion configuration; A power consumption configuration module, connected to the conversion configuration module, is used to determine the operating status, operating voltage, and operating frequency of each memory array bank; A control word generation module, connected to the conversion configuration module and the power consumption configuration module, is used to generate control words; The feedback update module, connected to the status acquisition module, data dwell decision module, and precision configuration module, is used to update the quantization bit width, ADC conversion configuration, or the operating status, operating voltage, and operating frequency of the next inference window based on the inference feedback parameters.

[0021] The multi-dimensional joint control method for in-memory computing edge inference chips provided by this invention incorporates historical access frequency, next window prediction probability, data transfer energy consumption cost, data block size and predicted reuse distance into the decision through data residency scoring. This ensures that data with high reuse and high data transfer cost resides preferentially in on-chip cache or on-chip local storage, effectively reducing external storage access and cross-level data transfer, and significantly reducing data transfer power consumption.

[0022] The multi-dimensional joint control method for in-memory computing edge inference chips provided by this invention combines the quantization bit width of the network layer with the ADC conversion configuration under the same framework. This enables non-critical layers to use lower conversion bit widths to reduce redundant analog-to-digital conversion power consumption, while maintaining higher conversion bit widths for critical layers to avoid accuracy loss. This achieves a fine balance between power consumption and accuracy and reduces the risk of error accumulation in critical layers.

[0023] The multi-dimensional joint control method for in-memory computing edge inference chips provided by this invention incorporates the working state, working voltage frequency, and data prefetching time of the array bank into a unified control configuration, enabling idle array banks to enter hold or sleep states in a timely manner, and matching the working voltage frequency with the current delay margin, temperature state, and array utilization rate, thereby taking into account the synergistic reduction of array idle leakage power consumption and dynamic computing power consumption.

[0024] The multi-dimensional joint control method for in-memory computing edge inference chips provided by this invention transmits operational information such as actual power consumption, latency, temperature change, prediction hit rate, ADC saturation rate, quantization error, and output confidence to the next inference window, enabling control parameters to be continuously corrected according to the actual operating state of the chip, thus avoiding the problem that a fixed configuration table cannot adapt to the dynamic environment on the edge.

[0025] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0026] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the multi-dimensional joint control method for an in-memory computing edge-side inference chip according to an embodiment of the present invention. Figure 2 A diagram illustrating data block residency scoring and hierarchical storage; Figure 3 This is a schematic diagram illustrating the generation of control words; Figure 4 Update the timing diagram for feedback from the inference window; Figure 5 This is a schematic diagram of the structure of the in-memory computing edge inference chip according to an embodiment of the present invention; In the diagram: 10-In-memory array bank, 20-On-chip cache, 30-On-chip local storage, 40-External storage interface, 50-Array gating unit, 60-Voltage and frequency control unit, 70-Quantization control unit, 80-Configurable ADC module, 90-Controller, 100-Status acquisition module, 200-Data dwell decision module, 300-Precision configuration module, 400-Conversion configuration module, 500-Power consumption configuration module, 600-Control word generation module, 700-Feedback update module. Detailed Implementation

[0027] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0028] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "A plurality of" should be understood as two or more.

[0031] Terminology Explanation: An inference window refers to the smallest time unit for generating and issuing control configurations. It can be the computation cycle corresponding to a complete inference task, a network layer computation cycle, a tensor tile computation cycle, or a control cycle consisting of multiple consecutive network layers. Within each inference window, operations such as state acquisition, data dwell scoring, quantization bit width determination, control configuration generation, and control word issuance are performed. After the window ends, the parameters for the next window are updated based on feedback.

[0032] A memory computing bank refers to a basic memory computing array unit within a memory computing chip that can be independently gated and powered. Each bank contains one or more basic memory computing array units (such as SRAM memory arrays, RRAM memory arrays, or memristor memory arrays) and can directly perform multiply-accumulate operations within or near its memory cells. Multiple banks can be independently activated, maintained, or put into sleep mode.

[0033] A computing unit in an in-memory array refers to a basic unit within an in-memory array that can be independently gating and managed for power. This term, along with "in-memory array bank," refers to the same physical entity—the basic unit within the in-memory array that can be independently gating and managed for power. This document consistently uses "in-memory array bank" to refer to this entity. In specific descriptions involving operating states (active / standby / dormant), "bank" is an abbreviation for "in-memory array bank," and the two terms have the same meaning.

[0034] A basic in-memory array cell refers to a lower-level basic computing unit that constitutes an in-memory array bank, such as an SRAM in-memory array cell or an RRAM in-memory array cell. An in-memory array bank may contain one or more basic in-memory array cells.

[0035] A tile (data block) refers to a sub-block of data obtained by partitioning a large-scale tensor. In edge inference chips, due to the limited capacity of on-chip cache, it is impossible to accommodate the large weight matrix of an entire network layer or the ultra-large intermediate feature map at once. Therefore, large tensors need to be partitioned into multiple smaller data blocks (tiles) that can adapt to the size of the on-chip cache, and the computation is completed in a time-sharing manner. In this paper, "tile" mainly appears in the following scenarios: Model weight tile: a partial weight matrix after partitioning; Intermediate feature map tile: a partial input / output feature map after partitioning; Tensor tile computation cycle: the computation cycle in units of a single data block.

[0036] A data block refers to a data unit that needs to be scheduled and moved during inference, and its scope is broader than that of a tile. A data block can refer to a single tile or a complete data object that has not been segmented (such as the entire input data block, historical sensor data blocks, etc.). In this paper, "data block" is the basic object for data residency scoring and scheduling, including but not limited to model weight tiles, input data blocks, intermediate feature map tiles, network layer output channel data, and historical sensor data blocks.

[0037] Hot data / warm data / cold data: This classification is based on data residency scores, assessing the frequency of data block access and its reuse value. Hot data refers to data with high access frequency, short reuse distance, and high transportation cost, and should reside in the storage tier with the fastest access speed. Cold data refers to data with low access frequency, long reuse distance, and low transportation cost, and can reside in external storage. Warm data falls in between.

[0038] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0039] Example 1 One embodiment of the present invention provides a multi-dimensional joint control method for an in-memory computing edge inference chip, which is applied to the heart rate abnormality recognition task of a smartwatch.

[0040] The inference model consists of multiple one-dimensional convolutional layers and fully connected layers. The model weights, intermediate feature maps, and input heart rate windows are all scheduled by tiles.

[0041] The in-memory computing edge inference chip includes multiple independently gated in-memory array banks, on-chip cache, on-chip local storage, external Flash interface, on-chip temperature detection circuit, array gating unit, voltage and frequency control unit, quantization control unit, configurable ADC module and closed-loop joint control device.

[0042] In this embodiment, the on-chip cache is used to retain the active tiles of the current inference window, rather than storing all model weights at once. The closed-loop joint control device executes the multi-dimensional joint control method of the in-memory computing edge inference chip according to the following steps: S101: Collect current running status parameters within the inference window; In this embodiment of the invention, the operating state parameters include data access state, network layer state, array state, chip state, inference constraints, and ADC output state.

[0043] For data access status, it can include one or more parameters such as the historical access count of each data block, the most recent access time, the data size, the current storage level, and the expected reuse distance. For example, it can record the total number of times each data block has been accessed in the past inference window, the time of the most recent access, the storage space occupied by the data block, the current level of on-chip cache / on-chip local storage / external storage, and the time interval from the next use predicted based on the access sequence.

[0044] Network layer state includes one or more of the following parameters: current network layer identifier (e.g., the l-th convolutional layer), network layer type (convolutional layer / pooling layer / fully connected layer, etc.), input tensor size, output tensor size, and network layer sensitivity parameter.

[0045] The array status includes one or more parameters such as storage array utilization, number of idle banks, number of activated banks, and number of dormant banks.

[0046] Chip status includes one or more parameters such as chip temperature, power supply status, current operating voltage, current operating frequency, and power budget. Chip temperature can be acquired by an on-chip temperature detection circuit, which may include a PTAT temperature detection circuit, a ring resonator temperature detection circuit, or a bandgap-based temperature detection circuit.

[0047] Inference constraints include the maximum allowable inference delay and the minimum allowable output confidence or minimum allowable inference accuracy.

[0048] ADC output status, including one or more parameters such as ADC output saturation rate, output dynamic range, quantization error estimate, or output confidence level.

[0049] S102: Based on the historical access frequency of the data block to be accessed, the predicted probability that the data block will be accessed in the next inference window, the energy cost of moving the data block from the current storage level, the size of the data block, and the predicted distance of the data block from the next use, a data residency score is generated, and the residency location of the data block in on-chip cache, on-chip local storage, or external storage is determined based on the data residency score. like Figure 2 As shown, in this embodiment of the invention, the data residency score is calculated using the following formula: R_i(k) = a1 × F_i(k) + a2 × P_i(k) + a3 × C_i - a4 × M_i - a5 ×D_i(k); Where F_i(k) is the access frequency of data block i in the historical inference window, P_i(k) is the predicted probability of data block i being accessed in the next inference window, C_i is the energy cost of moving data block i from the current storage level to the vicinity of the in-memory array, M_i is the size of data block i, D_i(k) is the predicted distance of data block i from the next use, and a1, a2, a3, a4 and a5 are weighting coefficients.

[0050] The predicted probability P_i(k) is output by the access prediction model. The inputs to the access prediction model include the data block identifier, the access sequence of the most recent N inference windows, the network layer identifier, the data block size, and the current storage level of the data block. The access prediction model can be an LSTM, GRU, Transformer encoder, Markov prediction model, or sliding window statistical model.

[0051] The specific steps for determining the residence location of the data block in on-chip cache, on-chip local storage, or external storage based on the data residence score include: The residence location of the data block is determined based on the comparison results of the data residence score with the first threshold and the second threshold. When the data residence score is greater than or equal to the first threshold, the data block i is configured as hot data and resides in the on-chip cache or the cache near the memory array. When the second threshold is less than or equal to the data residence score and less than the first threshold, the data block i is configured as warm data and resides in the on-chip local storage or the array bank to be activated. When the data residence score is less than the second threshold, the data block i is configured as cold data and resides in the external storage, and prefetching is performed before expected use.

[0052] In this embodiment of the invention, the weight of the current tile (i.e., data block) and the current input heart rate window of the convolutional layer have a high access frequency and a high energy consumption cost in the current inference window. Their data residency score is greater than the first threshold, so they reside in the on-chip cache.

[0053] The data residency score of the tile weights to be calculated in the next convolutional layer is between the second threshold and the first threshold, so it resides in on-chip local storage and is prefetched before expected use.

[0054] Historical heart rate data retention scores were below the second threshold, so they were kept in external flash memory and prefetched only during anomaly reviews.

[0055] S103: Based on the network layer sensitivity parameters, determine the quantization bit width for each network layer or output channel while satisfying accuracy and delay constraints; In this embodiment of the invention, the network layer sensitivity parameter represents the estimated accuracy loss of the network layer under different quantization bit widths, wherein the quantization bit width is selected from any one of a plurality of optional quantization bit widths. The network layer sensitivity parameter is generated offline using a calibration dataset, or corrected during runtime based on output confidence, output entropy, key layer output distribution drift, or quantization error estimates.

[0056] In this embodiment of the invention, when selecting the quantization bit width for each network layer or output channel, the optimization objective is to minimize the sum of the computing power consumption and ADC conversion power consumption of each network layer or output channel, and the constraints are that the total accuracy loss does not exceed the allowable accuracy loss threshold and the inference delay does not exceed the maximum allowable inference delay. The formula is as follows: minimize: Σ E_layer(l, b_l) + Σ E_adc(l, b_l) subject to: Σ S_l(b_l) <= E_acc L_total <= L_max b_l ∈ B Where E_layer(l, b_l) is the memory power consumption of network layer l with quantization bit width b_l, E_adc(l,b_l) is the ADC conversion power consumption of network layer l with quantization bit width b_l, E_acc is the allowable accuracy loss threshold, L_total is the estimated inference delay, L_max is the maximum allowable inference delay, and B is the set of optional quantization bit widths.

[0057] In this embodiment of the invention, different quantization bit widths are configured for convolutional layers and fully connected layers based on an offline-generated network layer sensitivity table. When the output confidence is lower than a confidence threshold, the output entropy is higher than an entropy threshold, the output distribution drift of a critical layer exceeds a drift threshold, the ADC saturation rate exceeds a saturation threshold, or the quantization error estimate exceeds an error threshold, the quantization bit width of the corresponding network layer or output channel is increased. When the output confidence is stable and the power consumption pressure is higher than a power consumption pressure threshold, the quantization bit width of non-critical network layers or non-critical output channels is decreased.

[0058] When the output confidence decreases, the ADC saturation rate increases, or the quantization error estimate exceeds the threshold, the closed-loop joint control device increases the quantization bit width of the corresponding network layer or output channel; when the output confidence is stable and the power consumption pressure increases, the closed-loop joint control device reduces the quantization bit width of non-critical network layers or non-critical output channels.

[0059] S104: Based on the data dwell location, the quantization bit width, the memory array utilization rate, the chip temperature, the power consumption budget, and the delay constraint, jointly determine the control configuration of the current inference window. The control configuration includes at least the operating state, operating voltage, operating frequency, and ADC conversion configuration corresponding to each memory array bank or output channel. In this embodiment of the invention, the control configuration of the current inference window is determined based on the data residence location, quantization bit width, in-memory array utilization, chip temperature, power budget, and latency constraints. The control configuration includes at least the operating state, operating voltage, operating frequency of each in-memory array bank, and the ADC conversion configuration corresponding to each network layer or output channel. The ADC conversion configuration includes the ADC quantization threshold or quantization boundary, as well as the data prefetch start time and prefetch data block identifier. The operating state of the in-memory array computing unit includes active, hold, or sleep states.

[0060] The control configuration is obtained by looking up the configuration table, solving the energy consumption model, or a combination of both. The formula for selecting the configuration target is as follows: minimize: E_move + E_compute + E_leak + E_adc subject to: A_est >= A_min L_est <= L_max T_est <= T_limit Where E_move is the data moving energy consumption, E_compute is the in-memory array computing energy consumption, E_leak is the idle array leakage energy consumption, E_adc is the ADC conversion energy consumption, A_est is the accuracy estimate or output confidence estimate, A_min is the minimum accuracy or confidence requirement, L_est is the inference delay estimate, L_max is the maximum allowable inference delay, T_est is the temperature estimate, and T_limit is the upper limit of temperature.

[0061] In this embodiment of the invention, the array bank state, voltage frequency, and ADC conversion configuration are determined based on the current chip temperature, array utilization, delay constraints, and power budget. When the array utilization is low and the delay margin is large, only the array bank required for the current tile calculation is activated, banks not involved in the current calculation are switched to hold or sleep states, and the operating voltage and operating frequency are adjusted to a lower configuration that meets the delay constraints. For critical convolutional layer outputs, a higher conversion bit width is used in the ADC conversion configuration; for non-critical fully connected layer outputs, the ADC module can be configured to use a lower conversion bit width or a coarser quantization boundary.

[0062] In embodiments of the present invention, the ADC conversion configuration may include configuring a SAR ADC, a segmented quantization ADC, a flash ADC, a successive approximation ADC, or a configurable ADC based on non-volatile devices. The conversion bit width, quantization threshold, or quantization boundary are configured according to the ADC conversion configuration. The focus of this invention is on the control relationship of the ADC conversion configuration, and it is not limited to a specific ADC circuit structure.

[0063] S105: Generate control words according to the control configuration and send them to the corresponding execution units to configure and execute the current inference window; Control words include: Cache control words are used to indicate data prefetching, data replacement, or data retention. Array gating control words are used to instruct each memory array bank to enter an active, hold, or sleep state; Voltage and frequency control word, used to indicate the operating voltage and operating frequency; The quantization control word is used to indicate the quantization bit width of the network layer or output channel; ADC control word, used to indicate ADC conversion configuration, including conversion bit width, quantization threshold or quantization boundary.

[0064] During the current inference window, the on-chip cache performs data prefetching and replacement according to the cache control word; the array gating unit controls the working state of the in-memory array bank according to the array gating control word; the voltage and frequency control unit configures the working voltage and frequency according to the voltage and frequency control word; the quantization control unit configures the quantization bit width of the network layer or output channel according to the quantization control word; and the ADC configures the conversion bit width, quantization threshold, or quantization boundary according to the ADC control word.

[0065] like Figure 3 As shown in the figure, this diagram illustrates the process of generating control words. It shows the process of generating a set of control words using the input states and constraints to generate the same control configuration.

[0066] S106: After the current inference window ends, collect inference feedback parameters and update the control parameters of the next inference window based on the inference feedback parameters.

[0067] After the current inference window ends, inference feedback parameters are collected. These parameters include at least two of the following: actual inference latency, actual power consumption or current integral value, temperature change, data prediction hit rate, ADC saturation rate, quantization error estimate, output confidence, and array bank idle time.

[0068] Then, update the data dwell score, quantization bit width, or control configuration of the next inference window based on the inference feedback parameters.

[0069] In an embodiment of the present invention: When the data prediction hit rate is lower than the hit rate threshold, update the weight coefficients in the data residency score.

[0070] When the ADC saturation rate is higher than the saturation threshold, increase the ADC conversion bit width of the corresponding network layer or output channel, or adjust the corresponding quantization boundary.

[0071] When the quantization error estimate is higher than the error threshold, increase the quantization bit width of the corresponding network layer or output channel.

[0072] When the chip temperature approaches its upper temperature limit, reduce the number of active banks, reduce the operating frequency, or increase the data prefetch lead time.

[0073] When the actual inference latency is lower than the latency limit and the output confidence is higher than the confidence threshold, reduce the quantization bit width of non-critical network layers or reduce the number of activation banks.

[0074] When the actual inference latency approaches the latency limit, increase the residency priority of critical data blocks or increase the number of active banks.

[0075] Figure 4This is a timing diagram for the feedback update of the inference window. The diagram shows the closed-loop update process from inference window k to inference window k+1. After the current window finishes execution, feedback parameters such as actual power consumption, inference latency, temperature change, prediction hit rate, ADC saturation rate, and quantization error estimate are collected and used to update the control parameters of the next window.

[0076] In this embodiment of the invention, after the current inference window ends, the actual inference latency, actual power consumption, temperature change, data prediction hit rate, ADC saturation rate, quantization error estimate, and output confidence are recorded. When the data prediction hit rate is lower than the hit rate threshold in multiple consecutive inference windows, the weighting coefficient in the data dwell score is adjusted; when the ADC saturation rate is higher than the saturation threshold, the ADC quantization boundary of the corresponding layer is adjusted; when the chip temperature approaches the upper temperature limit, the number of activated banks in the next inference window is reduced or the operating frequency is lowered.

[0077] Through the above steps S101 to S106, the data residence location, array bank state, operating voltage frequency, quantization bit width, and ADC conversion configuration are updated in a closed loop between adjacent inference windows.

[0078] Example 2 One embodiment of the present invention provides a multi-dimensional joint control method for an in-memory computing edge inference chip, which is applied to a voice wake-up task. Voice wake-up devices are typically in a continuous listening state, requiring high power consumption during standby and continuous operation. The inference model includes a front-end feature extraction layer and a back-end verification layer.

[0079] In this embodiment, the power consumption budget weight is set higher than the accuracy margin weight. For the frequently accessed front-end feature extraction layer weight, the data dwell decision module dwells it in on-chip local storage or on-chip cache; for the infrequently accessed back-end confirmation layer weight, it is prefetched only when the confidence level of the initial wake-up determination exceeds a preset threshold.

[0080] In quiet environments or when output confidence is stable, the front-end feature extraction layer is configured with a low quantization bit width, and the ADC conversion is configured with a low conversion bit width, thereby reducing continuous monitoring power consumption. When increased ambient noise causes a decrease in output confidence or an increase in output entropy, the feedback update module increases the quantization bit width and ADC conversion bit width of the back-end confirmation layer in the subsequent inference window, and increases the dwell priority of the back-end confirmation layer weights.

[0081] When the chip temperature rises or the battery power decreases, the closed-loop joint control device reduces the number of activated banks, lowers the operating frequency, and extends the sleep time of some array banks to meet the power consumption constraints of continuous monitoring scenarios.

[0082] In this embodiment, the closed-loop joint control method performs control according to steps S101 to S1066 as described in Embodiment 1. In step S102, the weight coefficients in the data residence scoring formula are adaptively set according to different scenarios. The specific working method is the same as in Embodiment 1, and will not be repeated here.

[0083] Example 3 One embodiment of the present invention provides a multi-dimensional joint control method for an in-memory computing edge inference chip. This embodiment is applied to a low-resolution image classification task. In the image classification model, the convolutional layer feature maps and convolutional kernel tiles have a large amount of data, and the data transfer power consumption accounts for a high proportion.

[0084] In this embodiment, the weight of transport energy consumption in the data residency scoring formula is increased, so that highly reusable feature map tiles and convolutional kernel tiles are preferentially resided in on-chip cache or on-chip local storage. For the first convolutional layer and the last classification layer with high sensitivity, the accuracy configuration module configures a higher quantization bit width; for the intermediate low-sensitivity layers, a lower quantization bit width is configured.

[0085] When the image frame rate requirement decreases, the closed-loop joint control unit reduces the operating frequency and increases the array bank sleep time. When the image frame rate requirement increases, the closed-loop joint control unit increases the residency priority of key tiles and increases the number of active banks. After the inference window ends, the configuration for the next inference window is updated based on the actual latency, prediction hit rate, ADC saturation rate, and quantization error estimate.

[0086] In this embodiment, the closed-loop joint control device performs control according to steps S101 to S106 as described in Embodiment 1. The specific working method is the same as in Embodiment 1, and will not be repeated here.

[0087] Example 4 One embodiment of the present invention provides an in-memory computing edge inference chip. Figure 5 This is a schematic diagram of the structure of an in-memory computing edge inference chip, as shown below. Figure 5 As shown, the in-memory computing edge inference chip includes multiple independently gated in-memory arrays (bank10), an on-chip cache (20), on-chip local storage (30), an external storage interface (40), an array gating unit (50), a voltage and frequency control unit (60), a quantization control unit (70), a configurable ADC module (80), and a controller (90). The controller is used to execute the aforementioned multi-dimensional joint control method.

[0088] In an embodiment of the present invention, the controller 90 includes: The status acquisition module 100 is used to acquire the current running status parameters of the inference window; The data residency decision module 200, connected to the status acquisition module 100, is used to generate a data residency score and determine the residency location of a data block. The precision configuration module 300, connected to the data dwell decision module 200, is used to determine the quantization bit width of the network layer or output channel; The conversion configuration module 400, connected to the accuracy configuration module 300, is used to determine the ADC conversion configuration; The power consumption configuration module 500 is connected to the conversion configuration module 400 and is used to determine the operating status, operating voltage and operating frequency of each memory array bank. The control word generation module 600 is connected to the conversion configuration module 400 and the power consumption configuration module 500, and is used to generate control words; The feedback update module 700 is connected to the status acquisition module 100, the data dwell decision module 200, and the precision configuration module 300. It is used to update the quantization bit width, ADC conversion configuration, or the working status, working voltage, and working frequency of the next inference window based on the inference feedback parameters.

[0089] The above description is merely a partial embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0090] Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0091] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A multi-dimensional joint control method for an in-memory computing edge inference chip, characterized in that, include: Within the inference window, collect current running status parameters; A data residency score is generated based on the historical access frequency of the data block to be accessed, the predicted probability of the data block being accessed in the next inference window, the energy cost of moving the data block from the current storage level, the size of the data block, and the predicted distance of the data block from the next use. The residency location of the data block in on-chip cache, on-chip local storage, or external storage is determined based on the data residency score. Based on the network layer sensitivity parameters, the quantization bit width is determined for each network layer or output channel while satisfying accuracy and delay constraints. Based on the data dwell location, the quantization bit width, the memory array utilization, the chip temperature, the power consumption budget, and the delay constraint, the control configuration of the current inference window is jointly determined. The control configuration includes at least the operating state, operating voltage, operating frequency, and ADC conversion configuration corresponding to each memory array bank or output channel. Control words are generated according to the control configuration and sent to the corresponding execution units to configure and execute the current inference window; After the current inference window ends, inference feedback parameters are collected, and the control parameters of the next inference window are updated based on the inference feedback parameters.

2. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The operating status parameters include data access status, network layer status, array status, chip status, inference constraints, and ADC output status.

3. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 2, characterized in that, The data access status includes one or more of the following: historical access count, most recent access time, data size, current storage level, and expected reuse distance for each data block; The network layer state includes one or more of the following: current network layer identifier, network layer type, input tensor size, output tensor size, and network layer sensitivity parameter; The array status includes one or more of the following: storage array utilization, number of idle banks, number of activated banks, and number of dormant banks; The chip status includes one or more of the following: chip temperature, power supply status, current operating voltage, current operating frequency, and power consumption budget; The inference constraints include the maximum permissible inference delay and the minimum permissible inference accuracy or output confidence. The ADC output state includes one or more of the following: ADC output saturation rate, output dynamic range, quantization error estimate, and output confidence level.

4. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The data residency score is calculated using the following formula: R_i(k) = a1 × F_i(k) + a2 × P_i(k) + a3 × C_i - a4 × M_i - a5 × D_i(k); Where F_i(k) is the access frequency of data block i in the historical inference window, P_i(k) is the predicted probability of data block i being accessed in the next inference window, C_i is the energy cost of moving data block i from the current storage level to the vicinity of the in-memory array, M_i is the size of data block i, D_i(k) is the predicted distance of data block i from the next use, and a1, a2, a3, a4 and a5 are weighting coefficients.

5. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The specific steps for determining the residence location of the data block in on-chip cache, on-chip local storage, or external storage based on the data residence score include: The residence location of the data block is determined based on the comparison results of the data residence score with the first threshold and the second threshold. When the data residence score is greater than or equal to the first threshold, the data block i is configured as hot data and resides in the on-chip cache or the cache near the memory array. When the second threshold is less than or equal to the data residence score and less than the first threshold, the data block i is configured as warm data and resides in the on-chip local storage or the array bank to be activated. When the data residence score is less than the second threshold, the data block i is configured as cold data and resides in the external storage, and prefetching is performed before expected use.

6. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The predicted probability is output by the access prediction model after predicting the access probability of the data block in future inference windows. The input of the access prediction model includes the identifier of the data block, the access sequence of the most recent multiple inference windows, the network layer to which the data block belongs, the size of the data block, and the current storage level of the data block.

7. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The network layer sensitivity parameter represents the accuracy loss estimate of the network layer under different quantization bit widths. The quantization bit width is selected from any one of a plurality of optional quantization bit widths. The network layer sensitivity parameter is generated by calibration dataset in the offline stage, or corrected in the runtime stage based on output confidence, output entropy, key layer output distribution drift or quantization error estimate.

8. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 7, characterized in that, When selecting the quantization bit width for each network layer or output channel, the optimization objective is to minimize the sum of the computing power consumption and ADC conversion power consumption of each network layer or output channel, with the constraints being that the total accuracy loss does not exceed the allowable accuracy loss threshold and the inference delay does not exceed the maximum allowable inference delay. The formula is as follows: minimize: Σ E_layer(l, b_l) + Σ E_adc(l, b_l) subject to: Σ S_l(b_l) <= E_acc L_total <= L_max b_l ∈ B Where E_layer(l, b_l) is the memory power consumption of network layer l with quantization bit width b_l, E_adc(l, b_l) is the ADC conversion power consumption of network layer l with quantization bit width b_l, E_acc is the allowable accuracy loss threshold, L_total is the estimated inference delay, L_max is the maximum allowable inference delay, and B is the set of optional quantization bit widths.

9. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, When the output confidence is lower than the confidence threshold, the output entropy is higher than the entropy threshold, the key layer output distribution drift exceeds the drift threshold, the ADC saturation rate exceeds the saturation threshold, or the quantization error estimate exceeds the error threshold, increase the quantization bit width of the corresponding network layer or output channel. When the output confidence is stable and the power consumption pressure is higher than the power consumption pressure threshold, reduce the quantization bit width of non-critical network layers or non-critical output channels.

10. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The ADC conversion configuration also includes the ADC quantization threshold or quantization boundary, as well as the data prefetch start time and prefetch data block identifier; the working state of each memory array bank includes active, hold, or sleep states.

11. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The control configuration is obtained by looking up a configuration table, solving an energy consumption model, or a combination of both. The formula for selecting the configuration target is as follows: minimize: E_move + E_compute + E_leak + E_adc subject to: A_est >= A_min L_est <= L_max T_est <= T_limit Where E_move is the data moving energy consumption, E_compute is the in-memory array computing energy consumption, E_leak is the idle array leakage energy consumption, E_adc is the ADC conversion energy consumption, A_est is the accuracy estimate or output confidence estimate, A_min is the minimum accuracy or confidence requirement, L_est is the inference delay estimate, L_max is the maximum allowable inference delay, T_est is the temperature estimate, and T_limit is the upper limit of temperature.

12. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The control word includes: Cache control words are used to indicate data prefetching, data replacement, or data retention. Array gating control words are used to instruct each memory array bank to enter an active, hold, or sleep state; Voltage and frequency control word, used to indicate the operating voltage and operating frequency; The quantization control word is used to indicate the quantization bit width of the network layer or output channel; ADC control word, used to indicate ADC conversion bit width, quantization threshold or quantization boundary.

13. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The inference feedback parameters include at least two of the following: actual inference latency, actual power consumption or current integral value, temperature change, data prediction hit rate, ADC saturation rate, quantization error estimate, output confidence, and array bank idle time.

14. The multi-dimensional joint control method for an in-memory computing edge inference chip according to claim 1, characterized in that, The specific steps for collecting inference feedback parameters and updating the control parameters of the next inference window based on the inference feedback parameters after the current inference window ends include: When the data prediction hit rate is lower than the hit rate threshold, the weight coefficients used in the data residency score are updated. When the ADC saturation rate is higher than the saturation threshold, increase the ADC conversion bit width of the corresponding network layer or output channel, or adjust the corresponding quantization boundary. When the quantization error estimate is higher than the error threshold, increase the quantization bit width of the corresponding network layer or output channel; When the chip temperature approaches the upper temperature limit, reduce the number of active banks, reduce the operating frequency, or increase the data prefetch lead time. When the actual inference latency is lower than the latency limit and the output confidence is higher than the confidence threshold, reduce the quantization bit width of non-critical network layers or reduce the number of activation banks. When the actual inference latency approaches the latency limit, increase the residency priority of critical data blocks or increase the number of active banks.

15. A memory computing edge inference chip, characterized in that, The chip includes: Multiple independently gated in-memory array banks, on-chip cache, on-chip local storage, external storage interface, array gating unit, voltage and frequency control unit, quantization control unit and configurable ADC module; And a controller configured to perform the method of any one of claims 1 to 14.

16. The in-memory computing edge inference chip according to claim 15, characterized in that, The controller includes: The status acquisition module is used to collect the current running status parameters of the inference window; The data residency decision module, connected to the status acquisition module, is used to generate a data residency score and determine the residency location of the data block; The precision configuration module, connected to the data dwell decision module, is used to determine the quantization bit width of the network layer or output channel; A conversion configuration module, connected to the accuracy configuration module, is used to determine the ADC conversion configuration; A power consumption configuration module, connected to the conversion configuration module, is used to determine the operating status, operating voltage, and operating frequency of each memory array bank; A control word generation module, connected to the conversion configuration module and the power consumption configuration module, is used to generate control words; The feedback update module, connected to the status acquisition module, data dwell decision module, and precision configuration module, is used to update the quantization bit width, ADC conversion configuration, or the operating status, operating voltage, and operating frequency of the next inference window based on the inference feedback parameters.

Images