Intermittent deep neural network reasoning method for energy harvesting Internet of Things

Abstract

The invention discloses an intermittent deep neural network reasoning method for an energy harvesting internet of things, and particularly relates to the field of neural network reasoning methods. Comprising the following steps: acquiring input data, model parameters and progress tracking variables of each layer of a deep neural network model from a nonvolatile memory; processing the input data through a plurality of sub-operations in each layer, updating a progress tracking variable when each sub-operation is completed, and writing the output of the sub-operations and the corresponding progress tracking variable into a nonvolatile memory; and if processing interruption occurs in the process of processing the input data through the plurality of sub-operations in each layer, determining the sub-operations which have been executed when the interruption occurs according to the progress tracking variable in the nonvolatile memory, and obtaining the input data and the model parameters required by the remaining sub-operations to complete the remaining sub-operations. Partition reasoning on the application program is not needed, and then energy estimation is avoided.

Description

Technical Field

[0001] The present application relates to the field of neural network reasoning methods, and in particular to an intermittent deep neural network reasoning method for energy harvesting Internet of Things. Background Art

[0002] In the field of smart IoT, IoT devices use various batteries as power sources and usually integrate sensors, microcontrollers and wireless communication functions. Battery technology faces several challenges: First, the limited life of the battery requires regular replacement, which is difficult and high-risk in some situations, such as pacemakers. Second, the environmental pollution caused by the use of a large number of batteries. After many micro IoT devices are put into use, their batteries are almost impossible to be recycled, causing serious pollution to the environment. With the widespread application of IoT devices, energy harvesting technology is used as an alternative energy solution. IoT devices driven by energy harvesting technology are collectively referred to as energy harvesting IoT devices. The core component of these energy harvesting IoT devices is the energy harvester, which is responsible for capturing energy from the environment in which the device is located, such as solar energy, wind energy, thermal energy or kinetic energy. In this way, energy harvesting IoT devices can use the natural resources of the surrounding environment to achieve self-powering, thereby reducing dependence on traditional batteries, reducing maintenance costs, and also reducing the impact on the environment. Energy harvesting IoT devices have been developed for a variety of devices, which are becoming more and more intelligent, and running complex deep neural network calculations on edge resource-constrained devices has become a future trend. Energy harvesting IoT devices are currently facing a problem that cannot be ignored: the energy collected from the surrounding environment is uncertain due to the variability of the objective environment, and the collected energy output is usually small and unstable. Due to the instability of the energy source, the device often suffers from power failure during task execution, resulting in program interruption and loss of computing context, and the calculation needs to be restarted from the beginning after the power is restored.

[0003] Checkpointing is a widely used technique in fault-tolerant computing. When program execution reaches a checkpoint, the system state is copied from volatile memory to non-volatile memory, and the program can resume execution from that point. This technique is widely used in intermittent computing. Checkpointing is programmer-friendly because it usually does not require programmers to specify specific data to be saved. This approach provides an effective strategy to solve the problem of computing interruptions caused by unstable power supply, especially for energy-harvesting IoT devices, which can ensure that the program can continue to run even when the power supply is frequently interrupted. Task-based breakpoint save and restore techniques rewrite the original application into a series of small, idempotent task sequences. The execution of a task is atomic, that is, it is either completed or aborted. Due to the idempotence of tasks, the key to these techniques is that the output of the task is not rewritten.

[0004] Traditional checkpoint-based intermittent computing techniques do not effectively reinitialize and wake up the accelerator, which may cause the CPU to fail to obtain the accelerator completion signal in time and make progress. In addition, static checkpoint settings may generate unnecessary overhead or be insufficient to ensure operational efficiency due to lack of adaptability, and the dynamic evaluation and optimization process itself will also bring significant time and energy overhead in energy-constrained environments. Task-based intermittent computing techniques rely heavily on accurate energy estimation of application partitions. However, deep neural networks have multiple computing layers with different levels of complexity, making it difficult to partition them appropriately. In addition, estimating the energy cost and available energy budget of tasks is not simple. Energy misestimation may lead to costly task re-execution, which affects the reasoning response time and increases the programming burden of programmers. Complex task division and management increase the difficulty of implementation and maintenance. Summary of the invention

[0005] The main purpose of this application is to provide an intermittent deep neural network reasoning method for energy harvesting Internet of Things, aiming to solve the problem of accurate energy estimation based on task-based intermittent computing technology relying on application partitioning.

[0006] To achieve the above-mentioned purpose, the present application provides an intermittent deep neural network inference method for an energy harvesting Internet of Things, including: obtaining input data, model parameters, and progress tracking variables of each layer of a deep neural network model from a non-volatile memory; processing the input data through multiple sub-operations in each layer, updating the progress tracking variables when each sub-operation is completed, and writing the output of the sub-operation and the corresponding progress tracking variables into the non-volatile memory; wherein the deep neural network model includes a convolutional layer, a pooling layer, and a fully connected layer, and the convolutional layer and the fully connected layer process the input data based on CPU operations and / or hardware accelerator operations, and the pooling layer processes the input data based on CPU operations; if a processing interruption occurs in the process of processing the input data through multiple sub-operations in each layer, the sub-operation that has been executed when the interruption occurs is determined according to the progress tracking variables in the non-volatile memory, and the input data and model parameters required for the remaining sub-operations are obtained to complete the remaining sub-operations.

[0007] Optionally, the progress tracking variable includes a first progress tracking variable and a second progress tracking variable.

[0008] Optionally, when the convolution layer processes the input data based on CPU operations, the method includes: obtaining the input data and model parameters of the convolution layer, as well as the first progress tracking variable and the second progress tracking variable from the non-volatile memory; extracting features through multiple sub-operations in sequence, updating the first progress tracking variable when each sub-operation is completed, and writing the output of the sub-operation and the corresponding first progress variable to the non-volatile memory; when all sub-operations are completed, obtaining an output feature map, updating the second progress tracking variable, and writing the second progress tracking variable to the non-volatile memory.

[0009] Optionally, when the convolution layer processes input data based on hardware accelerator operations, the method includes: obtaining the input data and model parameters of the convolution layer in the non-volatile memory, as well as the first progress tracking variable and the second progress tracking variable through direct memory access; extracting features through multiple sub-operations in sequence, updating the first progress variable after each sub-operation is completed, and writing the sub-operation results and the corresponding first progress variable into the non-volatile memory; when all sub-operations are completed, obtaining an output feature map, updating the second progress tracking variable, and writing the output feature map and the updated second progress tracking variable into the non-volatile memory through direct memory access.

[0010] Optionally, the sub-operation includes a multiplication operation and a corresponding accumulation operation.

[0011] Optionally, when the pooling layer processes the input data based on CPU operations, the method includes: obtaining the input data, model parameters and second progress tracking variables of the pooling layer from non-volatile memory; performing multiple sub-operations on the input data in sequence, generating and outputting an element at each position of the output feature map; taking the quotient and the remainder based on the product of the size of the current output feature map, and determining the position of the currently completed sub-operation; when each sub-operation is completed, updating the second progress tracking variable, and writing the output feature map of the current sub-operation and the corresponding second progress tracking variable into the non-volatile memory; wherein the sub-operation includes a pooling operation at each position.

[0012] Optionally, when the fully connected layer processes the input data based on CPU operations, the method includes: obtaining the input data and model parameters of the fully connected layer, and the second progress tracking variable from the non-volatile memory; traversing each output node, performing sub-operations on the input data corresponding to all input nodes one by one, and when the sub-operations are completed, updating the second progress tracking variable, and writing the output result of the sub-operation and the corresponding second progress tracking variable to the non-volatile memory; the sub-operations include weighted sum operations and applying an activation function, or the weighted sum operations and the sum is set to zero; wherein, when the value of the weighted sum is greater than zero, the output node applies the activation function, and when the value of the weighted sum is less than zero, the output node is set to zero.

[0013] Optionally, when the fully connected layer processes the input data based on hardware accelerator operations, the method includes: obtaining the input data and model parameters of the fully connected layer and the second progress tracking variable in the non-volatile memory through direct memory access; performing a sub-operation for each output channel, and after obtaining the sub-operation result of each output channel, updating the second progress tracking variable, and writing the sub-operation result and the corresponding second progress tracking variable into the non-volatile memory; wherein the sub-operation includes a multiplication-accumulation operation, and the result of the multiplication-accumulation operation is added to the bias value, and an activation function is applied.

[0014] Optionally, when executing the inference process of the deep neural network based on CPU operation, input data, model parameters, and progress tracking variables are all passed from the non-volatile memory to the corresponding layer of the deep neural network model through pointers.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: The intermittent deep neural network reasoning method for energy-harvesting Internet of Things of the present invention utilizes the unidirectional characteristics of deep neural network reasoning and the sequentiality of sub-operations to progressively advance the reasoning process without partitioning the reasoning of the application program, thereby avoiding energy estimation; the progress is calculated by the number of sub-operation completions, and the progress and the results of the corresponding sub-operations are stored in a non-volatile memory in a direction manner, and the CPU and the accelerator interact through the non-volatile memory to obtain the accelerator completion signal and make progress; only the sub-operation results are saved, and there is no need to save a large number of intermediate states, thereby minimizing the runtime cost and avoiding excessive space occupation; and the reasoning progress can be effectively captured and saved to avoid the progress loss caused by power interruption; after the interruption is recovered, the interruption position is determined according to the progress, the corresponding data of the unexecuted sub-operation is obtained, and it is passed to the CPU or hardware accelerator to recover the interrupted reasoning process, thereby improving the speed and accuracy of data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the intermittent deep neural network reasoning method for energy harvesting Internet of Things in this application; Figure 2 This is a comparison chart of execution time of Example 1 of the intermittent deep neural network reasoning method for energy harvesting Internet of Things of this application; Figure 3 This is a comparison chart of energy consumption of Example 1 of the intermittent deep neural network reasoning method for energy harvesting Internet of Things of this application; Figure 4 This is a graph of the inference throughput changes under different capacitances of Example 1 of the intermittent deep neural network inference method for energy harvesting Internet of Things of this application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] A single crystal solar panel with a rated output of 5V200mA is used as a power supply unit to simulate the solar energy collection process. The energy collection unit consists of a boost converter, a series of energy buffers, and a power control switch. In the energy collection intermittent computing system, considering that the output power of the solar panel is affected by many factors such as light intensity, weather conditions and angle, power fluctuations or unstable power supply may occur. Capacitors can store excess energy when the lighting conditions are good, and then release energy when the light is insufficient or there is no light, ensuring that the MCU obtains a continuous and stable power supply. The capacitor can act as an energy buffer to smooth the instantaneous fluctuations in the output of the solar panel and prevent power interruptions caused by short-term cloud cover or light changes. Under such a power supply configuration, the execution of a single complete reasoning process often cannot be completed in one go, so frequent power interruptions and recovery will occur. In view of the above situation, the present invention provides an intermittent deep neural network reasoning method for energy collection Internet of Things, such as Figure 1 As shown, the specific steps include: Before inference, you need to use the API to load the deep neural network model to perform inference of the deep neural network model. The specific inference method is as follows.

[0020] Step S1, obtaining input data, model parameters, and progress tracking variables of each layer of the deep neural network model from the non-volatile memory; processing the input data through multiple sub-operations in each layer, updating the progress tracking variables when each sub-operation is completed, and writing the output of the sub-operation and the corresponding progress tracking variables into the non-volatile memory; wherein the deep neural network model includes a convolutional layer, a pooling layer, and a fully connected layer, the convolutional layer and the fully connected layer process the input data based on CPU operations and / or hardware accelerator operations, and the pooling layer processes the input data based on CPU operations.

[0021] Exemplarily, the input of the convolutional layer may be an image, and the deep neural network model inference method of this embodiment may be used to perform image recognition, etc.

[0022] In this embodiment, the reasoning of the deep neural network model can be performed based on the CPU operation, or based on the CPU operation and the hardware accelerator operation. Specifically, when the hardware accelerator is enabled, the CPU enters a sleep state and is awakened after the hardware accelerator operation is completed. In addition, the update of the progress tracking variable can be achieved through a counter. Initially, the progress tracking variable is set to 0, and an update is performed once each sub-operation is completed, that is, the progress tracking variable is incremented, and the operation result of the sub-operation is saved at the same time. In this way, when the reasoning process is interrupted, the interruption position and the corresponding operation result of the completed sub-operation can be found according to the progress tracking variable, and then the calculation can be continued from the correct position. Utilizing the unidirectional characteristics of deep neural network reasoning and the sequential nature of sub-operations, the reasoning process is gradually advanced, and only the result of each sub-operation is saved, without the need to save a large number of intermediate states.

[0023] Furthermore, when the inference process of the deep neural network is executed based on CPU operation, the input data, model parameters, and progress tracking variables are all passed from the non-volatile memory to the corresponding layer of the deep neural network model through a pointer. The pointer contains the receiving address of the input data, model parameters, and progress tracking variables, and points to the corresponding processing layer.

[0024] The progress tracking variable includes a first progress tracking variable and a second progress tracking variable. When the convolution layer processes the input data based on CPU operation, the inference method of the convolution layer specifically includes the following steps: The input data and model parameters of the convolution layer, as well as the first progress tracking variable and the second progress tracking variable are obtained from the non-volatile memory through the pointer; the features are extracted through multiple sub-operations in sequence, the first progress tracking variable is updated when each sub-operation is completed, and the output of the sub-operation and the corresponding first progress variable are written to the non-volatile memory; when all sub-operations are completed, that is, when the convolution operation is completed, the feature map is output, the second progress tracking variable is updated, and the second progress tracking variable is written to the non-volatile memory.

[0025] Specifically, the input data of the convolution layer can be the original input data or the output data of the upper layer. Its model parameters include weights (convolution kernels) and biases. The sub-operation includes a multiplication operation and a corresponding accumulation operation, that is, each element-by-element multiplication of the elements in the area covered by the convolution kernel on the input data and the corresponding accumulation operation.

[0026] When the convolutional layer processes input data based on hardware accelerator operations, the methods include: The input data and model parameters of the convolution layer in the non-volatile memory, as well as the first progress tracking variable and the second progress tracking variable are obtained through direct memory access; the features are extracted through multiple sub-operations in sequence, the first progress variable is updated after each sub-operation is completed, and the sub-operation results and the corresponding first progress variable are written into the non-volatile memory; when all sub-operations are completed, the output feature map is obtained, the second progress tracking variable is updated, and the output feature map and the updated second progress tracking variable are written into the non-volatile memory through direct memory access. Among them, the sub-operation includes a multiplication operation and a corresponding accumulation operation.

[0027] After the output feature map and the updated second progress tracking variable are written to the non-volatile memory through direct memory access, the CPU needs to be awakened to execute the reasoning of the pooling layer through the CPU. The awakening method can be that when the current second progress tracking variable reaches a preset value, the CPU obtains the accelerator completion signal and makes progress, and the CPU starts to execute the reasoning of the pooling layer. The specific reasoning method includes: The input data, model parameters and second progress tracking variable of the pooling layer are obtained from the non-volatile memory; multiple sub-operations are performed on the input data in sequence, and elements are generated and output at the positions of the corresponding output feature maps; the output element positions of the currently completed sub-operations are determined based on the quotient and remainder of the product of the sizes of the current output feature maps; when each sub-operation is completed, that is, each time the position changes, the second progress tracking variable is updated, and the output feature map of the current sub-operation and the corresponding second progress tracking variable are written into the non-volatile memory; wherein the sub-operation includes a pooling operation at each position.

[0028] Specifically, the input data of the pooling layer can be the output of the convolutional layer. The product of the size of the output feature map is the product of the height H, the width W and the number of channels CH.

[0029] When the processing of input data by the fully connected layer is based on CPU operation, the method includes: obtaining the input data and model parameters of the fully connected layer, and the second progress tracking variable from the non-volatile memory; traversing each output node, performing sub-operations on the input data corresponding to all input nodes one by one, and when the sub-operations are completed, updating the second progress tracking variable, and writing the output result of the sub-operation and the corresponding second progress tracking variable into the non-volatile memory; the sub-operations include weighted sum operations and applying activation functions, or weighted sum operations and setting the sum to zero; wherein, when the value of the weighted sum is greater than zero, the output node applies the activation function, and when the value of the weighted sum is less than zero, the output node is set to zero.

[0030] Among them, the input data of the fully connected layer can be the output of the pooling layer, and the model parameters include weights.

[0031] When the fully connected layer processes input data based on hardware accelerator operations, the method includes: The input data and model parameters of the fully connected layer and the second progress tracking variable in the non-volatile memory are obtained through direct memory access; a sub-operation is performed on each output channel, and after obtaining the sub-operation result of each output channel, the second progress tracking variable is updated, and the sub-operation result and the corresponding second progress tracking variable are written into the non-volatile memory; wherein the sub-operation includes a multiplication and accumulation operation, and the result of the multiplication and accumulation operation is added to the bias value, and an activation function is applied.

[0032] Step S2: If a processing interruption occurs during the processing of input data through multiple sub-operations in each layer, for example, a power interruption occurs during step 1, causing the reasoning to stop. Then after the power is restored, the sub-operation that has been executed when the interruption occurs is determined based on the progress tracking variable in the non-volatile memory, and the input data and model parameters required for the remaining sub-operations are obtained to complete the remaining sub-operations.

[0033] In this embodiment, the sub-operations that have been executed when the interruption occurs are determined based on the progress tracking variables, and only the input data and model parameters required for the unexecuted sub-operations are obtained, which can reduce delays and energy overhead and enable normal operation even under relatively small energy buffer conditions; based on the progress tracking variables, recovery from the correct position can be ensured.

[0034] Example 1 The inference method of the present invention is used to infer the CNN model for image handwritten digit recognition. The parameters of the CNN model are shown in Table 1. Table 2 Dataset parameters

[0035] The evaluation metrics are set for runtime overhead and inference throughput according to the workload and capacitor size. The runtime overhead is measured by the execution time and energy consumption required to complete a full inference under stable continuous power conditions. Energy consumption can be measured using the EnergyTrace tool from Texas Instruments. A pure software deep neural network executed on the CPU and a deep neural network executed on a hardware accelerator are run separately. The inference throughput under intermittent power conditions is measured by the number of successful end-to-end inferences completed for all test methods and configurations within the same experimental cycle.

[0036] For the running overhead, the energy consumption and execution time of running deep neural networks in four cases are compared, including: Implementation 1, CPU-based (i.e., software-level) inference implementation; Implementation 2, CPU-based inference implementation of the pooling layer, and hardware accelerator-based inference implementation of the convolutional layer and fully connected layer; the above two do not handle intermittent computing issues, and as the baseline comparison experiment of this experiment, the state will be lost when the power is interrupted and the entire application will be restarted when the power is restored. Implementation 3, CPU-based inference implementation of intermittent complete deep neural network models; Implementation 4, CPU-based intermittent inference implementation of the pooling layer, and hardware accelerator-based inference implementation of the convolutional layer and fully connected layer.

[0037] Figure 2 and Figure 3 The execution time and energy consumption of four inference implementation methods to complete a single end-to-end inference of the CNN model under continuous power are shown. The first two implementation methods are used as the baseline experimental control group of this study, and the final results are normalized based on implementation 4. It takes 1486ms of execution time and consumes 11.03mJ of energy to complete an inference in the handwritten digit recognition application. It can be seen that the energy consumption is directly related to the inference execution time, so the two show similar trends. Comparing implementation one with implementation two, hardware acceleration provides 2 to 8 times of execution time acceleration and 3 to 10 times of energy savings. Comparing implementation one with implementation three, after considering power outages and adopting the inference method of the present invention, its execution time and energy consumption are on par with pure software implementation, indicating that the intermittent processing does not introduce excessive operating overhead.

[0038] In case of intermittent power supply, there is also a gap in inference throughput at different capacitance levels for all implementations and deep neural network applications. Higher capacitance (i.e. larger energy budget) allows longer execution time in case of intermittent power, thus achieving higher throughput. Larger and more complex deep neural network models require more time and energy to execute, resulting in reduced throughput. The number of executions to complete the full inference within 5 minutes was counted multiple times and averaged.

[0039] In order to examine the performance of the system under different energy conditions, experiments were conducted using energy buffer capacitors of different capacities under different energy budgets, such as Figure 4As shown, it can be seen from the figure that the inference throughput under different capacitances, under a capacitance of 2200 microfarads, the handwritten digit recognition model is executed 24 times using implementation three within a given time, and the number of executions using implementation four is 92 times. The handwritten digit recognition model requires several power cycles to complete one inference. Implementation one and implementation two do not have intermittent support, so when the available energy is not enough to complete a single end-to-end inference within one power cycle, they exhibit zero throughput. In the end, it can be seen that the intermittent deep neural network model inference system of the present invention can still ensure that the inference progress moves forward when a power cycle cannot supply a complete operation. The use of CPU-based inference operations (experiment three) in an environment without a hardware accelerator can also achieve the progress of the inference, and the intermittent deep neural network model (experiment four) inference system that can use a hardware accelerator is more efficient.

[0040] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An intermittent deep neural network inference method for energy harvesting Internet of Things, characterized in that: include: Get the input data, model parameters, and progress tracking variables of each layer of the deep neural network model from non-volatile memory; Process the input data through multiple sub-operations in each layer, update the progress tracking variables when each sub-operation is completed, and write the output of the sub-operation and the corresponding progress tracking variables to non-volatile memory; The deep neural network model includes a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer and the fully connected layer process input data based on CPU operations and / or hardware accelerator operations, and the pooling layer processes input data based on CPU operations. If a processing interruption occurs during the processing of input data through multiple sub-operations in each layer, the sub-operation that has been executed when the interruption occurs is determined according to the progress tracking variable in the non-volatile memory, and the input data and model parameters required for the remaining sub-operations are obtained to complete the remaining sub-operations.

2. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 1 is characterized in that: The progress tracking variables include a first progress tracking variable and a second progress tracking variable.

3. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 2 is characterized in that: When the convolution layer processes the input data based on CPU operation, the method includes: Obtain input data and model parameters of the convolutional layer, as well as a first progress tracking variable and a second progress tracking variable from the non-volatile memory; Extract features through multiple sub-operations in sequence, update the first progress tracking variable when each sub-operation is completed, and write the output of the sub-operation and the corresponding first progress variable into the non-volatile memory; When all sub-operations are completed, an output feature map is obtained, and the second progress tracking variable is updated and written into the non-volatile memory.

4. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 2 is characterized in that: When the convolution layer processes the input data based on hardware accelerator operation, the method includes: Obtain input data and model parameters of the convolutional layer in the non-volatile memory through direct memory access, as well as the first progress tracking variable and the second progress tracking variable; Extract features through multiple sub-operations in sequence, update the first progress variable after each sub-operation is completed, and write the sub-operation results and the corresponding first progress variable into the non-volatile memory; When all sub-operations are completed, an output feature map is obtained, the second progress tracking variable is updated, and the output feature map and the updated second progress tracking variable are written into the non-volatile memory through direct memory access.

5. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 3 or 4, characterized in that: The sub-operation includes a multiplication operation and a corresponding accumulation operation.

6. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 3 or 4, characterized in that: When the processing of the input data by the pooling layer is based on CPU operation, the method includes: Get the input data of the pooling layer, the model parameters and the second progress tracking variable from the non-volatile memory; Perform multiple sub-operations on the input data in sequence, generate an element at each position of the output feature map and output it; take the quotient and remainder of the product of the size of the current output feature map to determine the position of the currently completed sub-operation; When each sub-operation is completed, the second progress tracking variable is updated, and the output feature map of the current sub-operation and the corresponding second progress tracking variable are written into the non-volatile memory; The sub-operation includes a pooling operation at each position.

7. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 6 is characterized in that: When the fully connected layer processes the input data based on CPU operation, the method includes: Get the input data and model parameters of the fully connected layer and the second progress tracking variable from the non-volatile memory; Traversing each output node, performing sub-operations on the input data corresponding to all input nodes one by one, updating the second progress tracking variable when the sub-operation is completed, and writing the output result of the sub-operation and the corresponding second progress tracking variable into the non-volatile memory; The sub-operation includes weighted sum operation and application of activation function, or weighted sum operation and setting to zero; wherein, when the value of the weighted sum is greater than zero, the output node applies the activation function, and when the value of the weighted sum is less than zero, the output node is set to zero.

8. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 6 is characterized in that: When the fully connected layer processes the input data based on hardware accelerator operations, the method includes: Obtain input data and model parameters of the fully connected layer and the second progress tracking variable in the non-volatile memory through direct memory access; Performing a sub-operation on each output channel, after obtaining the sub-operation result of each output channel, updating the second progress tracking variable, and writing the sub-operation result and the corresponding second progress tracking variable into the non-volatile memory; The sub-operation includes a multiply-accumulate operation, and the result of the multiply-accumulate operation is added to the bias value, and an activation function is applied.

9. The intermittent deep neural network reasoning method for energy harvesting Internet of Things according to claim 1 is characterized in that: When the reasoning process of the deep neural network is executed based on CPU operation, the input data, model parameters, and progress tracking variables are all transferred from the non-volatile memory to the corresponding layer of the deep neural network model through pointers.

Images