A memory-based dynamic adaptive DNN inference method and system for embedded RTOS

Abstract

This invention provides a memory-dynamic adaptive DNN inference method and system for embedded RTOS, comprising: in an offline phase, parsing the computation graph structure of the model, identifying specific operator sequences and fusing them into new operators; pre-calculating the parameters required for inference of each operator; correcting all tensors in the model and removing redundant tensors; allocating memory for each tensor according to the execution order of each operator, and calculating the offset of each tensor relative to the starting position of the memory block; generating model code after each operator selects a kernel; in an online phase, based on the generated model code and a line-partial loading mechanism, allocating intermediate memory for intermediate tensors generated during the computation of each operator, and using a lightweight algorithm to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block based on the available memory size, thereby realizing model inference. This invention can automatically adjust the memory usage of deep neural network inference tasks under extremely limited and dynamically changing memory resource conditions.

Description

Technical Field

[0001] This invention relates to the field of neural network inference technology, and in particular to a memory-dynamic adaptive DNN inference method and system for embedded RTOS. Background Technology

[0002] With the widespread application of Internet of Things (IoT) devices in smart homes, industrial automation, and health monitoring, the demand for complex intelligent functions in embedded devices is constantly increasing. These functions rely on the inference capabilities of deep neural networks (DNNs). However, because embedded devices often depend on microcontroller units (MCUs) with extremely limited resources, their memory typically consists of only tens to hundreds of KB of SRAM. This resource-constrained hardware environment makes the efficient deployment of DNNs on embedded devices a significant challenge.

[0003] Current DNN inference frameworks, such as Tensorflow Lite Micro and TinyEngine, employ a layer-by-layer loading inference approach. However, this approach presents significant memory management challenges on MCU devices. First, layer-by-layer inference requires that the input and output tensors be simultaneously stored in memory before each layer finishes processing. This causes memory spikes to rise rapidly with the number of network layers, especially in models with complex network structures or large input data. In such cases, memory spikes may even exceed the device's physical memory limits, directly leading to system crashes or task failures. Second, the memory requirements of different DNN layers are often highly uneven. Some layers have extremely large intermediate tensors, consuming most of the memory resources, while other layers have smaller memory requirements. This uneven memory allocation further exacerbates the memory bottleneck problem, making inference performance optimization complex. Furthermore, layer-by-layer inference lacks adaptability to the dynamic changes in memory usage within embedded real-time operating systems (RTOS). In RTOS environments, memory resources are typically shared, and available memory can decrease or fluctuate at any time as other tasks run in the system. This means that if memory resources are insufficient at any point during the inference process, the DNN inference task may be forced to interrupt, leading to excessive inference latency or even failure. Furthermore, because layer-by-layer inference uses a fixed memory allocation method, the system cannot flexibly adjust according to real-time memory conditions, resulting in wasted memory resources or frequent interruptions in the inference process. These issues not only affect the stability of the inference task but also make it difficult to meet the real-time requirements of RTOS.

[0004] The high memory peaks, uneven memory distribution between layers, and insufficient dynamic memory management capabilities resulting from this layer-by-layer inference approach severely limit the application of deep neural networks on resource-constrained embedded devices. In particular, in environments like RTOS that require handling multiple real-time tasks, strict task deadlines further amplify these problems. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a memory-dynamic adaptive DNN inference method and system for embedded RTOS, in order to eliminate or improve one or more defects existing in the prior art.

[0006] On one hand, this invention provides a memory-based dynamic adaptive DNN inference method for embedded RTOS, the method including an offline compilation method and an online scheduling method, the method comprising the following steps:

[0007] In the offline compilation method, the computation graph structure of the model is parsed, specific operator sequences in the model are identified and fused into new operators; the parameters required for inference of each operator in the model are pre-calculated; all tensors in the model are corrected and redundant tensors are removed; according to the execution order of each operator in the model, the lifetime and memory of each tensor are planned, and the offset of each tensor relative to the starting position of the memory block is calculated; kernels are selected for each operator in the model; and model code is generated.

[0008] In the online scheduling method, based on the model code generated by the offline compilation method, intermediate memory is allocated for the intermediate tensors generated during the calculation of each operator based on the line-partial loading mechanism, and a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size; thus realizing the inference of the model.

[0009] In some embodiments of the present invention, the computational graph structure of the analytical model includes:

[0010] The input and output tensors of the operators in the model are divided into finer-grained computational units by row; the number of output rows of the first operator before fusion is controlled to reduce the memory usage of subsequent operators; and intermediate memory is reserved for intermediate tensors.

[0011] In some embodiments of the present invention, parameters required for inference of each operator in the model are pre-calculated, including fill size parameters and step size parameters. For the fill size parameters, the method further includes:

[0012] In the offline compilation method, the padding size parameter is preprocessed to calculate the padding value required for inference;

[0013] If a padding operator exists before the convolution operator, the padding operation is incorporated into the padding size parameter of the convolution operator; if the padding operator does not exist, the corresponding padding value is dynamically calculated based on the input and output shapes of the convolution operator and the type of the padding size parameter.

[0014] In some embodiments of the present invention, removing redundant tensors in the computation graph includes:

[0015] Remove the intermediate tensors of the new operator obtained by fusion, and retain only the input tensor of the first operator and the output tensor of the last operator before fusion, and remove the padding operator.

[0016] In some embodiments of the present invention, the row portion loading mechanism includes:

[0017] During the calculation of each operator, depending on the selected kernel size and step size, the corresponding number of output rows need to be calculated for each calculation. When there are duplicate rows in the current calculation and the previous calculation, the output of the duplicate rows is copied, and only the output of the new rows is calculated.

[0018] In some embodiments of the present invention, a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block, based on the available memory size, including:

[0019] A set of intermediate parameters is set to control the execution flow of the new operator obtained by fusing convolution operators, depthwise convolution operators, and convolution operators. This includes pre-calculating the number of high-dimensional padding rows of the depthwise convolution operator, initializing the first padding rows of the input tensor of the depthwise convolution operator to 0, setting the number of loop layers to the number of output rows of the depthwise convolution operator, iteratively calculating the values ​​of the intermediate parameters, the first convolution operator, the depthwise convolution operator, and the second convolution operator, and copying the rows of the input tensor of the depthwise convolution operator that need to be reused to complete the calculation of the fusion operator.

[0020] In some embodiments of the present invention, based on the execution order of the operators in the model, a lifetime and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated, including:

[0021] Calculate the maximum memory requirement for the model during inference; based on the maximum memory requirement, set the memory layout offline; allocate memory offsets to each memory block using a greedy strategy that sorts memory blocks by size.

[0022] In some embodiments of the present invention, in the online scheduling method, adjusting the number of rows loaded in each calculation based on the available memory includes:

[0023] Multiply the number of rows of the intermediate tensor by the corresponding width and number of channels to obtain the temporary intermediate memory required;

[0024] The available memory is obtained by subtracting the total memory from the intermediate memory.

[0025] The number of rows required for this calculation is obtained by dividing the available memory by the intermediate memory.

[0026] On the other hand, the present invention also provides a memory-dynamic adaptive DNN inference system for embedded RTOS, the system comprising:

[0027] The offline compilation module includes a computation graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module, and a code generation module. When the offline compilation module is executed, it can implement the steps of the offline compilation method in the memory-dynamic adaptive DNN inference method for embedded RTOS mentioned above.

[0028] An online scheduling module; the online scheduling module includes an online memory management module, and when the online scheduling module is executed, it can implement the steps of the online scheduling method in any of the above-mentioned memory-based dynamic adaptive DNN inference methods for embedded RTOS.

[0029] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods mentioned above.

[0030] This invention provides a memory-dynamic adaptive DNN inference method and system for embedded RTOS, comprising: in an offline phase, parsing the computation graph structure of the model, identifying specific operator sequences and fusing them into new operators; pre-calculating the parameters required for inference of each operator; correcting all tensors in the model and removing redundant tensors; allocating memory for each tensor according to the execution order of each operator, and calculating the offset of each tensor relative to the starting position of the memory block; generating model code after each operator selects a kernel; in an online phase, based on the generated model code and a line-partial loading mechanism, allocating intermediate memory for intermediate tensors generated during the computation of each operator, and using a lightweight algorithm to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block based on the available memory size, thereby realizing model inference. This invention can automatically adjust the memory usage of deep neural network inference tasks under extremely limited and dynamically changing memory resource conditions.

[0031] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0032] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0034] Figure 1 This is a framework diagram of a memory-dynamic adaptive DNN inference method for embedded RTOS according to an embodiment of the present invention.

[0035] Figure 2 This is a comparison diagram of the loading strategy of the upper part and the execution of layer-by-layer inference in one embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the loading strategy for the upper part of an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0038] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0039] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0040] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0041] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0042] To address the issues of high memory peaks, uneven memory distribution between layers, and insufficient dynamic memory management capabilities in existing deep neural network inference methods, this invention provides a memory-dynamic adaptive DNN inference method for embedded RTOS. This method primarily utilizes the residual block structure of the model subgraph to automatically adjust the number of rows in large intermediate tensors according to memory resource constraints, thereby achieving adaptive deep neural network inference. The method comprises two parts: offline compilation and online scheduling, such as... Figure 1 As shown, the method includes the following steps S101 to S102:

[0043] Step S101: In the offline compilation method, the computation graph structure of the model is parsed, specific operator sequences in the model are identified and fused into new operators; the parameters required for inference of each operator in the model are pre-calculated; all tensors in the model are corrected and redundant tensors are removed; according to the execution order of each operator in the model, the lifetime and memory of each tensor are planned, and the offset of each tensor relative to the starting position of the memory block is calculated; kernels are selected for each operator in the model; and finally, the model code is generated.

[0044] Step S102: In the online scheduling method, based on the model code generated by the offline compilation method, intermediate memory is allocated for the intermediate tensors generated during the calculation of each operator based on the line-partial loading mechanism, and a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block based on the available memory size; thus realizing the inference of the model.

[0045] In step S101, the offline part will be explained first.

[0046] In the offline compilation method, the computation graph structure of the model is first parsed, specific operator sequences in the model are identified, and they are then merged into new operators.

[0047] A computation graph is a graphical structure used to represent computational processes, typically employed in machine learning and deep learning to represent the computational flow of a model. A computation graph describes the operations and data flow within the model using nodes and edges.

[0048] In some embodiments, to reduce memory bottlenecks in operators such as depth-wise 2D convolution (DWConv2D), optimization is achieved by reducing the memory size of the input and output tensors of this operator. Taking MobileNetV2 as an example, the data in the TFLite model is typically arranged in NHWC (batch size-height-width-channels) order, a format that helps improve the efficiency of convolution computation on resource-constrained embedded devices (such as MCUs). Specifically, dividing the input and output tensors of the corresponding operators into finer-grained computational units by row and controlling the number of output rows of the first operator before fusion effectively reduces the memory footprint of subsequent operators (such as DWConv2D). Simultaneously, a smaller intermediate memory is pre-allocated for intermediate tensors during the offline phase to ensure that the corresponding operators complete smoothly with less memory usage. In the computation graph reconstruction part, the intermediate tensors and their corresponding intermediate memory are marked and written into the model code, where their size and position are ultimately controlled by an online scheduling method.

[0049] Identify specific operator sequences in the model and replace them with a new fusion operator. Taking the residual block structure as an example, to optimize three typical consecutive operators, such as 2D convolution (Conv2D), depthwise 2D convolution (DWConv2D), and 2D convolution (Conv2D), they can be fused into the CDC (Conv-DWConv-Conv) operator. By analyzing the model, the parameters of these three atoms are labeled as CDC-Conv1, CDC-DW, and CDC-Conv2, respectively, and all parameters required for inference are pre-computed offline. The residual block structure is a very common structure in object recognition and detection tasks. The residual block structure leads to extremely high peak memory usage at DWConv2D, thus becoming the memory bottleneck layer for the entire DNN inference task.

[0050] In some embodiments, the parameters required for inference, including padding and stride, are pre-calculated, thereby significantly reducing resource overhead during inference.

[0051] In this invention, a dedicated parameter fusion mechanism is designed to reduce redundant computations in the padding operator. Considering that the padding size parameter is an inherent property of the convolution operator, and the padding length needs to be dynamically calculated during layer-by-layer inference, these padding size parameters are preprocessed offline to improve efficiency and pre-calculate all the required padding values, thereby reducing computational overhead during inference.

[0052] Specifically, if a padding operator exists before the convolution operator (such as Conv2D), the padding operation is directly integrated into the padding size parameter of the convolution operator to avoid redundant calculations; if no padding operator exists, the corresponding padding value is dynamically calculated based on the input and output shapes of the convolution operator and the type of padding size parameter to ensure maximum efficiency of the inference process.

[0053] All tensors in the model are corrected to ensure that the model structure remains consistent with the optimized operators. Specifically, this involves processing the fused input tensors, output tensors, and intermediate memory required during inference, such as temporary memory that the Im2Col technique might use. Im2Col (Image to Column) is a common operation, especially in convolutional neural networks, that converts image data from the spatial domain into column vectors. It's an optimized method for convolution operations, and convolution can be efficiently implemented using matrix multiplication.

[0054] To further optimize memory usage for inference, unnecessary tensors in the model computation graph are removed, especially redundant intermediate tensors generated after operator fusion and output tensors used to fill operators. After removing intermediate tensors, the fused operators retain only the input tensor of the first operator and the output tensor of the last operator. Removing these redundant tensors significantly simplifies subsequent memory planning and ensures that it does not negatively impact the performance of the memory planner.

[0055] In some embodiments, necessary tensors need to be created to support the newly generated fusion operator. Taking the fusion of Conv2D, DWConv2D, and Conv2D into CDC as an example, some additional small input and output tensors may be introduced after fusion.

[0056] Based on the tensor correction process described above, memory utilization efficiency can be maximized, ensuring the smoothness and stability of the inference process. This fine-grained tensor adjustment strategy is particularly important on memory-constrained embedded devices, as it not only significantly reduces memory usage during inference but also effectively improves overall performance.

[0057] During the offline phase, its memory management is mainly responsible for managing the memory of the input and output tensors of each operator. The size of these memory is fixed. Therefore, their offsets in memory can be calculated in advance during the offline phase to ensure that the memory layout remains unchanged during inference, thereby minimizing memory usage.

[0058] In some embodiments, the maximum memory usage during model inference depends on the sum of all memory that needs to be stored in memory at a given moment. Therefore, it is sufficient to ensure that the memory layout obtained from offline computation is the same as this maximum value.

[0059] Specifically, the maximum memory requirement during model inference includes the memory needs of all intermediate results, activation values, weights, etc. Based on this maximum memory requirement, a memory layout is set up offline. This memory layout refers to the location of each memory block in memory according to a certain rule. A greedy strategy that sorts memory blocks by size is used to allocate memory offsets to each memory block. For example, larger memory blocks are allocated to the beginning of memory or to better locations. Typically, memory regions that avoid fragmentation or have less contention are chosen, thus ensuring that the memory addresses of each memory block do not overlap and that memory space is used efficiently.

[0060] To achieve optimal inference performance, the best kernel implementation is selected for different operators.

[0061] In some embodiments, taking the residual block mentioned above as an example, the Conv2D convolutional kernels at both ends are chosen as 1x1, and the DWConv2D convolutional kernel in the middle is chosen as 3x3. The 1x1 convolutional kernel can well ensure the inter-row independence of the input and output tensors. To minimize memory usage, the output result is controlled by setting the height of the second Conv2D convolutional kernel to 1, which corresponds to one row of output from DWConv2D. Since the height of the DWConv2D convolutional kernel is 3, one row of input from the last Conv2D needs to correspond to three rows of input from DWConv2D, i.e., three rows of output from the first convolution. In this way, each time the first Conv2D calculates three rows of output results, it is passed to DWConv2D for processing to generate one row of output, and then passed to the second Conv2D for calculation, finally generating one row of output. Then it returns to the first convolution to continue calculating the inference results for subsequent rows. In this case, the DWConv2D input tensor memory only needs to hold three rows, and the output tensor memory only needs to hold one row.

[0062] After the above processing and settings, the final model code is generated through compilation.

[0063] In step S102, online scheduling is implemented based on the model code generated in the offline stage. The additional intermediate memory required by each operator during the calculation process is uniformly allocated. A lightweight algorithm is used to automatically calculate the offset of each intermediate tensor relative to the starting position of the memory block based on the available memory size, so as to ensure full utilization of memory.

[0064] In some embodiments, the row loading strategy includes: during the computation of each operator, depending on the selected kernel size and step size, the output of the corresponding number of rows needs to be calculated for each computation; when there are duplicate row computations between the current computation and the previous computation, the output of the duplicate rows is copied, and only the output of the new rows is calculated.

[0065] like Figure 2 and Figure 3The diagram illustrates a row-partial loading strategy. This mechanism allows for the simultaneous storage of two large tensors (such as...) that would otherwise require simultaneous storage in memory during a layer-by-layer loading process. Figure 2 Tensors B and C are transformed into two intermediate memory blocks. Through this optimization, when the height of the tensor is large, the memory usage is significantly reduced, and the optimization effect is more obvious.

[0066] Specifically, referring to the kernel selection example above, the DWConv2D convolution kernel size is 3x3 with a stride of 1. In the first calculation, DWConv2D needs to compute the output of rows {0,1,2}, while the second calculation requires the output of rows {1,2,3}, where rows {1,2} have already been computed. To avoid redundant computation, the latter two rows are directly copied to the former two rows, requiring only the computation of a new row, which is then inserted into the third row of the DWConv2D input tensor.

[0067] When considering the inference performance of fusion operators, one issue is the aforementioned problem of repeated computation, and another issue is the filling size parameter and step size parameter.

[0068] Continuing with the CDC fusion operator example above, for DWConv2D, if the padding in higher dimensions is not zero, several rows need to be padded at the top of the input tensor. However, in actual kernel implementations, these padding operations can be skipped to improve operator performance. Under the row-partial loading strategy, the padding of each batch of input tensors in DWConv2D may be different. For example, if the stride of all three operators is 1, the second DWConv2D needs to calculate its padding value based on the relationship between the input and output. This will cause the first convolution to generate two rows of output during its first computation, but not during its last computation. In this case, the first batch of data in DWConv2D needs to be padded with a row of 0s before the first row, and the last batch of data needs to be padded with a row of 0s after the last row, or the height of the input tensor can be set to 2 when processing the last row.

[0069] To address the aforementioned parameter issues and reduce introduced overhead, this invention sets a set of intermediate parameters to control the execution flow of the CDC fusion operator, thereby achieving optimal performance.

[0070] The calculation formulas for each intermediate parameter are shown in formulas (1) to (8):

[0071] initialization:

[0072] load_line1=dw_kernel_h+dw_stride_h×(load_line0-1); (1)

[0073] load_line1=MIN(conv1_in_h+dw_pad_h,load_line1); (2)

[0074] left_rows_in=conv1_in_h; (3)

[0075] left_rows_out=conv1_out_h; (4)

[0076] dw_reuse=MAX(0,dw_kernel_h-dw_stride_h); (5)

[0077] Inside the loop:

[0078]

[0079] row=min(row,left_rows); (7)

[0080]

[0081] Where load_line0 represents the number of rows in the DWConv2D input tensor; load_line1 represents the number of rows in the DWConv2D output tensor; dw_kernel_h represents the height of the DWConv2D convolutional kernel; dw_stride_h represents the stride of the DWConv2D computation; dw_pad_h represents the padding in the high dimension of the DWConv2D; conv1_in_h represents the number of rows in the input tensor of the first Conv2D; conv1_out_h represents the number of rows in the output tensor of the first Conv2D; left_rows_in represents the number of uncomputed rows in the input tensor of the first Conv2D; left_rows_out represents the number of uncomputed rows in the output tensor of the first Conv2D; dw_reuse represents the number of rows in the input tensor of the DWConv2D that need to be reused; row represents the number of rows in the input tensor of the first convolutional kernel; strat represents the starting position of the output tensor of the first convolutional kernel in the input tensor of the DWConv2D; and i represents the number of iterations.

[0082] The C++ conditional operator can effectively optimize the performance of this lightweight parameter calculation method, reducing its impact on the performance of the operator kernel.

[0083] For minimum memory usage, the number of high-dimensional padding rows of DWConv2D is pre-calculated. Assuming the number of padding rows is denoted as n, the first n rows of the DWConv2D input tensor are initialized to 0. The number of loop layers is set to the number of output rows of DWConv2D. The values ​​of intermediate parameters, the first Conv2D, DWConv2D, and the second Conv2D are then calculated in a loop. The rows of the DWConv2D input tensor that need to be reused are copied to complete the calculation of the fusion operator.

[0084] It should be noted that the above formula is universal and applies to DWConv2D convolution kernel heights of 5, 7, etc. The data flow and principle are exactly the same as those for the case where the convolution kernel height is 3.

[0085] Based on the above explanation, at this stage, its memory management is mainly responsible for dynamically adjusting the size of intermediate memory according to the currently available memory during the operator calculation process, and selecting appropriate memory space for allocation and storage, so as to optimize the operator's calculation strategy and memory utilization.

[0086] In some embodiments, in the online scheduling method, the number of lines loaded for each computation is adjusted according to the available memory. For a residual block, as can be seen from the above formulas (1) and (2), load_line0 represents the number of lines of the DWConv2D input tensor, and load_line1 represents the number of lines of the DWConv2D output tensor. The relationship between these two parameters comes from the fixed formula of the convolution operator, as shown in formula (9):

[0087] input_height=filter_height+(output_height-1)×stride_height-

[0088] 2×padding_height; (9)

[0089] Wherein, input_height represents the number of rows of the DWConv2D input tensor, i.e., load_line0; output_height represents the number of rows of the DWConv2D output tensor, i.e., load_line1; filter_height represents the height of the convolution kernel; stride_height represents the computation stride in the high dimension; and padding_height represents the amount of zero padding added to the top and bottom edges of the input.

[0090] Multiply the number of rows in the intermediate tensor (i.e., the DWConv2D input and output tensors) by the corresponding width and number of channels to obtain the temporary intermediate memory required. Since the input and output tensors are placed at the beginning of the memory block, subtract the temporary intermediate memory required from the total memory to obtain the available memory, which is the memory remaining for dynamic row loading. Divide the available memory by the temporary intermediate memory required to obtain the number of rows required for this calculation.

[0091] Corresponding to the memory-dynamic adaptive DNN inference method for embedded RTOS, this invention also provides a memory-dynamic adaptive DNN inference system for embedded RTOS, such as... Figure 1As shown, the overall framework is denoted as StNN, which includes an offline compilation module and an online scheduling module. Specifically:

[0092] The offline compilation module includes a computation graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module, and a code generation module. When executed, this offline compilation module implements the steps of the offline compilation method in the memory-driven adaptive DNN inference method for embedded RTOS.

[0093] The online scheduling module includes an online memory management module, which, when executed, implements the steps of the online scheduling method in the memory-driven adaptive DNN inference method for embedded RTOS.

[0094] The present invention will be further illustrated by a specific embodiment described below.

[0095] Hardware preparation: The hardware required for the implementation of this algorithm is an STM32H743VI development board, a Cortex-M7 processor, and the open-source real-time operating system Zephyr.

[0096] Algorithm Implementation: A memory-based dynamic adaptive DNN inference method for embedded RTOS is implemented using Python and C++. Following the steps of this method, the C++ source file is generated by offline processing of the model using Python, a module for statistical latency is added, and after compilation, it is burned to the STM32H743VI development board via serial port. Then, the inference model is started by inputting the corresponding commands via serial port, and finally the end-to-end inference latency of the model is obtained.

[0097] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device performs the steps of the method as described above.

[0098] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0099] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0100] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0101] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A memory-based dynamic adaptive DNN inference method for embedded RTOS, characterized in that, The method includes an offline compilation method and an online scheduling method, and the method includes the following steps: In the offline compilation method, the computation graph structure of the model is parsed, the operator sequence in the model is identified, and they are merged into new operators; the parameters required for inference of each operator in the model are pre-calculated; all tensors in the model are corrected and redundant tensors are removed; according to the execution order of each operator in the model, the lifetime and memory of each tensor are planned, and the offset of each tensor relative to the starting position of the memory block is calculated; kernels are selected for each operator in the model; and model code is generated. In the online scheduling method, based on the model code generated by the offline compilation method, intermediate memory is allocated for intermediate tensors generated during the computation of each operator, using a line-partial loading mechanism. A lightweight algorithm is then used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block, based on the available memory size. This enables the model's inference. The line-partial loading mechanism includes: during the computation of each operator, based on the selected kernel size and step size, the corresponding number of output lines need to be calculated for each computation. When there are duplicate line calculations between the current and previous computations, the output of the duplicate lines is copied, and only the line counted is recorded. The lightweight algorithm includes: setting a set of intermediate parameters to control the execution flow of the new operator obtained by fusing the convolution operator, the depth convolution operator, and the convolution operator, including pre-calculating the number of high-dimensional padding rows of the depth convolution operator, initializing the first padding row of the input tensor of the depth convolution operator to 0, setting the number of loop layers to the number of output rows of the depth convolution operator, iteratively calculating the value of the intermediate parameters, the first convolution operator, the depth convolution operator, and the second convolution operator, and copying the rows that need to be reused in the input tensor of the depth convolution operator to complete the calculation of the fusion operator.

2. The memory-based dynamic adaptive DNN inference method for embedded RTOS according to claim 1, characterized in that, The computational graph structure of the analytical model includes: The input and output tensors of the operators in the model are divided into finer-grained computational units by row; the number of output rows of the first operator before fusion is controlled to reduce the memory usage of subsequent operators; and intermediate memory is reserved for intermediate tensors.

3. The memory-based dynamic adaptive DNN inference method for embedded RTOS according to claim 1, characterized in that, The method pre-calculates the parameters required for inference of each operator in the model, including a fill size parameter and a step size parameter. For the fill size parameter, the method further includes: In the offline compilation method, the padding size parameter is preprocessed to calculate the padding value required for inference; If a padding operator exists before the convolution operator, the padding operation is incorporated into the padding size parameter of the convolution operator; if the padding operator does not exist, the corresponding padding value is dynamically calculated based on the input and output shapes of the convolution operator and the type of the padding size parameter.

4. The memory-based dynamic adaptive DNN inference method for embedded RTOS according to claim 1, characterized in that, Removing redundant tensors from the computation graph includes: Remove the intermediate tensors of the new operator obtained by fusion, and retain only the input tensor of the first operator and the output tensor of the last operator before fusion, and remove the padding operator.

5. The memory-dynamic adaptive DNN inference method for embedded RTOS according to claim 1, characterized in that, Based on the execution order of the operators in the model, a lifetime and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated, including: Calculate the maximum memory requirement for the model during inference; based on the maximum memory requirement, set the memory layout offline; allocate memory offsets to each memory block using a greedy strategy that sorts memory blocks by size.

6. The memory-based dynamic adaptive DNN inference method for embedded RTOS according to claim 1, characterized in that, In the online scheduling method, adjusting the number of rows loaded in each calculation based on the available memory includes: Multiply the number of rows of the intermediate tensor by the corresponding width and number of channels to obtain the temporary intermediate memory required; The available memory is obtained by subtracting the total memory from the intermediate memory. The number of rows required for this calculation is obtained by dividing the available memory by the intermediate memory.

7. A memory-dynamic adaptive DNN inference system for embedded RTOS, characterized in that, The system includes: An offline compilation module; the offline compilation module includes a computation graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module, and a code generation module. When the offline compilation module is executed, it can implement the steps of the offline compilation method in the memory-dynamic adaptive DNN inference method for embedded RTOS as described in any one of claims 1 to 6. An online scheduling module; the online scheduling module includes an online memory management module, and when the online scheduling module is executed, it can implement the steps of the online scheduling method in the memory-based dynamic adaptive DNN inference method for embedded RTOS as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 6.

Images