Embedded RTOS-oriented memory dynamic adaptive DNN reasoning method and system

Abstract

The invention provides an embedded RTOS-oriented memory dynamic adaptive DNN reasoning method and system, and the method comprises the steps: analyzing a computational graph structure of a model, recognizing a specific operator sequence, and fusing the specific operator sequence into a new operator in an offline stage; calculating parameters required by reasoning of each operator in advance; correcting all tensors in the model, and removing redundant tensors; allocating a memory for each tensor according to the execution sequence of each operator, and calculating the offset of each tensor relative to the initial position of the memory block; selecting a kernel by each operator and then generating a model code; in the online stage, according to the generated model code, based on a line part loading mechanism, an intermediate memory is distributed for the intermediate tensor generated in the calculation process of each operator, the offset of each intermediate tensor relative to the initial position of an intermediate memory block is calculated according to the size of the available memory by using a lightweight algorithm, and model reasoning is realized. According to the method, the memory occupation size of the deep neural network reasoning task can be automatically adjusted under the condition of extremely limited and dynamically changed memory resources.

Description

Technical Field

[0001] The present invention relates to the field of neural network reasoning technology, and in particular to a memory dynamic adaptive DNN reasoning method and system for embedded RTOS. Background Art

[0002] With the widespread application of Internet of Things (IoT) devices in smart homes, industrial automation, health monitoring and other fields, the demand for complex intelligent functions in embedded devices is increasing. These functions rely on the reasoning ability of deep neural networks (DNN). However, since embedded devices mostly rely on microcontroller units (MCUs) with extremely limited resources, their memory usually uses only tens to hundreds of KB of SRAM. This resource-constrained hardware environment makes it a huge challenge to efficiently deploy DNNs on embedded devices.

[0003] Current DNN inference frameworks, such as Tensorflow Lite Micro and TinyEngine, use a layer-by-layer loading inference method. However, this method has significant memory management problems on MCU devices. First, layer-by-layer inference requires that the input and output tensors must be kept in memory at the same time before each layer is processed, which causes the memory peak to rise rapidly with the increase of the number of network layers. Especially in models with complex network structures or large input data, the memory peak may even exceed the physical memory limit of the device, directly leading to system crashes or task failures. Secondly, since the memory requirements of different DNN layers are often highly unbalanced, the intermediate tensors of some layers are extremely large and occupy most of the memory resources, while the memory requirements of other layers are small. This unbalanced memory allocation further exacerbates the memory bottleneck problem and makes it complicated to optimize inference performance. In addition, the layer-by-layer inference method lacks the ability to adapt to the dynamic changes of the memory of the embedded real-time operating system (RTOS). In an RTOS environment, memory resources are usually shared, and the available memory may decrease or fluctuate at any time as other tasks in the system run. This means that if there is insufficient memory resources at a certain point in the inference process, the DNN inference task may be forced to be interrupted, resulting in excessive delay or even failure of the inference. In addition, since the execution of layer-by-layer inference is a fixed memory allocation method, the system cannot be flexibly adjusted according to the real-time memory status, resulting in waste of memory resources or frequent interruptions of the inference process. These problems not only affect the stability of the inference task, but also make it difficult to meet the real-time requirements under RTOS.

[0004] The high memory peak, uneven memory distribution between layers, and insufficient dynamic memory management capabilities brought about by this layer-by-layer reasoning method seriously limit the application of deep neural networks on resource-constrained embedded devices, especially in an RTOS environment that needs to handle multiple real-time tasks. Strict task deadlines will further amplify these problems. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a memory dynamic adaptive DNN reasoning method and system for embedded RTOS to eliminate or improve one or more defects in the prior art.

[0006] On the one hand, the present invention provides a memory dynamic adaptive DNN reasoning method for embedded RTOS, the method comprising an offline compilation method and an online scheduling method, the method comprising the following steps:

[0007] In the offline compilation method, the computational graph structure of the model is parsed, a specific operator sequence in the model is identified, and the operator sequence is merged into a new operator; the parameters required for reasoning of each operator in the model are pre-calculated; all tensors in the model are corrected, and redundant tensors are removed; according to the execution order of each operator in the model, the life cycle and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated; a kernel is selected for each operator in the model; and a model code is generated;

[0008] In the online scheduling method, according to the model code generated by the offline compilation method, based on the row partial loading mechanism, intermediate memory is allocated for the intermediate tensors generated in the calculation process of each operator, and a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size; thereby realizing the reasoning of the model.

[0009] In some embodiments of the present invention, the computational graph structure of the parsing model includes:

[0010] Divide the input and output tensors of the operators of the model into finer-grained computing units by rows; control the number of output rows of the first operator before fusion to reduce the memory usage of subsequent operators; and reserve intermediate memory for intermediate tensors.

[0011] In some embodiments of the present invention, parameters required for reasoning of each operator in the model are pre-calculated, the parameters including a filling size parameter and a step size parameter. For the filling size parameter, the method further includes:

[0012] In the offline compilation method, the padding size parameter is pre-processed to calculate the padding value required for reasoning;

[0013] If there is a padding operator before the convolution operator, the padding operation is integrated into the padding size parameter of the convolution operator; if there is no padding operator, the corresponding padding value is dynamically calculated according to the input and output shapes of the convolution operator and the padding size parameter type.

[0014] In some embodiments of the present invention, removing redundant tensors in the computation graph includes:

[0015] Remove the intermediate tensors of the new operator obtained by fusion, retain only the input tensor of the first operator before fusion and the output tensor of the last operator, and remove the padding operator.

[0016] In some embodiments of the present invention, the row partial loading mechanism includes:

[0017] During the calculation process of each operator, according to the selected kernel size and step size, each calculation needs to calculate the output of the corresponding number of rows. When there are repeated rows in this calculation and the previous calculation, the output of the repeated rows will be copied and only the output of the new rows will be calculated.

[0018] In some embodiments of the present invention, a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size, including:

[0019] A set of intermediate parameters is set to control the execution flow of the new operator obtained by the convolution operator, the deep convolution operator and the fusion of the convolution operator, including pre-calculating the number of high-dimensional padding rows of the deep convolution operator, initializing the number of padding rows of the deep convolution operator input tensor to 0, setting the number of loop layers to the number of output rows of the deep convolution operator, cyclically calculating the values ​​of the intermediate parameters, the first convolution operator, the deep convolution operator, and the second convolution operator, and copying the rows of the deep convolution operator input tensor that need to be reused to complete the calculation of the fusion operator.

[0020] In some embodiments of the present invention, according to the execution order of each operator in the model, the life cycle and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated, including:

[0021] Calculate the maximum memory requirement during model inference; set the memory layout offline based on the maximum memory requirement; and allocate memory offsets to each memory using a greedy strategy sorted by memory block size.

[0022] In some embodiments of the present invention, in the online scheduling method, adjusting the number of rows loaded for each calculation according to the available memory includes:

[0023] Multiply the number of rows of the intermediate tensor by the corresponding width and number of channels to obtain the temporarily required intermediate memory;

[0024] Subtract the total memory from the intermediate memory to obtain the available memory;

[0025] The available memory is divided by the intermediate memory to obtain the number of rows required for this computation load.

[0026] On the other hand, the present invention also provides a memory dynamic adaptive DNN reasoning system for embedded RTOS, the system comprising:

[0027] An offline compilation module; the offline compilation module includes a computational graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module, and a code generation module. When the offline compilation module is executed, the steps of the offline compilation method in the memory dynamic adaptive DNN reasoning method for embedded RTOS as mentioned in any one of the above items can be implemented;

[0028] An online scheduling module; the online scheduling module includes an online memory management module, and when the online scheduling module is executed, it can implement the steps of the online scheduling method in the memory dynamic adaptive DNN reasoning method for embedded RTOS as mentioned in any one of the items mentioned above.

[0029] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any of the methods mentioned above when the computer program / instruction is executed by a processor.

[0030] The present invention provides a memory dynamic adaptive DNN reasoning method and system for embedded RTOS, including: in the offline stage, parsing the calculation graph structure of the model, identifying a specific operator sequence and fusing it into a new operator; pre-calculating the parameters required for reasoning of each operator; correcting all tensors in the model and removing redundant tensors; allocating memory to each tensor according to the execution order of each operator, and calculating the offset of each tensor relative to the starting position of the memory block; generating model code after each operator selects a kernel; in the online stage, according to the generated model code, based on the row partial loading mechanism, allocating intermediate memory to the intermediate tensors generated in the calculation process of each operator, using a lightweight algorithm, calculating the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size, and realizing model reasoning. The present invention can automatically adjust the memory usage size of the deep neural network reasoning task under extremely limited and dynamically changing memory resource conditions.

[0031] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0032] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0034] Figure 1 It is a framework diagram of a memory dynamic adaptive DNN reasoning method for embedded RTOS in one embodiment of the present invention.

[0035] Figure 2 This is a comparison diagram of the inline partial loading strategy and layer-by-layer reasoning execution according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of a partial loading strategy in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0038] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0040] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0041] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0042] In order to solve the problems of high memory peak, uneven memory distribution between layers and insufficient dynamic memory management capabilities in the existing deep neural network reasoning methods, the present invention provides a memory dynamic adaptive DNN reasoning method for embedded RTOS, which mainly utilizes the characteristics of the residual block structure of the model subgraph and automatically adjusts the number of rows of large intermediate tensors according to memory resource constraints to achieve adaptive deep neural network reasoning. The method includes two parts: offline compilation and online scheduling, such as Figure 1 As shown, the method includes the following steps S101-S102:

[0043] Step S101: In the offline compilation method, the computational graph structure of the model is parsed, a specific operator sequence in the model is identified, and it is merged into a new operator; the parameters required for reasoning of each operator in the model are pre-calculated; all tensors in the model are corrected, and redundant tensors are removed; according to the execution order of each operator in the model, the life cycle and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated; a kernel is selected for each operator in the model; and finally the model code is generated.

[0044] Step S102: In the online scheduling method, according to the model code generated by the offline compilation method, based on the row partial loading mechanism, intermediate memory is allocated for the intermediate tensors generated in the calculation process of each operator, and a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size; thus, the reasoning of the model is realized.

[0045] In step S101, the offline part is first described.

[0046] In the offline compilation method, the computational graph structure of the model is first parsed, and the specific operator sequences in the model are identified and fused into new operators.

[0047] Among them, the computation graph is a graphical structure used to represent the computation process, which is usually used to represent the computational flow of the model in the fields of machine learning and deep learning. The computation graph describes the operations and data flows in the model through nodes and edges.

[0048] In some embodiments, in order to reduce the memory bottleneck of the operator stage such as the depth wise 2D Convolution (DWConv2D), the memory size of the input tensor and output tensor of the operator is reduced for optimization. Taking MobileNetV2 as an example, the data of the TFLite model is usually arranged in the order of NHWC (batch size-height-width-channel). This format helps to improve the efficiency of convolution calculations on resource-constrained embedded devices (such as MCUs). Specifically: the input tensor and output tensor of the corresponding operator are divided into finer-grained computing units by row, and the number of output rows of the first operator before fusion is controlled, which can effectively reduce the memory usage of subsequent operators (such as DWConv2D). At the same time, a smaller intermediate memory is set for the intermediate tensor in advance in the offline stage to ensure that the corresponding operator is successfully completed with less memory usage. In the calculation graph reconstruction part, the intermediate tensor and the corresponding intermediate memory are marked to be written into the model code, and the size and position are finally controlled by the online scheduling method.

[0049] Identify specific operator sequences in the model and replace them with a new fused operator. Taking the residual block structure as an example, in order to optimize three typical continuous operators, such as two-dimensional convolution (Conv2D), deep two-dimensional convolution (DWConv2D), and two-dimensional convolution (Conv2D), they can be fused into a CDC (Conv-DWConv-Conv) operator. By parsing the model, the parameters of these three atoms are marked as CDC-Conv1, CDC-DW, and CDC-Conv2, respectively, and all parameters required for inference are pre-calculated in the offline stage. Among them, the residual block structure is a very common structure in target recognition and target detection tasks. The residual block structure will cause extremely high peak memory at DWConv2D, so it will become the memory bottleneck layer of the entire DNN inference task.

[0050] In some embodiments, the parameters required for inference, such as padding and stride, are pre-calculated, and the padding parameter can be used to significantly reduce resource overhead during inference.

[0051] In the present invention, a special parameter fusion mechanism is designed to reduce the redundant calculation of the padding operator. Considering that the padding size parameter is an inherent property of the convolution operator, the padding length needs to be dynamically calculated during layer-by-layer reasoning. Therefore, in order to improve efficiency, these padding size parameters are preprocessed in the offline stage, and all required padding values ​​are calculated in advance, thereby reducing the computational overhead during reasoning.

[0052] Specifically, if there is a padding operator before the convolution operator (such as Conv2D), the padding operation will be directly integrated into the padding size parameter of the convolution operator to avoid repeated calculations; if there is no padding operator, the corresponding padding value will be dynamically calculated based on the input and output shapes and padding size parameter type of the convolution operator to ensure that the efficiency of the inference process is maximized.

[0053] Correct all tensors in the model to ensure that the model structure is consistent with the optimized operator. Specifically, process the input tensors and output tensors generated after fusion, as well as the intermediate memory required during the inference process, such as the temporary memory that the Im2Col technology may use. Among them, Im2Col (Image to Column) is a common operation, especially in convolutional neural networks. It converts image data from the spatial domain to the form of column vectors. It is an optimized way of convolution operation, which can efficiently implement convolution operation through matrix multiplication.

[0054] To further optimize the memory usage of inference, remove the tensors that are no longer needed in the model calculation graph, especially the redundant intermediate tensors generated after the operator fusion and the output tensors of the padding operator. After removing the intermediate tensors, the fused operator only retains the input tensor of the first operator and the output tensor of the last operator. Removing the above redundant tensors can significantly simplify the subsequent memory planning and ensure that it does not negatively affect the performance of the memory planner.

[0055] In some embodiments, in order to support the newly generated fused operator, necessary tensors need to be created. Taking the above Conv2D, DWConv2D and Conv2D fused into CDC as an example, some additional small input and output tensors may be introduced after fusion.

[0056] Based on the above tensor correction processing, memory utilization efficiency can be maximized to ensure the smoothness and stability of the reasoning process. Especially on embedded devices with limited memory resources, this fine tensor adjustment strategy is particularly important, which can not only significantly reduce the memory usage during the reasoning process, but also effectively improve the overall performance.

[0057] In the offline stage, its memory management is mainly responsible for managing the memory of the input tensors and output tensors of each operator. The memory size is fixed. Therefore, their offsets in the memory can be calculated in advance in the offline stage to ensure that the memory layout remains unchanged during inference, thereby minimizing memory usage.

[0058] In some embodiments, the maximum memory usage during model inference depends on the sum of all memories that need to be stored in the memory at the same time. Therefore, it is only necessary to ensure that the memory layout obtained by offline calculation is the same as the maximum value.

[0059] Specifically, the maximum memory requirement for model inference includes all intermediate results, activation values, weights, and other memory requirements. Based on the maximum memory requirement, the memory layout is set offline. The memory layout refers to allocating each memory block to a location in the memory according to a certain rule. A greedy strategy that sorts by memory block size is used to allocate memory offsets to each memory. For example, larger memory blocks are allocated to the front or better position of the memory. Memory areas that are usually chosen are those that avoid fragmentation or have less competition, thereby ensuring that the memory addresses of each memory block do not overlap and that memory space is used efficiently.

[0060] To achieve optimal inference performance, the best kernel implementation is selected for different operators.

[0061] In some embodiments, still taking the above-mentioned residual block as an example, the Conv2D convolution kernels at both ends select 1x1, and the middle DWConv2D convolution kernel selects 3x3. The 1x1 convolution kernel can well ensure the inter-row independence of the input and output tensors. Under the condition of minimizing memory usage, the output result is controlled by setting the height of the second Conv2D convolution kernel to 1, which corresponds to a row of output of DWConv2D. Since the height of the convolution kernel of DWConv2D is 3, one row of input of the last Conv2D needs to correspond to three rows of input of DWConv2D, that is, three rows of output of the first convolution. In this way, each time the first Conv2D calculates three rows of output results, it is passed to DWConv2D for processing, generates a row of output, and then passed to the second Conv2D for calculation, and finally generates a row of output. Then return to the first convolution to continue calculating the reasoning results of subsequent rows. In this case, the input tensor memory of DWConv2D only needs to retain three rows, and the output tensor memory only needs to retain one row.

[0062] After the above processing and settings, compile and generate the final model code.

[0063] In step S102, online scheduling is implemented according to the model code generated in the offline stage, and the additional intermediate memory required by each operator during the calculation process is uniformly allocated. A lightweight algorithm is used to automatically calculate the offset of each intermediate tensor relative to the starting position of the memory block according to the size of the available memory to ensure full utilization of the memory.

[0064] In some embodiments, the partial row loading strategy includes: in the calculation process of each operator, according to the selected kernel size and step size, each calculation needs to calculate the output of the corresponding number of rows. When there are repeated row calculations between this calculation and the previous calculation, the output of the repeated rows is copied and only the output of the new rows is calculated.

[0065] like Figure 2 and Figure 3The row partial loading mechanism can save two large tensors (such as Figure 2 The tensors B and C on the above are converted into two intermediate memory blocks. Through this optimization, when the height of the tensor is large, the memory usage is significantly reduced and the optimization effect is more obvious.

[0066] Specifically, referring to the example of kernel selection above, the convolution kernel size of DWConv2D is 3x3, the stride is 1, and DWConv2D needs to calculate the output of {0,1,2} rows in the first calculation, while the second calculation requires the output of {1,2,3} rows, where {1,2} rows have been calculated. To avoid repeated calculations, the latter two rows are directly copied to the first two rows, and only one new row needs to be calculated and filled into the third row of the DWConv2D input tensor.

[0067] When considering the inference performance of the fusion operator, one is the problem of repeated calculations mentioned above, and the other is the padding size parameter and the step size parameter.

[0068] Still taking the CDC fusion operator above as an example, for DWConv2D, if the padding of the high dimension is not 0, it is necessary to pad several rows at the top of the input tensor. However, in the actual kernel implementation, these padding operations can be skipped directly to improve the performance of the operator. Under the row partial loading strategy, the padding of each batch of input tensors of DWConv2D may be different. For example, if the stride of the three operators is 1, the second DWConv2D needs to calculate its padding value based on the relationship between the input and output. This will cause the first convolution to generate two rows of output during the first calculation, but no output needs to be generated during the last calculation. In this case, the first batch of data of DWConv2D needs to be padded with a row of 0s before the first row, and the last batch of data needs to be padded with a row of 0s after the last row, or the height of the input tensor is set to 2 when processing the last row.

[0069] In order to solve the above parameter problem and reduce the introduced overhead, the present invention sets a set of intermediate parameters to control the CDC fusion operator execution flow to achieve optimal performance.

[0070] The calculation formulas of the intermediate parameters are shown in formulas (1) to (8):

[0071] initialization:

[0072] load_line1=dw_kernel_h+dw_stride_h×(load_line0-1); (1)

[0073] load_line1=MIN(conv1_in_h+dw_pad_h,load_line1); (2)

[0074] left_rows_in=conv1_in_h; (3)

[0075] left_rows_out=conv1_out_h; (4)

[0076] dw_reuse=MAX(0,dw_kernel_h-dw_stride_h); (5)

[0077] Inside the loop:

[0078]

[0079] row=min(row,left_rows); (7)

[0080]

[0081] Among them, load_line0 represents the number of rows of the DWConv2D input tensor; load_line1 represents the number of rows of the DWConv2D output tensor; dw_kernel_h represents the height of the DWConv2D convolution kernel; dw_stride_h represents the calculation step of DWConv2D; dw_pad_h represents the number of padding of DWConv2D in high dimensions; conv1_in_h represents the number of rows of the input tensor of the first Conv2D; conv1_out_h represents the number of rows of the output tensor of the first Conv2D; left_rows_in represents the number of uncalculated rows of the input tensor of the first Conv2D; left_rows_out represents the number of uncalculated rows of the output tensor of the first Conv2D; dw_reuse represents the number of rows of the input tensor of DWConv2D that need to be reused; row represents the number of rows of the input tensor of the first convolution kernel; strat represents the starting position of the output tensor of the first convolution kernel in the DWConv2D input tensor; i represents the number of loops.

[0082] The C++ conditional operator can optimize the performance of the above-mentioned lightweight parameter calculation method and reduce its impact on the performance of the operator kernel.

[0083] For the minimum memory usage, pre-calculate the number of high-dimensional padding rows of DWConv2D. Assuming the number of padding rows is n, initialize the first n rows of the DWConv2D input tensor to 0, and set the number of loop layers to the number of output rows of DWConv2D. Then, you can loop through the calculations of the intermediate parameters, the first Conv2D, DWConv2D, and the second Conv2D, and copy the rows of the DWConv2D input tensor that need to be reused to complete the calculation of the fusion operator.

[0084] It should be noted that the above formula is universal and is also applicable to the case where the convolution kernel height of DWConv2D is 5, 7, etc. The data flow and principle are exactly the same as the case where the convolution kernel height is 3.

[0085] Based on the above description, it can be seen that at the current stage, its memory management is mainly responsible for dynamically adjusting the size of the intermediate memory according to the current available memory during the operator calculation process, and selecting the appropriate memory space for allocation and storage to optimize the operator's calculation strategy and memory utilization.

[0086] In some embodiments, in the online scheduling method, the number of rows loaded for each calculation is adjusted according to the available memory. For a residual block, in the above formulas (1) and (2), it can be seen that load_line0 represents the number of rows of the DWConv2D input tensor, and load_line1 represents the number of rows of the DWConv2D output tensor. The relationship between these two parameters comes from the fixed formula of the convolution operator, as shown in formula (9):

[0087] input_height=filter_height+(output_height-1)×stride_height-

[0088] 2×padding_height; (9)

[0089] Among them, input_height represents the number of rows of the DWConv2D input tensor, that is, load_line0; output_height represents the number of rows of the DWConv2D output tensor, that is, load_line1; filter_height represents the height of the convolution kernel; stride_height represents the calculation step size in high dimensions; padding_height represents the amount of zero padding added to the upper and lower edges of the input.

[0090] Multiply the number of rows of the intermediate tensor (i.e., DWConv2D input and output tensors) by the corresponding width and number of channels to get the temporarily required intermediate memory. Because the input and output tensors are placed at the front of the memory block, the total memory minus the temporarily required intermediate memory is subtracted to get the available memory, that is, the remaining memory for row dynamic loading. The available memory is divided by the temporarily required intermediate memory to get the number of rows required for this calculation load.

[0091] Corresponding to the memory dynamic adaptive DNN reasoning method for embedded RTOS, the present invention also provides a memory dynamic adaptive DNN reasoning system for embedded RTOS, such as Figure 1As shown in the figure, the overall framework is denoted as StNN, which includes an offline compilation module and an online scheduling module. Specifically:

[0092] The offline compilation module includes a calculation graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module and a code generation module. When the offline compilation module is executed, the steps of the offline compilation method in the memory dynamic adaptive DNN reasoning method for embedded RTOS can be implemented.

[0093] The online scheduling module further includes an online memory management module, which, when executed, can implement the steps of the online scheduling method in the memory dynamic adaptive DNN reasoning method for embedded RTOS.

[0094] A specific embodiment is given below to further illustrate the present invention.

[0095] Hardware preparation: The hardware required for the implementation of this algorithm is the STM32H743VI development board, the hardware is Cortex-M7, and the RTOS used is the open source real-time operating system zephyr.

[0096] Algorithm implementation: Use Python and C++ languages ​​to implement a memory dynamic adaptive DNN reasoning method for embedded RTOS. According to the steps of this method, use Python offline processing model to generate C++ source file, add the module of statistical delay, compile and burn it to STM32H743VI development board through serial port, then input corresponding instructions through serial port to start reasoning model, and finally get the end-to-end reasoning delay corresponding to the model.

[0097] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device implements the steps of the method described above.

[0098] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0099] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0100] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0101] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A memory dynamic adaptive DNN reasoning method for embedded RTOS, characterized in that: The method comprises an offline compilation method and an online scheduling method, and the method comprises the following steps: In the offline compilation method, the computational graph structure of the model is parsed, a specific operator sequence in the model is identified, and the operator sequence is merged into a new operator; the parameters required for reasoning of each operator in the model are pre-calculated; all tensors in the model are corrected, and redundant tensors are removed; according to the execution order of each operator in the model, the life cycle and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated; a kernel is selected for each operator in the model; and a model code is generated; In the online scheduling method, according to the model code generated by the offline compilation method, based on the row partial loading mechanism, intermediate memory is allocated for the intermediate tensors generated in the calculation process of each operator, and a lightweight algorithm is used to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block according to the available memory size; thereby realizing the reasoning of the model.

2. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: The computational graph structure of the parsing model includes: Divide the input and output tensors of the operators of the model into finer-grained computing units by rows; control the number of output rows of the first operator before fusion to reduce the memory usage of subsequent operators; and reserve intermediate memory for intermediate tensors.

3. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: Precalculating parameters required for reasoning of each operator in the model, the parameters including a filling size parameter and a step size parameter. For the filling size parameter, the method further includes: In the offline compilation method, the padding size parameter is pre-processed to calculate the padding value required for reasoning; If there is a padding operator before the convolution operator, the padding operation is integrated into the padding size parameter of the convolution operator; if there is no padding operator, the corresponding padding value is dynamically calculated according to the input and output shapes of the convolution operator and the padding size parameter type.

4. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: Removing redundant tensors in the computation graph includes: Remove the intermediate tensors of the new operator obtained by fusion, retain only the input tensor of the first operator before fusion and the output tensor of the last operator, and remove the padding operator.

5. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: The row partial loading mechanism includes: During the calculation process of each operator, according to the selected kernel size and step size, each calculation needs to calculate the output of the corresponding number of rows. When there are repeated rows in this calculation and the previous calculation, the output of the repeated rows will be copied and only the output of the new rows will be calculated.

6. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 5, characterized in that: Use a lightweight algorithm to calculate the offset of each intermediate tensor relative to the starting position of the intermediate memory block based on the available memory size, including: A set of intermediate parameters is set to control the execution flow of the new operator obtained by the convolution operator, the deep convolution operator and the fusion of the convolution operator, including pre-calculating the number of high-dimensional padding rows of the deep convolution operator, initializing the number of padding rows of the deep convolution operator input tensor to 0, setting the number of loop layers to the number of output rows of the deep convolution operator, cyclically calculating the values ​​of the intermediate parameters, the first convolution operator, the deep convolution operator, and the second convolution operator, and copying the rows of the deep convolution operator input tensor that need to be reused to complete the calculation of the fusion operator.

7. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: According to the execution order of each operator in the model, the life cycle and memory are planned for each tensor, and the offset of each tensor relative to the starting position of the memory block is calculated, including: Calculate the maximum memory requirement during model inference; set the memory layout offline based on the maximum memory requirement; and allocate memory offsets to each memory using a greedy strategy sorted by memory block size.

8. The memory dynamic adaptive DNN reasoning method for embedded RTOS according to claim 1, characterized in that: In the online scheduling method, adjusting the number of rows loaded in each calculation according to the available memory includes: Multiply the number of rows of the intermediate tensor by the corresponding width and number of channels to obtain the temporarily required intermediate memory; Subtract the total memory from the intermediate memory to obtain the available memory; The available memory is divided by the intermediate memory to obtain the number of rows required for this computation load.

9. A memory dynamic adaptive DNN reasoning system for embedded RTOS, characterized in that: The system comprises: An offline compilation module; the offline compilation module includes a computational graph reconstruction module, an operator fusion module, a tensor correction module, an offline memory management module, a kernel selection module, and a code generation module. When the offline compilation module is executed, the steps of the offline compilation method in the memory dynamic adaptive DNN reasoning method for embedded RTOS as claimed in any one of claims 1 to 8 can be implemented; An online scheduling module; the online scheduling module includes an online memory management module, and when the online scheduling module is executed, it can implement the steps of the online scheduling method in the memory dynamic adaptive DNN reasoning method for embedded RTOS as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images