Efficient image recognition system based on embedded edge device

Abstract

The invention relates to a high-efficiency image recognition system based on embedded edge equipment, which belongs to the technical field of computer application and comprises a hardware acceleration module, an image recognition module and an image recognition module, reasoning tasks are distributed to different cores and a model quantification module through dual-core cooperation, quantification converts high-precision floating-point number representation into low-precision integers through weight and activation in a deep learning model, an automatic deployment module automatically converts the trained deep learning model into a format suitable for an embedded hardware platform, and the embedded hardware platform is used for realizing real-time reasoning. The data processing module is used for preprocessing and enhancing input data and ensuring that the data meet the input requirements of the deep learning model, and the reasoning engine module is responsible for loading and operating the optimized deep learning model on embedded equipment. And the efficient and reliable image recognition function is realized while the calculation, storage and energy consumption limitation of the embedded equipment is met.

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and specifically relates to an efficient image recognition system based on embedded edge devices. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI) and deep learning technology, image recognition systems based on deep neural networks (DNN) have been widely used. These systems have demonstrated strong performance in tasks such as image classification, object detection, and behavior recognition. However, traditional deep learning models often rely on high-performance computing devices such as cloud servers or GPUs to achieve efficient operation, which poses a huge challenge to resource-constrained embedded edge devices.

[0003] Embedded edge devices (such as STM32 series microcontrollers, Raspberry Pi, etc.) have become core components in Internet of Things (IoT) scenarios due to their low power consumption, miniaturization, and deployment flexibility. Especially in the fields of industrial automation, smart home, drone control, etc., embedded devices can complete data processing locally, reduce dependence on the cloud, and significantly reduce latency and network transmission costs. However, embedded devices are usually limited by limited computing resources (such as CPU / GPU performance), memory capacity, and energy budget. Traditional deep learning models directly deployed on these devices face the following problems: High model complexity: Modern deep learning models (such as Vision Transformer and ResNet) contain millions or even hundreds of millions of parameters, with high computational requirements, resulting in long inference times and difficulty meeting real-time requirements.

[0004] Limited storage resources: Embedded devices typically have only a few megabytes (MB) of storage space, which is difficult to accommodate unoptimized model files (usually tens or even hundreds of megabytes).

[0005] Energy Consumption Constraints: Most devices are battery powered and must achieve efficient computing within a limited power consumption range.

[0006] To solve the above problems, researchers have proposed a variety of model optimization and deployment technologies in recent years, including model compression (such as pruning and quantization), knowledge distillation, and hardware acceleration. These methods can significantly reduce computational complexity and storage requirements without significantly losing model accuracy. Among them, the application of model quantization and knowledge distillation on edge devices is particularly critical: Model quantization reduces the model's storage requirements and speeds up inference operations on low-precision hardware by reducing model weights and activations from high-precision (such as 32-bit floating point numbers) to low-precision (such as 8-bit integers).

[0007] Knowledge distillation uses a large model with higher accuracy (teacher model) to transfer knowledge to a smaller model (student model), reducing model parameters while retaining high recognition performance.

[0008] In addition, hardware platforms are also providing support for edge AI. For example, the STM32Cube.AI tool chain can automatically convert optimized deep learning models into a format supported by embedded hardware and use hardware acceleration units (such as DSP and NPU) to achieve efficient reasoning. This collaborative optimization of software and hardware makes it feasible to implement efficient image recognition systems on resource-constrained edge devices. Summary of the invention

[0009] Aiming at the problems of insufficient real-time performance, waste of computing resources, unreasonable memory allocation, etc. in the application of existing embedded devices in efficient image recognition systems, the present invention proposes an efficient image recognition system based on embedded edge devices.

[0010] Efficient image recognition system based on embedded edge devices, including: Hardware acceleration module, including hardware-accelerated convolution operations; assigning inference tasks to different cores through dual-core collaboration, where the main core is responsible for inference calculations of deep learning models, and the auxiliary core is responsible for input data preprocessing, result transmission, and task allocation; achieving high-speed data transmission between memory and peripherals through direct memory access channels; Model quantization module, which converts the weights and activations in deep learning models from high-precision floating-point representations to low-precision integers, reducing the model's storage requirements and computational complexity; Automatic deployment module, which automatically converts the trained deep learning model into a format suitable for embedded hardware platforms and supports subsequent code generation and hardware adaptation; The data processing module preprocesses and enhances the input data to ensure that the data meets the input requirements of the deep learning model; The inference engine module is responsible for loading and running the optimized deep learning model on the embedded device.

[0011] As an improvement of the present invention, in the hardware acceleration module, the convolution operation is shown in the following formula, which is a matrix multiplication form: (1), where is the input feature map, size is , is the convolution kernel, size is , is the bias term, is the output feature map. In embedded devices, matrix multiplication is performed by hardware acceleration units combined with DMA channels to preload weights and input data.

[0012] As an improvement of the present invention, in the hardware acceleration module, the total execution time of the collaboration between the main and auxiliary cores is expressed as follows: (2), where and Represent the execution time of the main core and the auxiliary core respectively. By optimizing the task allocation strategy, and Achieve load balancing.

[0013] As an improvement of the present invention, in the hardware acceleration module, the block cache stores the input data, weights and intermediate calculation results in the on-chip SRAM in blocks to avoid frequent access to the off-chip memory. When the pipeline processing calculates the current convolution layer, the DMA channel preloads the weights and input data of the next layer. The pipeline efficiency of the data flow is expressed as shown in the following formula: (3), this module optimizes the pipeline design Close to 1.

[0014] As an improvement of the present invention, the model quantization module, 1) The quantization process is as follows: 1.1) Calculate the quantization scale factor s, as shown in the following formula: (4), where W is the weight tensor, N is the number of quantization bits, are the maximum and minimum values ​​of the weight respectively; 1.2) Map the weight from floating point numbers to integers, as shown in the formula: (5), where is the quantized integer value, z is the zero offset used to align the range of quantized values, W is the weight tensor, s is the quantization scale factor; 1.3) Dequantization during inference, as shown in the following formula: (6), where is the floating point value after dequantization, s is the quantization scale factor, is the quantized integer value, z is the zero point offset; 2) Dynamic range adjustment of activation quantization. For the activation value during model inference, the quantization range is dynamically adjusted according to each batch of input data, including: 2.1) Calculating the minimum value of activation and maximum value ; 2.2) Determine the activation quantization scale factor and Zero point: (7.1), (7.2); 2.3) The activation value is quantized as follows: (8) During inference, the activation value is restored using the dequantization formula: (9); 3) Quantization training: Quantization-aware training technology is added during model training to adapt to the low-precision computing environment after quantization, including: Inserting pseudo-quantization nodes: Simulating the quantization process during training, including quantization and dequantization operations. Pseudo-quantization is shown in the following formula: (10) Add a regularization term for quantization error to the training loss: (11), where: It's a mission loss. is the quantization error term, is the weight balance coefficient. The training process: optimize the model parameters through standard back propagation and adjust the quantization parameters at the same time. S and Z , so that the model can still maintain high precision in the quantized environment; 4) This module converts the quantized model into a format supported by embedded hardware through the STM32Cube.AI tool chain. The quantized weights and activation values ​​are stored in the on-chip memory, and the DSP or AI acceleration unit of the embedded hardware is used to perform low-precision reasoning. The optimization formula for quantized memory usage is: (12), where , is the ratio of floating point precision to quantization precision.

[0015] As an improvement of the present invention, the data processing module includes: data preprocessing; data enhancement; and data transmission and buffer optimization, wherein the data transmission and buffer optimization includes: 1) DMA channel transmission: In embedded devices, when data is transferred from an external storage device to the memory, direct memory access technology is used to avoid processor intervention. The data transmission time is expressed as: (18), where: N is the number of data blocks transmitted, B is the size of each block of data, Bandwidth is the transfer bandwidth of DMA; 2) Data buffering mechanism: A multi-level cache structure is set up in the data processing module to ensure that the preprocessed data can be quickly accessed during inference. The first-level cache stores the current batch data, and the second-level cache stores historical data or weights.

[0016] As an improvement of the present invention, the inference engine module includes: 1) Model loading and initialization: The inference engine module first loads the optimized model weights and structure, and initializes the inference running environment, including: 1.1) Model loading: The model is stored in the Flash or external memory of the embedded device in a quantized format. During the model loading phase, the inference engine loads the weights and topology into the on-chip memory: (19), where: is the weight stored in Flash, [s:e] indicates the range of the currently loaded model weight block, is the weight part loaded into SRAM; 1.2) Initialization process: The inference engine completes the following tasks during initialization: allocate memory areas for intermediate calculation results, configure operating parameters of the hardware acceleration unit, and verify the integrity of the model topology structure; 2) The inference engine module combines hardware resources and model characteristics to dynamically allocate computing tasks to different hardware units, including: 2.1) Task allocation strategy: For dual-core embedded devices, the inference engine adopts the main and auxiliary core collaborative working mechanism, where the total inference time is expressed as: (20), are the inference time of two cores respectively. By optimizing the allocation strategy, Achieve load balancing; 3) Efficient reasoning process. The execution process of the reasoning engine consists of multiple steps, and a pipeline structure is used to achieve parallel computing and data transmission, including: 3.1) Input data loading, pre-load the next batch of data through the DMA channel to reduce the waiting time between data loading and calculation: (22), where Indicates the amount of data that needs to be transferred via DMA. Indicates the maximum speed at which the DMA channel transfers data; 3.2) Computation module call, for core operations, the inference engine calls the hardware accelerator to perform calculations, the calculation formula is: (23), where: X is the input data, W is the weight matrix, B is the bias vector, ReLU It is the activation function. After the inference is completed, the inference engine stores the result in the output buffer and transmits it to the external device through the serial port, I2C or SPI interface.

[0017] As an improvement of the present invention, the dual-core collaborative processing step includes dividing the reasoning computing task into a computing task that the main core is responsible for and a data task that the auxiliary core is responsible for; the auxiliary core provides the pre-processed image data to the main core through a queue mechanism, and after the main core completes the reasoning task, it returns the result to the auxiliary core through shared memory, and the auxiliary core is responsible for post-processing and output of the result.

[0018] As an improvement of the present invention, the floating-point model is converted into a low-precision model through quantization technology, and the number of parameters is reduced by using a compression algorithm, so that the deep learning model can run efficiently on an embedded hardware platform with limited resources. The specific steps are as follows: Step 1: Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: (24), where X represents the floating point value to be quantized; Step 2: Select the quantization precision: For N-bit quantization, the range of integers is: (25); Step 3, quantization factor calculation: Calculate the quantization factor to map the floating point number to the specified integer interval. The scale factor is determined by the ratio of the maximum and minimum values: (26), where and are the maximum and minimum values ​​of floating point numbers, respectively. and is the quantization range of integer representation; Step 4, quantization formula: map floating point numbers to integers through scale factors: (27), where x is a floating point value, round is a rounding operation, q ( x ) is the quantized integer value, is the quantitative factor; Step 5: Dequantization operation: During inference, the quantized integer value needs to be mapped back to a floating-point value. The dequantization formula is: (28), where It is a floating point value after dequantization. The insertion points include the weights and activation values ​​of each layer, using the formula: (29), among which, s is the scaling factor, and is the activation value range.

[0019] As an improvement of the present invention, in the process of automatic deployment and adaptation function, it includes: 1) Model conversion and optimization: Automated deployment first requires converting the trained model from the framework into a format suitable for the embedded platform, including: 1.1) Conversion from floating-point model to quantized model: Converting the floating-point model to a low-precision model greatly reduces the model's storage and computing requirements. The quantization operation will select the appropriate accuracy based on the hardware capabilities of the target platform; 1.2) Model compression and optimization: Pruning: Reducing the size and computational complexity of the model by removing unimportant neural network connections; Knowledge distillation: Learning the knowledge of a large model through a smaller student network to reduce the complexity of the model; Matrix decomposition: Using mathematical methods to optimize the convolutional layer; 2) Target hardware adaptation and compilation: Different embedded platforms have different hardware architectures, computing capabilities, and storage resources. Therefore, automated deployment needs to adapt the model according to the specific hardware and optimize the use of computing and storage resources, including: 2.1) Hardware feature identification. During the automated deployment process, the system needs to automatically detect the characteristics of the target hardware device in order to select appropriate model conversion and optimization strategies for the target hardware; 2.2) Hardware acceleration support. Embedded devices integrate dedicated hardware acceleration units. Automated deployment tools need to support the compatibility of optimized models with hardware acceleration units to fully utilize the computing power of the hardware; 2.3) Code generation and compilation. The automated deployment tool converts the model into C language code or other embedded-supported formats. The compiler generates binary files that are directly executed on the hardware based on the characteristics of the target hardware; 2.4) Memory and storage management. The memory and storage resources of embedded devices are limited. Therefore, it is necessary to dynamically adjust the memory allocation strategy through automated deployment tools to ensure that the model can run in limited memory and storage space; 3) Automated deployment and debugging: including model conversion and compilation, and also involves automatically loading the final model into the target device, including: deploying the compiled model to the embedded device through the yaml file. For microcontrollers such as STM32, the ST-Link debugger and serial port are used for firmware loading. The file contains the relevant links of the B-CAMS-OMV camera, which is connected in hardware through a flexible flat cable.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention develops an efficient image recognition system for embedded edge devices, which should combine model optimization technology with automated deployment tools to achieve efficient and reliable image recognition functions while meeting the computing, storage and energy consumption constraints of embedded devices. This can not only improve the intelligence level of equipment, but also provide technical support for fields such as intelligent manufacturing, smart medical care and intelligent transportation. This system achieves efficient image recognition on resource-constrained embedded platforms by combining technologies such as model quantization, hardware acceleration, and dynamic task allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a basic idea framework diagram of the present invention; Figure 2 It is a practical operation flow chart of the present invention; Figure 3 It is a schematic diagram of the training results of the present invention; Figure 4 It is a schematic diagram of the quantification results of the present invention; Figure 5 This is a schematic diagram of successful deployment and normal operation of the present invention. DETAILED DESCRIPTION

[0022] The present invention aims at the problems of insufficient real-time performance, waste of computing resources, unreasonable memory allocation, etc. in the application of existing embedded devices in efficient image recognition systems. Figures 1 to 5 An efficient image recognition system based on embedded edge devices is proposed. Figure 3 The upper middle figure is the accuracy curve, where the horizontal axis is the number of parameter updates step, the vertical axis is the sample training cycle epoch, validation is the validation set, and train is the training set; the lower figure is the loss curve, where the horizontal axis is the number of parameter updates step, the vertical axis is the sample training cycle epoch, validation is the validation set, and train is the training set; Figure 4 The floating model confusion matrix validation set, the horizontal axis is the model prediction label, the vertical axis is the real label, daisy is daisy, dandelion is dandelion, roses is rose, sunflowers is sunflower, tulips is tulip, and the overall accuracy is 91.83%. This system achieves efficient image recognition on resource-constrained embedded platforms by combining model quantization, hardware acceleration, and dynamic task allocation.

[0023] Module 1: Hardware acceleration module. The hardware acceleration module is one of the core components of this invention, which aims to optimize the inference efficiency of deep learning models by utilizing the hardware resources of embedded devices (such as DSP, NPU and DMA channels). This module improves the computing performance of embedded devices through the following technologies.

[0024] The hardware acceleration module includes hardware accelerated convolution operations. Convolution operations are the most computationally intensive part of the image recognition model. This module maps the computational tasks of the convolution layer to the hardware matrix multiplication unit, accelerating the computation through efficient matrix partitioning and pipeline operations. The convolution operation is represented as the following matrix multiplication form: (1), where: is the input feature map, size is , is the convolution kernel, size is , is the bias term, is the output feature map.

[0025] In embedded devices, matrix multiplication is performed by hardware acceleration units, combined with DMA channels to preload weights and input data to reduce data transfer latency.

[0026] Dual-core collaboration mechanism. Embedded devices such as STM32 usually have a dual-core architecture (such as Cortex-M7 and Cortex-M4). This module uses dual-core collaboration to assign reasoning tasks to different cores: the main core (Cortex-M7): responsible for the reasoning calculation of the deep learning model, including convolution, pooling, and full connection operations; the auxiliary core (Cortex-M4): responsible for input data preprocessing, result transmission, and task allocation. The total execution time of the main and auxiliary core collaboration is expressed as: (2), where and Represent the execution time of the main core and the auxiliary core respectively. By optimizing the task allocation strategy, and Achieve load balancing.

[0027] In order to reduce the access delay of the off-chip memory, including memory optimization and data flow design, this module adopts the following optimization strategies: Block cache: Store input data, weights, and intermediate calculation results in blocks in on-chip SRAM to avoid frequent access to off-chip memory; Pipeline processing: When calculating the current convolution layer, the DMA channel preloads the weights and input data of the next layer to maximize hardware utilization; the pipeline efficiency of the data flow can be expressed as: (3), this module optimizes the pipeline design Close to 1, ensuring that data transfer does not become a bottleneck for computation.

[0028] Module 2: Model quantization module. The model quantization module is one of the key technologies for implementing efficient image recognition on embedded edge devices. Quantization converts the weights and activations in deep learning models from high-precision floating-point representations (such as FP32) to low-precision integers (such as INT8), significantly reducing the model's storage requirements and computational complexity while maintaining high recognition accuracy.

[0029] 1) The core process of weight quantization: Weight quantization refers to converting the weight parameters in the model from floating point numbers W to low-precision integers. The consistency of the numerical range is maintained through the scaling factor s and the zero point z.

[0030] The quantization process is as follows: Calculate the quantization scale factor s: (4), where: W is the weight tensor; N is the number of quantization bits (for example, N=8 for INT8); The maximum and minimum values ​​of the weights are respectively. Map weights from floating point numbers to integers: (5), where is the quantized integer value, and z is the zero offset, which is used to align the range of quantized values. Dequantization during inference: (6), where It is the floating point value after dequantization.

[0031] 2) Dynamic range adjustment of activation quantization: For the activation values ​​(intermediate feature maps) during model inference, the quantization range is usually dynamically adjusted based on each batch of input data. The specific steps include: calculating the minimum value of the activation and maximum value ; Determine activation quantization scale factor and Zero point: (7.1), (7.2); Activation value quantization formula: (8) During inference, the activation value is restored using the dequantization formula: (9).

[0032] 3) Quantization training: Direct application of quantization may cause a significant decrease in model accuracy. Therefore, quantization-aware training technology (QAT) is added during model training to adapt to the low-precision computing environment after quantization. Core steps: Insert pseudo-quantization nodes: simulate the quantization process during training, including quantization and dequantization operations. The pseudo-quantization formula is: (10) Add a regularization term for quantization error to the training loss: (11), where: is the task loss (such as cross entropy), is the quantization error term, is the weight balancing coefficient; Training process: Optimize model parameters through standard back-propagation, and adjust quantization parameters S and Z so that the model can still maintain high accuracy in the quantized environment.

[0033] 4) Reasoning advantages after quantization, including: Reduced storage requirements: The size of the model after quantization is reduced by about 75%. For example, the size of the FP32 model is 100MB, and the INT8 model only requires 25MB. Faster reasoning speed: On hardware that supports integer operations, the reasoning speed of the quantized model is 2-4 times that of the original model. Reduced power consumption: Using low-precision integer calculations, power consumption is reduced by about 50% compared to floating-point calculations.

[0034] 5) Adaptation to embedded hardware. This module converts the quantized model into a format supported by embedded hardware through the STM32Cube.AI toolchain. The quantized weights and activation values ​​are stored in on-chip memory (such as SRAM), and low-precision reasoning is performed using the DSP or AI acceleration unit of the embedded hardware. The optimization formula for quantized memory usage is: (12), where It is the ratio of floating point precision to quantization precision (for example, the ratio of FP32 to INT8 is 4).

[0035] Module 3: Automatic deployment module. The deployment module is used again to illustrate that in the present invention, the deployment part has realized an automated process in terms of training, quantization, and evaluation.

[0036] The present invention provides an automated operation that facilitates deployment. The specific steps are as follows: modify the various parameters in the yaml file. The code of the yaml file mainly consists of the following parts: general, operation_mode, dataset, preprocessing, tools, deployment, mlflow, hydra. 1) The general part provides information about the project and the path of the model to be deployed. 2) The operation_mode attribute is the operation mode and should be set to deployment. 3) The dataset part mainly has the class_name attribute, which specifies the class of the training model. 4) The preprocessing part is the preprocessing information, which mainly includes the following attributes: a) resizing: the most recent and only supported option of C code. b) aspect_ratio: one of fit, crop, or fill. If cropping, resize the image by cropping it into a square without aspect ratio distortion. If filling, add black borders above and below the image to make it square, otherwise the appropriate aspect ratio may not be retained. c) color_mode: one of: "grayscale", "rgb", or "bgr". 5) tools and deployment: To deploy the model in the STM32H747I-DISCO board, we will use STM32Cube.AI to convert the model to optimized C code and use STM32CubeIDE to build the C application and flash the board. There are several main parts: version: specifies the STM32Cube.AI version used to benchmark the model. optimization: string, defines the optimization used to generate the C model, options: "balanced", "time", "ram". path_to_stm32ai: path to the locally downloaded stm32ai executable file, otherwise False. path_to_cubeIDE: path to the stm32cubeide executable file. c_project_path: path to the stm32ai application code project. IDE: GCC, only supports options for stm32ai application code. verbosity: 0 or 1. Mode 0 is silent, mode 1 displays messages when building and flashing C applications on the STM32 target. serie: The STM32H7 series is used in this invention. Note that this option only supports the STM32AI application code option. board: Select the stm32 version to be used. The STM32H747I-DISCO is used in this invention. 6) mlflow and hydra: parts must always exist in the YAML configuration file.The hydra section can be used to specify the name of the directory that holds the experiment directories and / or a pattern to use for naming the experiment directories. Using the YAML code below, each time you run Model Zoo, an experiment directory is created containing all the directories and files created during the run. The names of the experiment directories are all unique because they are based on the date and time of the run.

[0037] Module 4: Data processing module. The data processing module is an important component of the image recognition system of the present invention. It is designed to preprocess and enhance the input data to ensure that the data meets the input requirements of the deep learning model and improve the robustness and recognition performance of the model. Through multi-channel data preprocessing and enhancement technology, this module can effectively optimize the input data flow and provide high-quality data input for the subsequent reasoning process, including: 1) Data preprocessing. Data preprocessing is a key step before model inference, which is used to format the original input data into a form that the model can directly process. The preprocessing process of this module includes the following aspects: a) Image format conversion. Since the image data received by the embedded device may come from devices such as cameras and sensors, the format may not be consistent with the model input requirements, so format conversion is required. For example, converting an RGB image to a grayscale image or normalizing it: (13), where R, G, and B are the red, green, and blue channels of the image, is the converted grayscale image; b) Cropping and scaling. The model usually requires a fixed-size input image, such as 224×224. This module adjusts the input image to the input size of the model through cropping and scaling operations: (14), where: is the scaling ratio; is the pixel coordinate in the target image. c) Normalization. Normalize the image pixel values ​​and scale the values ​​to the range required by the model (usually [0, 1] or [-1, 1]) to accelerate the convergence of the model: (15), where is the mean, is the standard deviation; 2) Data augmentation: Data augmentation techniques are used to generate diverse input samples during training and inference to improve the generalization and robustness of the model; The module supports the following enhancement methods: a) Random flip and rotation: Randomly flip the horizontal or vertical direction of the image during training, and randomly rotate it by a certain angle: (16), where is the rotation angle; b) Brightness and contrast adjustment: By randomly adjusting the brightness and contrast of the image, we simulate inputs under different lighting conditions: (17), where is the contrast factor, is the brightness offset; 3) Data transmission and buffer optimization: This module optimizes the transmission efficiency of input data streams through efficient transmission and buffer mechanisms while preprocessing data, reducing the impact of data loading on model reasoning. Including: a) DMA channel transfer. In embedded devices, when data is transferred from an external storage device (such as a camera sensor) to the memory, direct memory access (DMA) technology is used to avoid processor intervention. The data transfer time is expressed as: (18), where: N is the number of data blocks transmitted, B is the size of each data block, and Bandwidth is the transmission bandwidth of DMA; b) Data caching mechanism. A multi-level cache structure is set up in the data processing module to ensure that the preprocessed data can be quickly accessed during inference: Level 1 cache (SRAM): stores the current batch data to reduce access latency; Level 2 cache (Flash): stores historical data or weights for loading during task scheduling.

[0038] Module 5: Inference engine module. The inference engine module is the core execution unit of the present invention, responsible for loading and running the optimized deep learning model on the embedded device to achieve real-time and efficient image recognition. This module combines the model optimization results with the hardware characteristics, dynamically allocates computing tasks, rationally utilizes hardware resources, and provides online inference support, including: 1) Model loading and initialization. The inference engine module first loads the optimized model weights and structure, and initializes the inference running environment. This includes: a) Model loading. The model is stored in the Flash or external memory of the embedded device in a quantized format. During the model loading phase, the inference engine loads the weights and topology into the on-chip memory (SRAM): (19), where: is the weight stored in Flash, [s:e] represents the range of the currently loaded model weight block, It is the weight part loaded into SRAM; b) Initialization process. The inference engine completes the following tasks during initialization: allocate memory areas for intermediate calculation results; configure operating parameters of hardware acceleration units (such as DSP and NPU); verify the integrity of the model topology to ensure compatibility with the hardware; 2) Dynamic task allocation. To improve reasoning efficiency, the reasoning engine module combines hardware resources and model characteristics to dynamically allocate computing tasks to different hardware units, including: a) Task allocation strategy. For dual-core embedded devices (such as STM32's Cortex-M7 and Cortex-M4), the inference engine adopts a main-auxiliary core collaborative working mechanism: the main core (Cortex-M7): performs computationally intensive tasks (such as convolution, fully connected layer operations); the auxiliary core (Cortex-M4): performs data preprocessing, task scheduling, and result transmission. The total inference time is expressed as: (20), by optimizing the allocation strategy, Achieve load balancing; b) Computational resource utilization. By analyzing the memory requirements and computing time of each layer of calculation, the inference engine dynamically adjusts the task allocation strategy to maximize resource utilization: (twenty one); 3) Efficient reasoning process. The execution process of the reasoning engine consists of multiple steps, and a pipeline structure is used to achieve parallelization of calculation and data transmission: a) Input data loading. Preload the next batch of data through the DMA channel to reduce the waiting time between data loading and calculation: (twenty two); b) Computation module call. For core operations such as convolution and activation, the inference engine calls the hardware accelerator (such as DSP) to perform calculations. The calculation formula is: (23), where: X is the input data, W is the weight matrix, B is the bias vector, and ReLU is the activation function; c) Result output. After the inference is completed, the inference engine stores the result in the output buffer and transmits it to the external device through the serial port (UART), I2C or SPI interface.

[0039] The image recognition hardware acceleration method for embedded devices has the following main features: Dual-core collaborative processing: The main core is used to perform deep learning inference calculations, and the auxiliary core is used for data loading, preprocessing, and task distribution; Hardware matrix acceleration: The convolution operation of the model is performed through the dedicated hardware matrix multiplier of the embedded device; DMA data transfer: High-speed data transfer between memory and peripherals is achieved through direct memory access (DMA) channels, reducing processor intervention; Memory optimization management: By dividing the storage areas of SRAM and Flash, the memory resources required for model weights, activation values, and intermediate feature maps are dynamically allocated to improve computing efficiency.

[0040] Furthermore, the dual-core collaborative processing method mainly includes the following steps: The inference computing tasks are divided into computing tasks for the main core and data tasks for the auxiliary core; The auxiliary core provides the pre-processed image data to the main core through a queue mechanism; After the main core completes the inference task, it returns the result to the auxiliary core through the shared memory, and the auxiliary core is responsible for post-processing and outputting the result.

[0041] Using optimization methods based on quantization and model compression, the storage space and computational complexity of the model can be reduced without significantly reducing the accuracy of the model. Through quantization technology, the floating-point model is converted into a low-precision model, and the compression algorithm is used to reduce the number of parameters, so that the deep learning model can run efficiently on an embedded hardware platform with limited resources. The specific steps are as follows: 1) Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: (24), where X represents the floating point value to be quantized; 2) Select quantization accuracy: The quantization accuracy is usually 8 or 16 bits. For 8-bit quantization, the range of integers is usually set to [-128, 127]. For N-bit quantization, the range of integers is: (25); 3) Quantization factor calculation: Calculate the quantization factor (or scale factor), which is used to map the floating point number to a specified integer range. The scale factor is usually determined by the ratio of the maximum and minimum values: (26), where and are the maximum and minimum values ​​of floating point numbers, respectively. and is the quantization range of integer representation; 4) Quantization formula: Map floating point numbers to integers through scale factors: (27), where x is a floating point value, round is a rounding operation, and q(x) is the quantized integer value; 5) Dequantization operation: During inference, the quantized integer value needs to be mapped back to a floating-point value. The dequantization formula is: (28), where is a floating point value after dequantization. The insertion points include the weights and activation values ​​of each layer, using the formula: (29), where s is the scaling factor, and is the activation value range.

[0042] In order to simplify the process of migrating deep learning models from the cloud to embedded devices, the present invention provides an automated model deployment framework. During the deployment process, the framework can automatically convert the trained deep learning model into a format suitable for embedded hardware platforms, and support subsequent code generation and hardware adaptation through tools such as STM32Cube.AI, and the deployment is fast and easy to operate. In the entire project operation, in order to facilitate the implementation of operations such as training, quantization, evaluation and deployment, a complete and easy-to-operate yaml file is provided.

[0043] Furthermore, in the process of realizing the automated deployment and adaptation functions, there are mainly the following steps: 1) Model conversion and optimization. Automated deployment first requires converting the trained model from the framework (TensorFlow is used in this paper) into a format suitable for embedded platforms. There are two parts: a) Conversion from floating-point model to quantized model: Convert the floating-point model to a low-precision model (such as INT8, INT16), which greatly reduces the storage and computing requirements of the model. The quantization operation will select the appropriate precision based on the hardware capabilities of the target platform; b) Model compression and optimization: Pruning: Reduce the size and computational complexity of the model by removing unimportant neural network connections. Knowledge distillation: Learn the knowledge of a large model (teacher network) through a smaller student network to reduce the complexity of the model. Matrix decomposition: Use mathematical methods such as low-rank decomposition to optimize convolutional layers, etc.; 2) Target hardware adaptation and compilation: Different embedded platforms (such as STM32) have different hardware architectures, computing capabilities, and storage resources. Therefore, automated deployment requires adapting the model to the specific hardware to optimize computing and storage resource usage. This includes: a) Hardware feature identification: During the automated deployment process, the system needs to automatically detect the characteristics of the target hardware device (such as computing power, storage, cache size, etc.) in order to select the appropriate model conversion and optimization strategy for the target hardware; b) Hardware acceleration support: Embedded devices such as STM32 usually integrate dedicated hardware acceleration units (such as DSP, AI accelerator). Automated deployment tools need to support compatibility between optimized models and hardware acceleration units to fully utilize the computing power of the hardware; c) Code generation and compilation: Automated deployment tools (such as STM32Cube.AI) convert the model into C language code or other embedded supported formats. The compiler generates binary files that can be executed directly on the hardware according to the characteristics of the target hardware; d) Memory and storage management: The memory and storage resources of embedded devices are usually limited, so it is necessary to dynamically adjust the memory allocation strategy through automated deployment tools to ensure that the model can run in limited memory and storage space; 3) Automated deployment and debugging: including model conversion and compilation, and also involves automatically loading the final model into the target device. The specific implementation method is as follows: deploy the compiled model to the embedded device through the yaml file. For microcontrollers such as STM32, the firmware is loaded using the ST-Link debugger, serial port, etc. The file contains the relevant links of the B-CAMS-OMV camera, which is connected in hardware through a flexible flat cable.

[0044] In the actual operation of this invention, there are mainly the following steps: training, quantification, evaluation, benchmarking, prediction, and deployment.

[0045] Training: Prepare a suitable data set. The directory tree of the data set is as follows: dataset_root_directory / class_a / a_image_1.jpg a_image_2.jpg class_b / b_image_1.jpg b_image_2.jpg Configure your own related yaml files. For specific operations, please refer to the automatic deployment module in the invention content.

[0046] In the src directory of the previous folder, activate the virtual environment and run the code: python stm32ai_main.py --config-path . / config_file_examples / --config-name training_config.yaml After running, the trained h5 model can be found in the corresponding experiments_outputs / folder. All training and evaluation artifacts are saved in the current output simulation directory outputs / {run_time}, which can be visualized using the tensorboard --logdir logs command.

[0047] Quantization: Prepare the dataset to be used for quantization, either the complete training set or a specific training set dedicated to activation calibration. If you only want to quantize the model on a random portion of the quantization set, simply set the percentage value in the quantization_split parameter. Configure the relevant yaml files.

[0048] Run the command in the corresponding src file to perform the quantization operation: python stm32ai_main.py --config-path . / config_file_examples / --config-name chain_eqe_config.yaml The rest of the operation modes are basically similar. First, modify the yaml file, then obtain the corresponding results from the corresponding folder, and finally deploy the model that has undergone training, quantization, evaluation, and other operations to the hardware to achieve the corresponding functions.

[0049] It is worth noting that in the process of performing the above operations, we must first connect the STM microcontroller we use to the compatible camera module.

[0050] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. An efficient image recognition system based on embedded edge devices, characterized by: include: Hardware acceleration module, including hardware accelerated convolution operations; The reasoning tasks are assigned to different cores through dual-core collaboration, where the main core is responsible for the reasoning calculation of the deep learning model, and the auxiliary core is responsible for input data preprocessing, result transmission and task allocation; High-speed data transfer between memory and peripherals is achieved through direct memory access channels; Model quantization module, which converts the weights and activations in deep learning models from high-precision floating-point representations to low-precision integers, reducing the model's storage requirements and computational complexity; Automatic deployment module, which automatically converts the trained deep learning model into a format suitable for embedded hardware platforms and supports subsequent code generation and hardware adaptation; The data processing module preprocesses and enhances the input data to ensure that the data meets the input requirements of the deep learning model; The inference engine module is responsible for loading and running the optimized deep learning model on the embedded device.

2. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: In the hardware acceleration module, the convolution operation is shown in the following formula, which is a matrix multiplication form: (1), where is the input feature map, size is , is the convolution kernel, size is , is the bias term, is the output feature map. In embedded devices, matrix multiplication is performed by hardware acceleration units combined with DMA channels to preload weights and input data.

3. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: In the hardware acceleration module, the total execution time of the main and auxiliary cores in collaboration is expressed as follows: (2), where and Represent the execution time of the main core and the auxiliary core respectively. By optimizing the task allocation strategy, and Achieve load balancing.

4. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: In the hardware acceleration module, the block cache stores the input data, weights and intermediate calculation results in the on-chip SRAM in blocks to avoid frequent access to the off-chip memory. When the pipeline processing calculates the current convolution layer, the DMA channel preloads the weights and input data of the next layer. The pipeline efficiency of the data flow is expressed as follows: (3), this module optimizes the pipeline design Close to 1.

5. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: The model quantization module, 1) The quantification process is as follows: 1.1) Calculate the quantization scale factor s , as shown below: (4), where W is the weight tensor, N is the number of quantization bits, are the maximum and minimum values ​​of the weight respectively; 1.2) Map weights from floating point numbers to integers as shown in the formula: (5), where is the quantized integer value, z is the zero offset used to align the range of quantized values, W is the weight tensor, s is the quantization scale factor; 1.3) Dequantization during inference, as shown in the following formula: (6), where is the floating point value after dequantization, s is the quantization scale factor, is the quantized integer value, z is the zero offset; 2) Dynamic range adjustment of activation quantization. For activation values ​​during model inference, the quantization range is dynamically adjusted based on each batch of input data, including: 2.1) Calculate the minimum activation value and maximum value ; 2.2) Determine the activation quantization scale factor and Zero point: (7.1), (7.2), 2.3) The activation value quantization is shown as follows: (8) During inference, the activation value is restored using the dequantization formula: (9); 3) Quantization training adds quantization-aware training technology during model training to adapt to the low-precision computing environment after quantization, including: Inserting pseudo-quantization nodes: Simulating the quantization process during training, including quantization and dequantization operations. Pseudo-quantization is shown in the following formula: (10) Add a regularization term for quantization error to the training loss: (11), where: It's a mission loss. is the quantization error term, is the weight balance coefficient; training process: optimize the model parameters through standard back propagation, and adjust the quantization parameters S and Z at the same time, so that the model can still maintain high accuracy in the quantized environment; 4) This module converts the quantized model into a format supported by embedded hardware through the STM32Cube.AI tool chain. The quantized weights and activation values ​​are stored in the on-chip memory, and low-precision reasoning is performed using the DSP or AI acceleration unit of the embedded hardware. The quantized memory usage optimization formula is: (12), where It is the ratio of floating point precision to quantization precision.

6. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: The data processing module includes: data preprocessing; data enhancement; and data transmission and buffer optimization, wherein the data transmission and buffer optimization includes: 1) DMA channel transmission: In embedded devices, when data is transferred from an external storage device to the memory, direct memory access technology is used to avoid processor intervention. The data transmission time is expressed as: (18) , where: is the number of data blocks transmitted, is the size of each block of data, is the transfer bandwidth of DMA; 2) Data buffering mechanism: A multi-level cache structure is set up in the data processing module to ensure that the preprocessed data can be quickly accessed during inference. The first-level cache stores the current batch data, and the second-level cache stores historical data or weights.

7. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: The inference engine module comprises: 1) Model loading and initialization: The inference engine module first loads the optimized model weights and structure, and initializes the inference running environment, including: 1.1) Model loading: The model is stored in the Flash or external memory of the embedded device in a quantized format. During the model loading phase, the inference engine loads the weights and topology into the on-chip memory: (19) , where: is the weight stored in Flash, Indicates the range of the currently loaded model weight block, is the weight part loaded into SRAM; 1.2) Initialization process: The inference engine completes the following tasks during initialization: allocate memory areas for intermediate calculation results, configure operating parameters of the hardware acceleration unit, and verify the integrity of the model topology structure; 2) The inference engine module combines hardware resources and model characteristics to dynamically allocate computing tasks to different hardware units, including: 2.1) Task allocation strategy: For dual-core embedded devices, the inference engine adopts the main and auxiliary core collaborative working mechanism, where the total inference time is expressed as: (20), The inference time of the two cores is respectively, and the allocation strategy is optimized so that Achieve load balancing; 3) Efficient reasoning process. The execution process of the reasoning engine consists of multiple steps, and a pipeline structure is used to achieve parallel computing and data transmission, including: 3.1) Input data loading, pre-load the next batch of data through the DMA channel to reduce the waiting time between data loading and calculation: (22), where Indicates the amount of data that needs to be transferred via DMA. Indicates the maximum speed at which the DMA channel transfers data; 3.2) Computation module call, for core operations, the inference engine calls the hardware accelerator to perform calculations, the calculation formula is: (23), where: is the input data, is the weight matrix, is the bias vector, It is the activation function. After the inference is completed, the inference engine stores the result in the output buffer and transmits it to the external device through the serial port, I2C or SPI interface.

8. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: The dual-core collaborative processing steps include dividing the inference computing tasks into computing tasks that the main core is responsible for and data tasks that the auxiliary core is responsible for; the auxiliary core provides pre-processed image data to the main core through a queue mechanism. After the main core completes the inference task, it returns the result to the auxiliary core through shared memory. The auxiliary core is responsible for post-processing and output of the result.

9. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: Through quantization technology, the floating-point model is converted into a low-precision model, and the compression algorithm is used to reduce the number of parameters, so that the deep learning model can run efficiently on the embedded hardware platform with limited resources. The specific steps are as follows: Step 1: Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: (24), where Represents the floating point value to be quantized; Step 2: Select the quantization precision: For N-bit quantization, the range of integers is: (25); Step 3, quantization factor calculation: Calculate the quantization factor to map the floating point number to the specified integer interval. The scale factor is determined by the ratio of the maximum and minimum values: (26), where and are the maximum and minimum values ​​of floating point numbers, respectively. and is the quantization range represented by integers; Step 4, quantization formula: map floating point numbers to integers through scale factors: (27), where is a floating point value, round is a rounding operation, is the quantized integer value, is the quantitative factor; Step 5: Dequantization operation: During inference, the quantized integer value needs to be mapped back to a floating-point value. The dequantization formula is: (28), where It is a floating point value after dequantization. The insertion points include the weights and activation values ​​of each layer, using the formula: (29), where s is the scaling factor, and is the activation value range.

10. The efficient image recognition system based on embedded edge devices according to claim 1, characterized in that: In the process of automated deployment and adaptation functions, it includes: 1) Model conversion and optimization: Automated deployment first requires converting the trained model from the framework into a format suitable for the embedded platform, including: 1.1) Conversion from floating-point model to quantized model: Convert the floating-point model to a low-precision model, which greatly reduces the storage and computing requirements of the model. The quantization operation will select the appropriate precision based on the hardware capabilities of the target platform. 1.2) Model compression and optimization: Pruning, reducing the size and computational complexity of the model by removing unimportant neural network connections; knowledge distillation, learning the knowledge of a large model through a smaller student network to reduce the complexity of the model; matrix decomposition, using mathematical methods to optimize the convolutional layer; 2) Target hardware adaptation and compilation: Different embedded platforms have different hardware architectures, computing capabilities, and storage resources. Therefore, automated deployment requires adapting the model to the specific hardware and optimizing the use of computing and storage resources, including: 2.1) Hardware feature identification: During the automated deployment process, the system needs to automatically detect the characteristics of the target hardware device in order to select the appropriate model conversion and optimization strategy for the target hardware; 2.2) Hardware acceleration support: Embedded devices integrate dedicated hardware acceleration units. Automated deployment tools need to support compatibility between optimized models and hardware acceleration units to fully utilize the computing power of the hardware. 2.3) Code generation and compilation: The automated deployment tool converts the model into C language code or other embedded supported formats. The compiler generates binary files that are directly executed on the hardware based on the target hardware characteristics. 2.4) Memory and storage management: The memory and storage resources of embedded devices are limited, so it is necessary to dynamically adjust the memory allocation strategy through automated deployment tools to ensure that the model can run in limited memory and storage space; 3) Automated deployment and debugging: This includes model conversion and compilation, and also involves automatically loading the final model into the target device, including: The compiled model is deployed to the embedded device through the yaml file. For microcontrollers such as STM32, the ST-Link debugger and serial port are used to load the firmware. The file contains the relevant links of the B-CAMS-OMV camera, which is connected to the hardware through a flexible flat cable.

Images