Efficient image recognition system based on embedded edge device

Abstract

The application relates to an efficient image recognition system based on an embedded edge device, and belongs to the technical field of computer application, which comprises a hardware acceleration module, a convolution operation of hardware acceleration; an inference task is distributed to different cores through double-core cooperation; a model quantization module; quantization is realized by converting weights and activations in a deep learning model from high-precision floating-point number representation to low-precision integer; an automatic deployment module automatically converts a trained deep learning model into a format suitable for an embedded hardware platform; a data processing module pre-processes and enhances input data, so that the data meet the input requirements of the deep learning model; and an inference engine module is responsible for loading and running the optimized deep learning model on the embedded device; the application combines model optimization technology and automatic deployment tools, meets the calculation, storage and energy consumption restrictions of the embedded device, and realizes efficient and reliable image recognition functions.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, specifically a high-efficiency image recognition system based on embedded edge devices. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence (AI) and deep learning technologies, image recognition systems based on deep neural networks (DNNs) have been widely used. These systems have demonstrated powerful performance in tasks such as image classification, object detection, and action recognition. However, traditional deep learning models often rely on high-performance computing devices such as cloud servers or GPUs to achieve efficient operation, which poses a significant challenge to resource-constrained embedded edge devices.

[0003] Embedded edge devices (such as STM32 series microcontrollers and Raspberry Pi) have become core components in Internet of Things (IoT) scenarios due to their low power consumption, small size, and deployment flexibility. Especially in fields such as industrial automation, smart homes, and drone control, embedded devices can perform data processing locally, reducing reliance on the cloud and significantly lowering latency and network transmission costs. However, embedded devices are typically limited by limited computing resources (such as CPU / GPU performance), memory capacity, and energy budgets. Deploying traditional deep learning models directly on these devices faces the following challenges:

[0004] High model complexity: Modern deep learning models (such as Vision Transformer and ResNet) contain millions or even hundreds of millions of parameters, which have high computational requirements, resulting in long inference times and making it difficult to meet real-time requirements.

[0005] Limited storage resources: Embedded devices typically have only a few megabytes (MB) of storage space, which is insufficient to accommodate unoptimized model files (usually tens or even hundreds of megabytes).

[0006] Energy consumption limitations: Most devices are battery powered and must achieve efficient computing within a limited power consumption range.

[0007] To address these issues, researchers have proposed various model optimization and deployment techniques in recent years, including model compression (such as pruning and quantization), knowledge distillation, and hardware acceleration. These methods can significantly reduce computational complexity and storage requirements without substantially sacrificing model accuracy. Among these, model quantization and knowledge distillation are particularly crucial for their application on edge devices.

[0008] Model quantization reduces model weights and activations from high precision (such as 32-bit floating-point numbers) to low precision (such as 8-bit integers), which not only reduces the model's storage requirements but also accelerates inference operations on low-precision hardware.

[0009] Knowledge distillation uses a large, highly accurate model (teacher model) to transfer knowledge to a smaller model (student model), reducing model parameters while maintaining high recognition performance.

[0010] Furthermore, hardware platforms are also supporting edge AI. For example, the STM32Cube.AI toolchain can automatically convert optimized deep learning models into a format supported by embedded hardware and utilize hardware acceleration units (such as DSPs and NPUs) for efficient inference. This hardware-software co-optimization approach makes it feasible to implement efficient image recognition systems on resource-constrained edge devices. Summary of the Invention

[0011] This invention addresses the problems of insufficient real-time performance, wasted computing resources, and unreasonable memory allocation in the application of existing embedded devices in high-efficiency image recognition systems, and proposes a high-efficiency image recognition system based on embedded edge devices.

[0012] High-efficiency image recognition systems based on embedded edge devices include:

[0013] The hardware acceleration module includes hardware-accelerated convolution operations; it distributes inference tasks to different cores through dual-core collaboration, with the main core responsible for inference computation of the deep learning model and the auxiliary core responsible for input data preprocessing, result transmission, and task allocation; and it achieves high-speed data transmission between memory and peripherals through direct memory access channels.

[0014] The model quantization module reduces the model's storage requirements and computational complexity by converting the weights and activations in the deep learning model from high-precision floating-point representations to low-precision integers.

[0015] The automatic deployment module automatically converts the trained deep learning model into a format suitable for embedded hardware platforms and supports subsequent code generation and hardware adaptation.

[0016] The data processing module preprocesses and enhances the input data to ensure that the data meets the input requirements of the deep learning model.

[0017] The inference engine module is responsible for loading and running optimized deep learning models on embedded devices.

[0018] As an improvement to the present invention, in the hardware acceleration module, the convolution operation is as shown in the following formula, which is in the form of matrix multiplication: (1), where It is the input feature map, with a size of , It is a convolution kernel, with a size of , It is a bias term. It is the output feature map. In embedded devices, matrix multiplication is performed by a hardware acceleration unit, combined with DMA channels to preload weights and input data.

[0019] As an improvement to the present invention, the total execution time of the main-auxiliary core collaboration in the hardware acceleration module is expressed by the following formula: (2), where, and These represent the execution times of the main core and the auxiliary core, respectively. By optimizing the task allocation strategy, it is possible to... and Achieve load balancing.

[0020] As an improvement of the present invention, in the hardware acceleration module, the block cache stores the input data, weights, and intermediate calculation results in blocks in the on-chip SRAM to avoid frequent access to off-chip memory. When the pipelined processing calculates the current convolutional layer, the DMA channel preloads the weights and input data of the next layer. The pipeline efficiency of the data flow is expressed as follows: (3) This module optimizes the pipeline design to It is close to 1.

[0021] As an improvement to the present invention, the model quantization module,

[0022] 1) The quantization process is as follows: 1.1) Calculate the quantization scaling factor s, as shown in the following formula: (4), of which W It is a weight tensor. N It is the number of quantization bits. These are the maximum and minimum values ​​of the weights, respectively; 1.2) Map the weights from floating-point numbers to integers, as shown in the formula: (5), among which, It is the quantized integer value. z It is the zero-point offset, used to align the range of quantized values. W It is a weight tensor. s 1.3) The inverse quantization during inference is shown in the following formula: (6), among which, This is the dequantized floating-point value. s To quantify the scaling factor, It is the quantized integer value. z1) Zero-point offset; 2) Dynamic range adjustment of activation quantization. For activation values ​​during model inference, the quantization range is dynamically adjusted according to each batch of input data, including: 2.1) Calculating the minimum activation value. and maximum value 2.2) Determine the activation quantification ratio factor and Midnight: (7.1), (7.2); 2.3) The activation value quantization is shown in the following formula: (8) During inference, use the dequantization formula to recover the activation value: (9); 3) Quantization training incorporates quantization-aware training techniques during model training to adapt to the low-precision computing environment after quantization, including: inserting pseudo-quantization nodes: simulating the quantization process during training, including quantization and dequantization operations, as shown in the following formula: (10) Add a regularization term for quantization error to the training loss: (11), where: It's a mission loss. It is the quantization error term. These are the weight balancing coefficients. The training process involves optimizing the model parameters through standard backpropagation, while simultaneously adjusting the quantization parameters. S and Z This ensures the model maintains high accuracy even in the quantized environment; 4) This module uses the STM32Cube.AI toolchain to convert the quantized model into a format supported by embedded hardware. The quantized weights and activation values ​​are stored in on-chip memory, and low-precision inference is performed using the DSP or AI acceleration unit of the embedded hardware. The memory usage optimization formula for quantization is: (12), of which , is the ratio of floating-point precision to quantization precision.

[0023] As an improvement of the present invention, the data processing module includes: data preprocessing; data enhancement; and data transmission and buffering optimization, wherein the data transmission and buffering optimization includes:

[0024] 1) DMA Channel Transfer: In embedded devices, when data is transferred from external storage devices to memory, Direct Memory Access (DMA) technology is used to avoid processor intervention. The data transfer time is expressed as: (18), of which: N The number of data blocks to be transmitted. B For the size of each data block, Bandwidth For DMA transfer bandwidth;

[0025] 2) Data Buffering Mechanism: A multi-level cache structure is set up in the data processing module to ensure that the preprocessed data can be accessed quickly during inference. The first-level cache stores the current batch of data, and the second-level cache stores historical data or weights.

[0026] As an improvement of the present invention, the inference engine module includes:

[0027] 1) Model Loading and Initialization: The inference engine module first loads the optimized model weights and structure, and initializes the inference runtime environment, including:

[0028] 1.1) Model Loading: The model is stored in the embedded device's Flash or external memory in a quantized format. During the model loading phase, the inference engine loads the weights and topology into on-chip memory. (19), of which: These are the weights stored in Flash. [s:e] represents the range of the currently loaded model weight block. It is the weight portion loaded into SRAM;

[0029] 1.2) Initialization process: During initialization, the inference engine completes the following tasks: allocating memory areas for intermediate calculation results, configuring the running parameters of the hardware acceleration unit, and verifying the integrity of the model topology.

[0030] 2) The inference engine module dynamically allocates computational tasks to different hardware units based on hardware resources and model characteristics, including:

[0031] 2.1) Task Allocation Strategy: For dual-core embedded devices, the inference engine adopts a main-slave core collaborative working mechanism, where the total inference time is expressed as: (20) These are two core inference times, and by optimizing the allocation strategy, we can achieve... Achieve load balancing;

[0032] 3) Efficient inference process: The execution process of the inference engine consists of multiple steps, employing a pipelined structure to achieve parallelization of computation and data transmission, including:

[0033] 3.1) Input data loading: Preload the next batch of data via the DMA channel to reduce the waiting time between data loading and computation. (22), among which, This indicates the amount of data that needs to be transferred via DMA. Indicates the maximum speed at which data can be transferred via the DMA channel;

[0034] 3.2) Calculation module call: For core operations, the inference engine calls the hardware accelerator to perform calculations. The calculation formula is as follows: (23), of which: X It is the input data. W It is a weight matrix. B It is a bias vector. ReLU It is the activation function. After inference is completed, the inference engine stores the result in the output buffer and transmits it to an external device via serial port, I2C or SPI interface.

[0035] As an improvement of the present invention, the dual-core collaborative processing steps include dividing the inference calculation task into a calculation task handled by the main core and a data task handled by the auxiliary core; the auxiliary core provides preprocessed image data to the main core through a queue mechanism; after the main core completes the inference task, it returns the result to the auxiliary core through shared memory; and the auxiliary core is responsible for the post-processing and output of the result.

[0036] As an improvement to this invention, a quantization technique is used to convert the floating-point model into a low-precision model, and a compression algorithm is employed to reduce the number of parameters, enabling the deep learning model to run efficiently on resource-constrained embedded hardware platforms. The specific steps are as follows:

[0037] Step 1, Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: (24), where X represents the floating-point value to be quantized;

[0038] Step 2, Select quantization precision: For N-bit quantization, the range of integers is: (25);

[0039] Step 3, Quantization Factor Calculation: The quantization factor is calculated to map floating-point numbers to a specified integer range. The scaling factor is determined by the ratio of the maximum to the minimum value. (26), among which, and These are the maximum and minimum values ​​of the floating-point number, respectively. and It is the quantization range of integer representation;

[0040] Step 4, Quantization Formula: Map floating-point numbers to integers using a scaling factor: (27), among which, x It is a floating-point value. round It's a rounding operation. q ( x () is the quantized integer value. For quantification factors;

[0041] Step 5, Dequantization: During inference, the quantized integer values ​​need to be mapped back to floating-point values. The dequantization formula is: (28), among which, It is a dequantized floating-point value. The insertion point includes the weights and activation values ​​of each layer, using the formula: (29), among which, s It is a scaling factor. and It is the range of activation values.

[0042] As an improvement to the present invention, the process of automated deployment and adaptation includes:

[0043] 1) Model conversion and optimization: Automated deployment first requires converting the trained model from the framework to a format suitable for embedded platforms, including: 1.1) Conversion from floating-point models to quantized models: Converting floating-point models to low-precision models significantly reduces the model's storage and computational requirements. The quantization operation selects an appropriate precision based on the target platform's hardware capabilities; 1.2) Model compression and optimization: Pruning, reducing model size and computational complexity by removing unimportant neural network connections; Knowledge distillation, reducing model complexity by learning knowledge from a larger model through a smaller student network; Matrix factorization, optimizing convolutional layers using mathematical methods.

[0044] 2) Target Hardware Adaptation and Compilation: Different embedded platforms have different hardware architectures, computing capabilities, and storage resources. Therefore, automated deployment requires adapting the model to the specific hardware and optimizing the use of computing and storage resources. This includes: 2.1) Hardware Characteristic Identification: During automated deployment, the system needs to automatically detect the characteristics of the target hardware device to select appropriate model conversion and optimization strategies; 2.2) Hardware Acceleration Support: Embedded devices integrate dedicated hardware acceleration units. Automated deployment tools need to support compatibility between the optimized model and the hardware acceleration unit to fully utilize the hardware's computing power; 2.3) Code Generation and Compilation: Automated deployment tools convert the model into C language code or other embedded-supported formats. The compiler generates binary files that can be executed directly on the hardware based on the target hardware characteristics; 2.4) Memory and Storage Management: Embedded devices have limited memory and storage resources. Therefore, automated deployment tools need to dynamically adjust memory allocation strategies to ensure that the model can run within limited memory and storage space.

[0045] 3) Automated deployment and debugging: This includes model conversion and compilation, as well as automatically loading the final model into the target device. This includes deploying the compiled model to the embedded device via a YAML file. For microcontrollers like STM32, the ST-Link debugger and serial port are used for firmware loading. The file contains the relevant links for the B-CAMS-OMV camera, and the hardware is connected via a flexible flat cable.

[0046] Compared to existing technologies, the advantages of this invention are as follows: This invention develops a high-efficiency image recognition system for embedded edge devices. This system combines model optimization techniques with automated deployment tools to achieve efficient and reliable image recognition while meeting the computational, storage, and energy consumption limitations of embedded devices. This not only enhances the intelligence level of devices but also provides technical support for fields such as intelligent manufacturing, smart healthcare, and intelligent transportation. This system achieves high-efficiency image recognition on resource-constrained embedded platforms by combining model quantization, hardware acceleration, and dynamic task allocation technologies. Attached Figure Description

[0047] Figure 1 This is a basic conceptual architecture diagram of the present invention;

[0048] Figure 2 This is a flowchart illustrating the actual operation of the present invention;

[0049] Figure 3 This is a schematic diagram of the training results of the present invention;

[0050] Figure 4 This is a schematic diagram of the quantification results of the present invention;

[0051] Figure 5 This diagram illustrates the successful deployment and normal operation of the present invention. Detailed Implementation

[0052] This invention addresses the problems of insufficient real-time performance, wasted computing resources, and unreasonable memory allocation in the application of existing embedded devices in high-efficiency image recognition systems, and combines them with the appendix. Figures 1 to 5 A high-efficiency image recognition system based on embedded edge devices is proposed, wherein... Figure 3 The top-middle figure is the accuracy curve, where: the horizontal axis represents the number of parameter updates (steps), the vertical axis represents the number of training epochs, validation represents the validation set, and train represents the training set; the bottom figure is the loss curve, where the horizontal axis represents the number of parameter updates (steps), the vertical axis represents the number of training epochs, validation represents the validation set, and train represents the training set. Figure 4 The model confusion matrix validation set is shown, with the x-axis representing the model's predicted label and the y-axis representing the true label. "daisy" represents daisy, "dandelion" represents dandelion, "roses" represents rose, "sunflowers" represents sunflower, and "tulips" represents tulip. The overall accuracy is 91.83%. This system achieves efficient image recognition on resource-constrained embedded platforms by combining model quantization, hardware acceleration, and dynamic task allocation technologies.

[0053] Module 1: Hardware Acceleration Module. The hardware acceleration module is one of the core components of this invention, designed to optimize the inference efficiency of deep learning models by utilizing the hardware resources of embedded devices (such as DSP, NPU, and DMA channels). This module improves the computing performance of embedded devices through the following technologies.

[0054] The hardware acceleration module includes hardware-accelerated convolution operations. Convolution is the most computationally intensive part of an image recognition model. This module maps the computational tasks of the convolutional layer to hardware matrix multiplication units, accelerating computation through efficient matrix partitioning and pipelined operations. The convolution operation is represented as the following matrix multiplication form: (1), where: It is the input feature map, with a size of , It is a convolution kernel, with a size of , It is a bias term. It is the output feature map.

[0055] In embedded devices, matrix multiplication is performed through hardware acceleration units, which, combined with DMA channels to preload weights and input data, reduce data transfer latency.

[0056] Dual-core collaboration mechanism. Embedded devices such as STM32 typically have a dual-core architecture (e.g., Cortex-M7 and Cortex-M4). This module utilizes dual-core collaboration to distribute inference tasks across different cores: the main core (Cortex-M7) is responsible for inference computations of the deep learning model, including operations such as convolution, pooling, and fully connected layers; the auxiliary core (Cortex-M4) is responsible for input data preprocessing, result transmission, and task allocation. The total execution time of the main-auxiliary core collaboration is expressed as: (2), where, and These represent the execution times of the main core and the auxiliary core, respectively. By optimizing the task allocation strategy, it becomes possible to... and Achieve load balancing.

[0057] To reduce access latency to off-chip memory, memory optimization and data flow design are also included. This module adopts the following optimization strategies:

[0058] Block caching: Input data, weights, and intermediate calculation results are stored in blocks in on-chip SRAM to avoid frequent access to off-chip memory;

[0059] Pipeline processing: When computing the current convolutional layer, the DMA channel preloads the weights and input data of the next layer to maximize hardware utilization; the pipeline efficiency of the data flow can be expressed as: (3) This module optimizes the pipeline design to A value close to 1 ensures that data transmission does not become a bottleneck for computation.

[0060] Module 2: Model Quantization Module. The model quantization module is one of the key technologies for achieving efficient image recognition on embedded edge devices in this invention. Quantization significantly reduces the storage requirements and computational complexity of the model by converting the weights and activations in the deep learning model from high-precision floating-point representations (such as FP32) to low-precision integers (such as INT8), while maintaining high recognition accuracy.

[0061] 1) The core process of weight quantization: Weight quantization refers to converting the weight parameters in the model from floating-point numbers W to low-precision integers. The numerical range is kept consistent by using a scaling factor s and a zero point z.

[0062] The quantization process is as follows: Calculate the quantization scaling factor s: (4), where: W is the weight tensor; N is the number of bits for quantization (e.g., N=8 for INT8); These are the maximum and minimum values ​​of the weights, respectively. Mapping the weights from floating-point numbers to integers: (5), among which, This is the quantized integer value, and z is the zero-point offset used to align the range of quantized values. Dequantization during inference: (6), among which, This is the dequantized floating-point value.

[0063] 2) Dynamic Range Adjustment of Activation Quantization: For activation values ​​(intermediate feature maps) during model inference, the quantization range is typically dynamically adjusted based on each batch of input data. Specific steps include: calculating the minimum activation value. and maximum value Determine the activation quantization ratio factor and Midnight: (7.1), (7.2); Activation value quantization formula: (8) During inference, use the dequantization formula to recover the activation value: (9).

[0064] 3) Quantization training: Direct application of quantization may lead to a significant decrease in model accuracy. Therefore, Quantization-aware training (QAT) is added during model training to adapt to the low-precision computing environment after quantization.

[0065] Core step: Inserting pseudo-quantization nodes: Simulating the quantization process during training, including quantization and dequantization operations. The pseudo-quantization formula is: (10) Add a regularization term for quantization error to the training loss: (11), where: It is the task loss (such as cross-entropy). It is the quantization error term. It is the weighting balance coefficient;

[0066] Training process: The model parameters are optimized through standard backpropagation, while the quantization parameters S and Z are adjusted so that the model can maintain high accuracy in the quantized environment.

[0067] 4) Advantages of quantization inference include: Reduced storage requirements: Quantization reduces model size by approximately 75%. For example, an FP32 model is 100MB, while an INT8 model requires only 25MB. Faster inference speed: On hardware supporting integer operations, the inference speed of the quantized model is 2-4 times faster than the original model. Reduced power consumption: Using low-precision integer calculations reduces power consumption by approximately 50% compared to floating-point calculations.

[0068] 5) Embedded Hardware Adaptation. This module uses the STM32Cube.AI toolchain to convert the quantized model into a format supported by embedded hardware. The quantized weights and activation values ​​are stored in on-chip memory (such as SRAM), and low-precision inference is performed using the embedded hardware's DSP or AI acceleration unit. The memory usage optimization formula for quantization is as follows: (12), among which, It is the ratio of floating-point precision to quantization precision (for example, the ratio of FP32 to INT8 is 4).

[0069] Module 3: Automatic Deployment Module. Again, we will use the deployment module for illustration. In this invention, the deployment process, including training, quantization, and evaluation, has been fully automated.

[0070] This invention provides an automated operation for easy deployment. The specific steps are as follows: Modify the various parameters in the YAML file. The YAML file mainly consists of the following parts: general, operation_mode, dataset, preprocessing, tools, deployment, mlflow, and hydra. 1) The general part provides information about the project and the path to the model to be deployed. 2) The operation_mode attribute is the operation mode and should be set to deployment. 3) The dataset part mainly contains the class_name attribute, which specifies the class of the training model. 4) The preprocessing part contains preprocessing information, which mainly includes the following attributes: a) resizing: The only option recently supported by C code. b) aspect_ratio: One of fit, crop, or pad. If cropped, the image size is adjusted by cropping the image to a square without aspect ratio distortion. If padded, black borders are added above and below the image to make it square; otherwise, a proper aspect ratio may not be preserved. c) color_mode: One of "grayscale", "rgb", or "bgr". 5) Tools and Deployment: To deploy the model on the STM32H747I-DISCO board, we will use STM32Cube.AI to convert the model into optimized C code, and use STM32CubeIDE to build the C application and flash it to the board. The main parts are as follows: `version`: Specifies the STM32Cube.AI version used for benchmarking the model. `optimization`: String, defines the optimizations used to generate the C model; options: "balanced", "time", "ram". `path_to_stm32ai`: The path to the locally downloaded stm32ai executable file; otherwise, it is False. `path_to_cubeIDE`: The path to the stm32cubeIDE executable file. `c_project_path`: The path to the stm32ai application code project. `IDE`: GCC, an option that only supports stm32ai application code. `verbsity`: 0 or 1. Mode 0 is silent; mode 1 displays messages when building and flashing the C application on the STM32 target. serie: This invention uses the STM32H7 series. Note that this option only supports STM32AI application code. board: Select the STM32 version to use. This invention uses STM32H747I-DISCO. 6) mlflow and hydra: These parts must always exist in the YAML configuration file.The `hydra` section can be used to specify the name of the directory where the experiment directories are stored and / or a pattern for naming the experiment directories. Using the YAML code below, an `experiment` directory is created each time Model Zoo is run, containing all directories and files created during the run. The names of the experiment directories are all unique because they are based on the date and time of the run.

[0071] Module 4: Data Processing Module. The data processing module is a crucial component of the image recognition system of this invention. It aims to preprocess and enhance the input data to ensure it meets the input requirements of the deep learning model, while simultaneously improving the model's robustness and recognition performance. Through multi-channel data preprocessing and enhancement techniques, this module effectively optimizes the input data stream, providing high-quality data input for subsequent inference processes, including:

[0072] 1) Data Preprocessing. Data preprocessing is a crucial step before model inference, used to format the raw input data into a form that the model can directly process. The preprocessing flow in this module includes the following aspects:

[0073] a) Image format conversion. Since the image data received by embedded devices may come from cameras, sensors, etc., its format may be inconsistent with the model's input requirements, thus requiring format conversion. For example, converting RGB images to grayscale or performing normalization: (13), where R, G, and B are the red, green, and blue channels of the image, It is the converted grayscale image;

[0074] b) Cropping and Scaling. Models typically require fixed-size input images, such as 224×224. This module resizes the input image to the model's input size using cropping and scaling operations: (14), of which: This refers to the scaling ratio; c) Normalization. Normalize the image pixel values, scaling them to the range required by the model (usually [0, 1] or [-1, 1]) to accelerate model convergence. (15), of which, The mean, Standard deviation;

[0075] 2) Data augmentation: Data augmentation techniques are used to generate diverse input samples during training and inference to improve the model's generalization ability and robustness;

[0076] The module supports the following enhancement methods:

[0077] a) Random flipping and rotation: During training, the image is randomly flipped horizontally or vertically, and randomly rotated by a certain angle. (16), among which, The rotation angle;

[0078] b) Brightness and contrast adjustment: Simulate input under different lighting conditions by randomly adjusting the brightness and contrast of the image. (17), among which, Contrast coefficient, This is the brightness offset.

[0079] 3) Data Transmission and Buffering Optimization: This module optimizes the transmission efficiency of the input data stream through efficient transmission and buffering mechanisms while preprocessing the data, reducing the impact of data loading on model inference. This includes:

[0080] a) DMA Channel Transfer. In embedded devices, when data is transferred from external storage devices (such as camera sensors) to memory, Direct Memory Access (DMA) technology is used to avoid processor intervention. Data transfer time is expressed as: (18), where: N is the number of data blocks to be transmitted, B is the size of each data block, and Bandwidth is the transmission bandwidth of DMA;

[0081] b) Data caching mechanism. A multi-level caching structure is set up in the data processing module to ensure that the preprocessed data can be accessed quickly during inference: Level 1 cache (SRAM): stores the current batch of data to reduce access latency; Level 2 cache (Flash): stores historical data or weights for loading during task scheduling.

[0082] Module 5: Inference Engine Module. The inference engine module is the core execution unit of this invention, responsible for loading and running the optimized deep learning model on the embedded device to achieve real-time, efficient image recognition. This module combines model optimization results with hardware characteristics to dynamically allocate computational tasks, rationally utilize hardware resources, and provide online inference support, including:

[0083] 1) Model Loading and Initialization. The inference engine module first loads the optimized model weights and structure, and initializes the inference runtime environment. This includes:

[0084] a) Model Loading. The model is stored in a quantized format in the embedded device's Flash or external memory. During the model loading phase, the inference engine loads the weights and topology into on-chip memory (SRAM): (19), of which: These are the weights stored in Flash. [s:e] represents the range of the currently loaded model weight block. It is the weight portion loaded into SRAM;

[0085] b) Initialization process. During initialization, the inference engine completes the following tasks: allocates memory areas for intermediate computation results; configures the operating parameters of hardware acceleration units (such as DSP and NPU); and verifies the integrity of the model topology to ensure hardware compatibility.

[0086] 2) Dynamic Task Allocation. To improve inference efficiency, the inference engine module dynamically allocates computational tasks to different hardware units, taking into account hardware resources and model characteristics.

[0087] a) Task allocation strategy. For dual-core embedded devices (such as STM32's Cortex-M7 and Cortex-M4), the inference engine adopts a main-slave core collaborative working mechanism: Main core (Cortex-M7): performs computationally intensive tasks (such as convolution and fully connected layer operations); Slave core (Cortex-M4): performs data preprocessing, task scheduling, and result transmission. The total inference time is expressed as: (20) By optimizing the allocation strategy, so that Achieve load balancing;

[0088] b) Calculate resource utilization. By analyzing the memory requirements and computation time of each layer of computation, the inference engine dynamically adjusts the task allocation strategy to maximize resource utilization: (twenty one);

[0089] 3) Efficient Inference Process. The execution process of the inference engine consists of multiple steps, employing a pipelined structure to parallelize computation and data transmission:

[0090] a) Input data loading. Preload the next batch of data via the DMA channel to reduce the waiting time between data loading and computation. (twenty two);

[0091] b) Computation Module Call. For core operations such as convolution and activation, the inference engine calls hardware accelerators (such as DSPs) to perform calculations. The calculation formula is as follows: (23), where: X is the input data, W is the weight matrix, B is the bias vector, and ReLU is the activation function;

[0092] c) Result output. After inference is complete, the inference engine stores the result in the output buffer and transmits it to an external device via a serial port (UART), I2C, or SPI interface.

[0093] A hardware acceleration method for image recognition in embedded devices, the main features of which include:

[0094] Dual-core collaborative processing: The main core is used to perform deep learning inference calculations, while the auxiliary core is used for data loading, preprocessing, and task distribution;

[0095] Hardware matrix acceleration: Perform convolution operations on the model using a dedicated hardware matrix multiplier on an embedded device;

[0096] DMA data transfer: Enables high-speed data transfer between memory and peripherals through Direct Memory Access (DMA) channels, reducing processor intervention;

[0097] Memory optimization management: By dividing the storage areas of SRAM and Flash, the memory resources required for model weights, activation values ​​and intermediate feature maps are dynamically allocated to improve computational efficiency.

[0098] Furthermore, the dual-core collaborative processing method mainly includes the following steps:

[0099] The inference computation task is divided into computation tasks handled by the main core and data tasks handled by the auxiliary core.

[0100] The auxiliary core provides preprocessed image data to the main core through a queuing mechanism;

[0101] After the main core completes the inference task, it returns the result to the auxiliary core through shared memory. The auxiliary core is responsible for the post-processing and output of the result.

[0102] This study employs optimization methods based on quantization and model compression to reduce the model's storage space and computational cost without significantly decreasing its accuracy. Quantization converts the floating-point model into a low-precision model, and compression algorithms reduce the number of parameters, enabling the deep learning model to run efficiently on resource-constrained embedded hardware platforms. The specific steps are as follows:

[0103] 1) Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: (24), where X represents the floating-point value to be quantized;

[0104] 2) Select quantization precision: Quantization precision is typically chosen as 8-bit or 16-bit. For 8-bit quantization, the integer range is usually set to [-128, 127]. For N-bit quantization, the integer range is: (25);

[0105] 3) Quantization Factor Calculation: Calculate the quantization factor (or scaling factor), which maps floating-point numbers to a specified integer range. The scaling factor is typically determined by the ratio of the maximum to the minimum value. (26), among which, and These are the maximum and minimum values ​​of the floating-point number, respectively. and It is the quantization range of integer representation;

[0106] 4) Quantization formula: Mapping floating-point numbers to integers using a scaling factor: (27), where x is a floating-point value, round is a rounding operation, and q(x) is a quantized integer value;

[0107] 5) Dequantization: During inference, it is necessary to map the quantized integer values ​​back to floating-point values. The dequantization formula is: (28), among which, This is the dequantized floating-point value. The insertion point includes the weights and activation values ​​for each layer, using the formula: (29), where s is the scaling factor. and It is the range of activation values.

[0108] To simplify the migration of deep learning models from the cloud to embedded devices, this invention provides an automated model deployment framework. During deployment, this framework automatically converts trained deep learning models into a format suitable for embedded hardware platforms and supports subsequent code generation and hardware adaptation using tools such as STM32Cube.AI, making deployment quick and easy. Throughout the project, a complete and user-friendly YAML file is provided to facilitate training, quantization, evaluation, and deployment operations.

[0109] Furthermore, the process of achieving automated deployment and adaptation mainly involves the following steps:

[0110] 1) Model Conversion and Optimization. Automated deployment first requires converting the trained model from the framework (this invention uses TensorFlow) into a format suitable for embedded platforms, which involves the following two parts:

[0111] a) Conversion from floating-point model to quantized model: Converting the floating-point model to a low-precision model (such as INT8, INT16) significantly reduces the model's storage and computational requirements. The quantization operation selects an appropriate precision based on the target platform's hardware capabilities;

[0112] b) Model Compression and Optimization: Pruning: Reducing model size and computational complexity by removing unimportant neural network connections. Knowledge Distillation: Reducing model complexity by learning knowledge from a larger model (teacher network) through a smaller student network. Matrix Factorization: Optimizing convolutional layers using mathematical methods such as low-rank factorization;

[0113] 2) Target Hardware Adaptation and Compilation: Different embedded platforms (such as STM32) have different hardware architectures, computing capabilities, and storage resources. Therefore, automated deployment requires adapting the model to the specific hardware and optimizing the use of computing and storage resources. This includes:

[0114] a) Hardware characteristic identification: During the automated deployment process, the system needs to automatically detect the characteristics of the target hardware device (such as computing power, storage, cache size, etc.) in order to select appropriate model conversion and optimization strategies for the target hardware;

[0115] b) Hardware acceleration support: Embedded devices such as STM32 typically integrate dedicated hardware acceleration units (such as DSPs and AI accelerators). Automated deployment tools need to support compatibility between the optimized model and the hardware acceleration unit in order to fully utilize the hardware's computing power;

[0116] c) Code Generation and Compilation: Automated deployment tools (such as STM32Cube.AI) convert the model into C language code or other embedded-supported formats. The compiler generates binary files that can be directly executed on the hardware based on the target hardware characteristics;

[0117] d) Memory and storage management: Embedded devices typically have limited memory and storage resources, so it is necessary to dynamically adjust memory allocation strategies through automated deployment tools to ensure that the model can run within the limited memory and storage space;

[0118] 3) Automated Deployment and Debugging: This includes model conversion and compilation, as well as automatically loading the final model into the target device. The specific implementation method is as follows: the compiled model is deployed to the embedded device via a YAML file. For microcontrollers like STM32, firmware loading is performed using an ST-Link debugger and serial port. The file contains links to the B-CAMS-OMV camera, and the hardware is connected via a flexible flat cable.

[0119] The actual operation of this invention mainly involves the following steps: training, quantification, evaluation, benchmarking, prediction, and deployment.

[0120] Training: Prepare a suitable dataset. The dataset's directory tree is as follows:

[0121] dataset_root_directory /

[0122] class_a /

[0123] a_image_1.jpg

[0124] a_image_2.jpg

[0125] class_b /

[0126] b_image_1.jpg

[0127] b_image_2.jpg

[0128] Configure your relevant YAML files. For detailed instructions, please refer to the automatic deployment module in the invention description.

[0129] In the parent folder src directory, activate the virtual environment and run the code:

[0130] python stm32ai_main.py --config-path . / config_file_examples / --config-name training_config.yaml

[0131] After running, the trained h5 model can be found in the corresponding experiments_outputs / folder. All training and evaluation artifacts are stored in the current output simulation directory outputs / {run_time}, and can be visualized using the general command tensorboard --logdir logs.

[0132] Quantization: Prepare the dataset for quantization. This can be the complete training set or a specific training set used for activation calibration. If you only want to quantize the model on a random portion of the quantization set, simply set the percentage value in the `quantization_split` parameter. Configure the relevant YAML files accordingly.

[0133] Run the command in the corresponding src file to perform the quantization operation:

[0134] python stm32ai_main.py --config-path . / config_file_examples / --config-name chain_eqe_config.yaml

[0135] The operation mode for the rest is basically similar. First, modify the YAML file, then obtain the corresponding results from the corresponding folder, and finally deploy the model that has undergone training, quantization, evaluation and other operations to the hardware to achieve the corresponding functions.

[0136] It is worth noting that in the process of performing the above operations, we must first connect the STM microcontroller we are using to the camera module.

[0137] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A high-efficiency image recognition system based on embedded edge devices, characterized in that, include: Hardware acceleration module, including hardware-accelerated convolution operations; By using dual-core collaboration, inference tasks are assigned to different cores, with the main core responsible for inference computation of the deep learning model and the auxiliary core responsible for input data preprocessing, result transmission and task allocation. High-speed data transfer between memory and peripherals is achieved through direct memory access channels; In the hardware acceleration module, the block cache stores the input data, weights, and intermediate computation results in blocks in the on-chip SRAM to avoid frequent access to off-chip memory. During pipelined processing, when computing the current convolutional layer, the DMA channel preloads the weights and input data for the next layer. The pipeline efficiency of the data flow is expressed as follows: This module optimizes the pipeline design to bring η close to 1; In the model quantization module, quantization reduces the model's storage requirements and computational complexity by converting the weights and activations in the deep learning model from high-precision floating-point representation to low-precision integers. The model quantization module includes the following steps: 1) The quantification process is as follows: 1.1) Calculate the quantization scaling factor k, as shown in the following formula: Where g is the weight tensor, N is the quantization bit depth, and max(g) and min(g) are the maximum and minimum values ​​of the weight, respectively. 1.2) Map the weights from floating-point numbers to integers, as shown in the formula: Among them, Q w is the quantized integer value, z is the zero offset used to align the range of quantized values, g is the weight tensor, and k is the quantization scaling factor; 1.3) Inverse quantization during inference, as shown in the following formula: in, Q is the dequantized floating-point value, k is the quantization scaling factor, and Q is the dequantized floating-point value. w It is the quantized integer value, and Z is the zero offset; 2) Dynamic range adjustment of activation quantization: For activation values ​​during model inference, the quantization range is dynamically adjusted based on each batch of input data, including: 2.1) Calculate the minimum value min(A) and the maximum value max(A) of the activation; 2.2) Determine the activation quantization ratio factor S A and Z A Zero point, where N is the number of quantization bits: 2.3) Activation value quantization is shown in the following formula: During inference, the activation value is recovered using the dequantization formula: 3) Quantization training incorporates quantization-aware training techniques during model training to adapt to the low-precision computing environment after quantization, including: Inserting pseudo-quantization nodes: This simulates the quantization process during training, including quantization and dequantization operations. The pseudo-quantization is shown in the following equation, where k is the quantization scaling factor: Add a regularization term for quantization error to the training loss: L total =L task +λ*L quantization (11) Where: L task It's a mission loss, L quantization This is the quantization error term, and λ is the weighting balance coefficient. Training process: The model parameters are optimized through standard backpropagation, and the quantization parameter S is adjusted to ensure that the model can maintain high accuracy in the quantized environment. 4) This module uses the STM32Cube.AI toolchain to convert the quantized model into a format supported by embedded hardware. The quantized weights and activation values ​​are stored in on-chip memory, and low-precision inference is performed using the embedded hardware's DSP or AI acceleration unit. The memory usage optimization formula for quantization is as follows: Where PrecisionFactor is the ratio of floating-point precision to quantization precision, and Memory... original This refers to the original storage space occupied when the model is not quantized; By using quantization techniques, floating-point models are converted into low-precision models, and compression algorithms are employed to reduce the number of parameters, enabling deep learning models to run efficiently on resource-constrained embedded hardware platforms. The specific steps are as follows: Step 1, Determine the quantization interval: Use the maximum and minimum values ​​to define the quantization interval: my quant =min(x),max quant =max(x) (24), Where x represents the floating-point value to be quantized; Step 2, select the quantization precision: N is the number of quantization bits, and its integer range is: range=2 (N-1) -1 (25), Step 3, Quantization scaling factor calculation: The quantization scaling factor is used to map floating-point numbers to a specified integer range. The quantization scaling factor is determined by the ratio of the maximum and minimum values. Where, max float and min float These are the maximum and minimum values ​​of a floating-point number, respectively. quant and min quant It is the quantization range of integer representation; Step 4, Quantization Formula: Map floating-point numbers to integers using a scaling factor: Where x is a floating-point value, round is the rounding operation, q(x) is the quantized integer value, and k is the quantization scale factor; Step 5, Dequantization: During inference, the quantized integer values ​​need to be mapped back to floating-point values. The dequantization formula is: in, It is the dequantized floating-point value, t is the scaling factor, and the insertion point includes the weights and activation values ​​of each layer, using the formula: Where t is the scaling factor, x min and x max These are the allowed numerical boundaries during the quantization / dequantization process; The automatic deployment module automatically converts the trained deep learning model into a format suitable for embedded hardware platforms and supports subsequent code generation and hardware adaptation. The data processing module preprocesses and enhances the input data to ensure that the data meets the input requirements of the deep learning model. The inference engine module is responsible for loading and running optimized deep learning models on embedded devices; The inference engine module includes: 1) Model Loading and Initialization: The inference engine module first loads the optimized model weights and structure, and initializes the inference runtime environment, including: 1.1) Model Loading: The model is stored in the embedded device's Flash or external memory in a quantized format. During the model loading phase, the inference engine loads the weights and topology into on-chip memory. IN RAM =In Flase[s:e] (19), Among them: W Flase These are the weights stored in Flash. [s:e] represents the range of the currently loaded model weight block. W RAM It is the weight portion loaded into SRAM; 1.2) Initialization process: During initialization, the inference engine completes the following tasks: allocating memory areas for intermediate calculation results, configuring the running parameters of the hardware acceleration unit, and verifying the integrity of the model topology. 2) The inference engine module dynamically allocates computational tasks to different hardware units based on hardware resources and model characteristics, including: 2.1) Task Allocation Strategy: For dual-core embedded devices, the inference engine adopts a main-slave core collaborative working mechanism, where the total inference time is expressed as: T total =max(T core1 ,T core2 ) (20), T core1 ,T core2 These are the two core inference times. By optimizing the allocation strategy, T... core1 ≈T core2 Achieve load balancing; 3) Efficient inference process: The execution process of the inference engine consists of multiple steps, employing a pipelined structure to achieve parallelization of computation and data transmission, including: 3.1) Input data loading: Preload the next batch of data via the DMA channel to reduce the waiting time between data loading and computation. Where DataSize represents the amount of data that needs to be transferred via DMA, and DMABandwidth represents the maximum speed at which the DMA channel can transfer data. 3.2) Calculation module call: For core operations, the inference engine calls the hardware accelerator to perform calculations. The calculation formula is as follows: Y = ReLU(X*M+B) (23), Where X is the input data, M is the weight matrix, B is the bias vector, and ReLU is the activation function. After inference is completed, the inference engine stores the result in the output buffer and transmits it to an external device via serial port, I2C or SPI interface.

2. The high-efficiency image recognition system based on embedded edge devices according to claim 1, characterized in that, In the hardware acceleration module, the convolution operation is shown in the following formula, which is in matrix multiplication form: Y = X * W + B (1), Where X is the input feature map of size H×D, W is the convolution kernel of size K×K, B is the bias term, and Y is the output feature map. In embedded devices, matrix multiplication is performed through a hardware acceleration unit, combined with DMA channels to preload weights and input data.

3. The high-efficiency image recognition system based on embedded edge devices according to claim 1, characterized in that, In the hardware acceleration module, the total execution time of the main and auxiliary cores working together is expressed by the following formula: T total =max(T core1 +T core2 ) (2), Among them, T core1 and T core2 These represent the execution times of the main core and the auxiliary core, respectively. By optimizing the task allocation strategy, T... core1 and T core2 Achieve load balancing.

4. The high-efficiency image recognition system based on embedded edge devices according to claim 1, characterized in that, The data processing module includes: Data preprocessing; data augmentation; and data transfer and buffering optimization, wherein data transfer and buffering optimization includes: 1) DMA Channel Transfer: In embedded devices, when data is transferred from external storage devices to memory, Direct Memory Access (DMA) technology is used to avoid processor intervention. The data transfer time is expressed as: Where: N is the number of data blocks to be transmitted, B is the size of each data block, and Bandwidth is the DMA transmission bandwidth; 2) Data Buffering Mechanism: A multi-level cache structure is set up in the data processing module to ensure that the preprocessed data can be accessed quickly during inference. The first-level cache stores the current batch of data, and the second-level cache stores historical data or weights.

5. The high-efficiency image recognition system based on embedded edge devices according to claim 1, characterized in that, The dual-core collaborative processing steps include dividing the inference computation task into a computation task handled by the main core and a data task handled by the auxiliary core. The auxiliary core provides preprocessed image data to the main core through a queue mechanism. After the main core completes the inference task, it returns the result to the auxiliary core through shared memory. The auxiliary core is responsible for the post-processing and output of the result.

6. The high-efficiency image recognition system based on embedded edge devices according to claim 1, characterized in that, The process of automating deployment and adaptation includes: 1) Model conversion and optimization: Automated deployment first requires converting the trained model from the framework into a format suitable for embedded platforms, including: 1.1) Conversion from floating-point model to quantization model: Convert the floating-point model to a low-precision model, and select the appropriate precision for the quantization operation based on the hardware capabilities of the target platform; 1.2) Model compression and optimization: Pruning, by removing unimportant neural network connections; Knowledge distillation, by learning the knowledge of a large model through a smaller student network; Matrix factorization, using mathematical methods to optimize convolutional layers; 2) Target Hardware Adaptation and Compilation: Different embedded platforms have different hardware architectures, computing capabilities, and storage resources, including: 2.1) Hardware characteristic identification: During the automated deployment process, the system needs to automatically detect the characteristics of the target hardware device; 2.2) Hardware acceleration support: Embedded devices integrate dedicated hardware acceleration units, and automated deployment tools need to support compatibility between the optimized model and the hardware acceleration unit; 2.3) Code generation and compilation: The automated deployment tool converts the model into C language code or other embedded supported formats, and the compiler generates binary files that can be executed directly on the hardware according to the characteristics of the target hardware; 2.4) Memory and storage management: Embedded devices have limited memory and storage resources, therefore, it is necessary to dynamically adjust memory allocation strategies through automated deployment tools; 3) Automated deployment and debugging: This includes model conversion and compilation, as well as automatically loading the final model onto the target device, including: The compiled model is deployed to the embedded device via a YAML file. For microcontrollers like STM32, the firmware is loaded using the ST-Link debugger and serial port. The file contains the relevant links for the B-CAMS-OMV camera, and the hardware is connected via a flexible flat cable.

Images