Embedded people counting method and system based on dual-model architecture

Abstract

The present invention discloses an embedded people counting method and system based on a dual-model architecture, wherein the embedded people counting method based on the dual-model architecture comprises the following steps: S1, construction of a first model and a second model, S2, person detection using the first model, and S3, person counting using the second model; the present invention implements low-power, low-cost embedded system edge computing by designing a lightweight model and deploying it in an ARM architecture MCU. The system completes data processing and calculation locally, avoiding data uploading to a cloud server, thereby effectively improving data security; in addition, the present invention adopts a dual-model construction, so that the system uses the first model for person detection in an unmanned state, realizes rapid reasoning, significantly reduces the ineffective consumption of computing resources, and further reduces overall power consumption.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to an embedded people counting method and system based on a dual-model architecture. Background Art

[0002] With the development of society, security management and crowd control in public places are becoming increasingly important. In crowded places like shopping malls, train stations, and airports, accurate and real-time headcounting is crucial for security monitoring, resource allocation, and data analysis. Traditional counting methods fall into two main categories: hardware-based and computer vision-based.

[0003] Hardware-based methods: These methods primarily use infrared sensors, ultrasonic sensors, and other devices to count people. These methods are complex to install and maintain and are susceptible to environmental factors, resulting in lower accuracy.

[0004] Computer vision-based methods use cameras to capture video images and perform crowd counting using image processing techniques. This approach offers the following advantages: contactless operation, easy installation, and adaptability to various scenarios. However, this method relies on high-performance computing equipment, which poses challenges in data security, high costs, and high power consumption. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an embedded people counting method and system based on a dual-model architecture, which can effectively overcome the limitations of traditional hardware equipment methods and at the same time solve the problems of data security, cost and power consumption based on computer vision methods, providing an efficient, reliable and economical solution for counting people in public places.

[0006] To solve the above technical problems, the technical solution of the present invention is: an embedded people counting method based on a dual-model architecture, comprising the following steps:

[0007] S1. Construction of the first model and the second model:

[0008] S11. Use the COCO2017 dataset to extract portrait images to create a dataset;

[0009] S12. Using the dataset to train a first model and a second model using TensorFlow, where the first model uses the MobileNet V1 model and the second model uses the improved Tiny YOLO V2 model;

[0010] S13, deploying the first model and the second model in an ARM architecture microcontroller unit;

[0011] S2. Person detection of the first model:

[0012] S21, receiving the collected current image information;

[0013] S22, pre-processing the collected image information and entering the first model;

[0014] S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information;

[0015] S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and the process proceeds to step S3;

[0016] S3. People counting of the second model:

[0017] S31, receiving the collected current image information;

[0018] S32, pre-processing the collected image information and entering the second model;

[0019] S33, reasoning of the second model: the second model uses the improved Tiny YOLO V2 model to output graphic information;

[0020] S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

[0021] As a preferred technical solution, both the first model and the second model adopt externally adjustable hyperparameters, the hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression.

[0022] As an optimal technical solution, both the first model and the second model use full integer quantization to reduce model size and computational complexity.

[0023] As a preferred technical solution, the deployment of the first model and the second model both adopts model conversion technology, using CMSIS-NN to convert the model into a format that can run on embedded devices.

[0024] As a preferred technical solution, the image information preprocessing includes image format processing and image compression.

[0025] As a preferred technical solution, the second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure:

[0026] The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'padding same' strategy and no bias; a batch normalization layer; and a ReLU activation function.

[0027] The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0028] The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0029] The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0030] The first depthwise separable convolutional block includes: a depthwise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0031] The second depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'padding same' strategy.

[0032] The third depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 128 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0033] The fourth depthwise separable convolutional block includes: a depthwise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0034] The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, a 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

[0035] Another preferred technical solution is an embedded people counting system based on a dual-model architecture, including: a data acquisition module, which uses a camera to capture real-time image information; a data preprocessing module, which performs image format conversion and image compression on the captured image information so that the image information adapts to the input standard of the model; a model inference module, which uses a microcontroller unit to run the first model or the second model; a data post-processing module, which dequantizes the output of the first model to determine the presence of people, and decodes and performs non-maximum suppression on the output of the second model to obtain number of people information.

[0036] As an optimal technical solution, it also includes: an information display module, which displays the original image, processed image information and the current hyperparameter information of the system through the screen; a hyperparameter change module, which triggers external interrupts by pressing buttons to modify the system hyperparameters.

[0037] Due to the adoption of the above technical solution, the beneficial effects of the present invention are as follows: the present invention realizes low-power, low-cost embedded system edge computing by designing a lightweight model and deploying it in an ARM architecture MCU. The system completes data processing and calculation locally, avoiding uploading data to the cloud server, thereby effectively improving data security; in addition, the present invention adopts a dual-model construction, so that the system uses the first model to perform person detection in an unmanned state, realizes rapid reasoning, significantly reduces the ineffective consumption of computing resources, and further reduces overall power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following drawings are intended only to illustrate and explain the present invention, and are not intended to limit the scope of the present invention.

[0039] Figure 1 is a flow chart of a method according to an embodiment of the present invention; DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and examples. In the following detailed description, certain exemplary embodiments of the present invention are described by way of illustration only. It is understood that those skilled in the art will recognize that the described embodiments may be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the drawings and description are illustrative in nature and are not intended to limit the scope of the claims.

[0041] like Figure 1 As shown, an embedded people counting method based on a dual-model architecture includes the following steps:

[0042] S1. Construction of the first and second models. This step serves as the basis for model operation.

[0043] S11. Use the COCO2017 dataset to extract portrait images to create a dataset;

[0044] S12. Using the dataset to train a first model and a second model using TensorFlow, where the first model uses the MobileNet V1 model and the second model uses the improved Tiny YOLO V2 model;

[0045] S13, deploying the first model and the second model in an ARM architecture microcontroller unit;

[0046] S2. Person detection of the first model

[0047] S21, receiving the collected current image information;

[0048] S22, pre-processing the collected image information to meet the model input requirements and enter the first model;

[0049] S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information;

[0050] S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and the process proceeds to step S3;

[0051] S3. People counting of the second model

[0052] S31, receiving the collected current image information;

[0053] S32, pre-processing the collected image information to meet the model input requirements and enter the second model;

[0054] S33, reasoning of the second model: the second model uses the improved Tiny YOLO V2 model to output graphic information;

[0055] S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

[0056] The COCO2017 dataset is used for training. The COCO dataset is a large database widely used for object detection, image segmentation, and image annotation. It belongs to the existing technology and will not be described in detail here.

[0057] Preferably, the steps of preparing the data set in step S11 are specifically as follows:

[0058] S111: Open and read the COCO annotation file and load its content into the variable coco_data;

[0059] S112: traverse the images list in coco_data, and for each image, initialize a Boolean variable copy_image to mark whether the current image needs to be copied, construct the image file name image_file_name, and construct the corresponding label file name label_file_name;

[0060] S113: For each picture, traverse the annotations list and check whether the annotation belongs to the current picture through image_id matching. If so, obtain the category_id of the annotation;

[0061] S114: Find the corresponding category name class_name in the categories list. If class_name is in the provided classes list, set copy_image to True, indicating that the image needs to be copied.

[0062] S115: Get the index class_id of the category and get the bounding box (bbox) information from the annotation. The bounding box format of COCO is [x, y, width, height], and the YOLO format is [class_id, x_center, y_center, width, height], where x_center and y_center are the normalized center coordinates relative to the width and height of the image. The bounding box coordinates need to be converted to YOLO format and the result is saved in the corresponding label file.

[0063] S116: If copy_image is True, copy the image from coco_images_dir to export_folder.

[0064] For the second model, the training mainly involves filtering out human annotation information in the COCO dataset and converting the COCO format to the YOLO format. For the training of the first model, it is necessary to construct a Visual WakeWords dataset from the COCO dataset. The specific steps are as follows: For each picture in the dataset, a label of 1 or 0 is assigned based on whether there is an object of interest (such as a person). If there is at least one bounding box in the picture corresponding to the object of interest and the area of ​​the bounding box exceeds a certain threshold (for example, 0.5% of the image area), the picture is marked as 1; if these conditions are not met, it is marked as 0.

[0065] In the Tiny YOLO V2 model, each grid cell is responsible for detecting objects in an image and outputting a bounding box for each object. The specific location and size of these bounding boxes are determined by the parameters predicted by the corresponding grid cell. Prior boxes are a set of bounding boxes defined before training that help the network learn the location and size of objects.

[0066] In Tiny YOLO V2, bounding boxes are derived through cluster analysis of the training dataset. The K-means clustering algorithm is applied to cluster the bounding boxes in the training data using the distance formula: d = 1 – IOU, where IOU stands for intersection over union. The K-means algorithm found the five optimal cluster centers, or bounding boxes, as shown in the table below.

[0067] Table 1 Prior box parameters

[0068]

[0069] Preferably, the parameters for training the first model in step S12 include three categories: image preprocessing, image augmentation, and training hyperparameters. Image preprocessing and image augmentation process the data set, while training parameters are specific to the model and affect the recognition effect of the model. The training parameters of the first model are as follows:

[0070] A. Image Preprocessing

[0071] S121: Scale the image to 1 / 127.5 and offset it to -1.

[0072] S122: using a nearest neighbor interpolation method to ensure that the adjusted image conforms to the aspect ratio of the original image;

[0073] B. Image Augmentation

[0074] S123: random contrast value is 0.4, random brightness value is 0.05;

[0075] S124: horizontal random inversion;

[0076] S125: Random translation with width and height adjustment factors of 0.25, using reflection fill mode and nearest neighbor interpolation;

[0077] S126: random rotation factor of 0.125, reflection fill mode and nearest neighbor interpolation;

[0078] S127: Randomly scale width and height by a factor of 0.25, using reflection fill mode and nearest neighbor interpolation;

[0079] S128: random distortion factor of 0.0515, using wrap-around fill mode and nearest neighbor interpolation;

[0080] C. Training Hyperparameters

[0081] S129: Use the MobileNet V1 model with an alpha parameter of 0.25 and an input shape of (96,96,3);

[0082] S1210: Pre-trained weights use ImageNet pre-trained weights;

[0083] S1211: Dropout layer dropout rate is 0.3;

[0084] S1212: batch size is 64 samples;

[0085] S1213: training for a total of 1000 cycles;

[0086] S1214: Use Adam optimizer with a learning rate of 0.001;

[0087] S1215: Use the ReduceLROnPlateau callback function to reduce the learning rate when the validation set accuracy stops improving. The learning rate decay factor is 0.5. The number of tolerance cycles is 40, and the lower limit of the learning rate is 1.0e-05.

[0088] S1216: Use the EarlyStopping callback function to stop training and restore the optimal weights when the validation set accuracy stops improving, with a tolerance of 60 cycles.

[0089] Preferably, the parameters for training the second model in step S12 include four categories: image preprocessing, image augmentation, training hyperparameters, and post-processing parameters. Image preprocessing, image augmentation, and training hyperparameters have the same functions as those in the first model. Post-processing parameters do not participate in model training and are only called during model evaluation, such as when calculating mAP. The training parameters of the second model are as follows:

[0090] A. Image preprocessing:

[0091] S1217: Scale the image to 1 / 255 with no offset.

[0092] S1218: Using the nearest neighbor interpolation method to ensure that the adjusted image conforms to the aspect ratio of the original image;

[0093] B. Image augmentation

[0094] S1219: The random rotation angle is set to 5 degrees;

[0095] S1220: Translation is set to 0.1;

[0096] S1221: Horizontal flip is set to 1.0;

[0097] S1222: Gaussian blur set to 3.0;

[0098] S1223: Linear contrast set to a random value between 0.75 and 1.5;

[0099] C. Training hyperparameters:

[0100] S1224: Batch size is set to 64;

[0101] S1225: The number of training rounds is set to 1000;

[0102] S1226: Use the Adam optimizer with a learning rate of 0.01.

[0103] S1227: Use the ReduceLROnPlateau callback function, with a learning rate decay factor of 0.05, a tolerance of 50 cycles, and a learning rate lower limit of 1.0e-06;

[0104] S1228: Use the EarlyStopping callback function with a tolerance of 80 cycles;

[0105] D. Post-processing parameters:

[0106] S1229: The confidence threshold is set to 0.001;

[0107] S1230: The non-maximum suppression (NMS) threshold is set to 0.5;

[0108] S1231: The IoU evaluation threshold is set to 0.4;

[0109] S1232: The maximum number of detection frames is set to 980.

[0110] Preferably, the first model in step S12 adopts the MobileNet V1 model. In order to deploy the deep learning network model on the ARM architecture microcontroller unit (MCU), the algorithm needs to be optimized. The Visual Wake Word model based on the Resnet-14 network is an open source model and belongs to the prior art and will not be described here. The ResNet-14 deep residual network solves the gradient vanishing and gradient explosion problems in deep network training by introducing residual blocks, but the traditional convolution operation has a large amount of calculation, and the use of residual blocks also increases the computational burden, resulting in an increase in weight information. Taking into account the real-time and lightweight requirements of embedded systems, the Visual Wake Word algorithm is optimized, and the MobileNet V1 model is used to replace the ResNet-14 network model. See Table 2 for the MobileNet V1 model. The MobileNet V1 model utilizes the design of depthwise separable convolution, which significantly reduces the number of model parameters and computational requirements, making the model more lightweight and more suitable for mobile devices and embedded systems with limited resources.

[0111] Table 2 MobileNet V1 model network structure

[0112]

[0113]

[0114] The first model uses the MobileNet V1 architecture, which is a lightweight network model that replaces ordinary convolution with depthwise separable convolution. Compared with ordinary convolution, it has fewer parameters and computational complexity.

[0115] Preferably, the second model in step S12 adopts an improved Tiny YOLO V2 model. Tiny YOLO V2 is a simplified, lightweight variant of the YOLO target detection algorithm, and its network structure consists of only 9 convolutional layers. Tiny YOLOV2 is a simplified, lightweight variant of the YOLO target detection algorithm, and its network structure consists of only 9 convolutional layers. The purpose of this design is to maintain relatively good detection performance in a resource-constrained environment, while significantly reducing the complexity and number of parameters of the network to adapt to embedded devices or platforms with weak computing power. Thanks to this design, Tiny YOLO V2 can achieve real-time target detection under these conditions, opening up broader application prospects for object recognition applications. The specific structure of its network model is shown in Table 3.

[0116] Table 3 Tiny YOLO V2 network structure

[0117]

[0118] In order to improve the performance of the Tiny YOLO V2 model, the second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure, see Table 4 for synchronization:

[0119] The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'padding same' strategy and no bias; a batch normalization layer; and a ReLU activation function.

[0120] The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0121] The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0122] The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0123] The first depthwise separable convolutional block includes: a depthwise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0124] The second depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'padding same' strategy.

[0125] The third depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 128 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0126] The fourth depthwise separable convolutional block includes: a depthwise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0127] The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, a 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

[0128] The length of the detection vector is determined according to the number of object categories and the number of bounding box parameters in the actual application scenario. In this embodiment, the length of the detection vector is set to 30.

[0129] Table 4 Improved Tiny YOLO V2 model backbone network structure

[0130]

[0131] The second model uses an improved Tiny YOLO V2 architecture, designated Tiny YOLO V2 LC. It first uses four convolutional layers with 3×3 kernels and a stride of 2 to gradually reduce the size of the feature map, primarily extracting features at different scales and better capturing target information in the subsequent object detection task. This is followed by four depthwise separable convolutional layers with 3×3 kernels and a stride of 3. Each convolutional layer is followed by BatchNormalization and ReLU activation functions. By flexibly utilizing depthwise separable convolutions and reducing the number of parameters, the design of a low-power embedded people detection system based on the ARM architecture ensures a more lightweight model while maintaining high object detection accuracy.

[0132] In the present invention, the first model performs person presence detection, which is essentially an image classification model, and the model architecture uses depthwise separable convolution in its entirety; the second model performs person number counting, which is essentially a target detection model, and in the early stage of inference, ordinary convolution is still used to better extract features; since the second model only partially uses depthwise separable convolution, and the target detection task is more complex than the image classification task, compared with the first model, the first model has smaller computational complexity and parameter quantity, faster inference speed, and occupies less resources.

[0133] Preferably, the deployment of step S13 includes:

[0134] S131: Convolution and depth-wise separable convolution are built using the NNFunctions API;

[0135] S132: The ReLU activation function is built using the NNSupportFunctions API.

[0136] To port the deep model written in Python to C language so that it can run on the ARM core, the deployment of the first and second models both adopts model conversion technology, using CMSIS-NN to convert the model into a format that can run on embedded devices. This is a well-known method in the prior art and will not be repeated here.

[0137] Preferably, both the first and second models use full integer quantization. Full integer quantization replaces the common 32-bit or 64-bit floating-point number of network parameters with a smaller number of bits to reduce the space occupied by model parameter storage, ultimately achieving the purpose of model compression. Full integer quantization uses TensorFlow Lite to convert both weights and activations into integers, including the following steps:

[0138] Calibration: Run the model and record the range of activations to determine the appropriate scaling factors and zero point;

[0139] Quantize weights: Use the calibrated parameters to convert weights from floating point numbers to integers;

[0140] Quantize activations: Convert activations from floating point numbers to integers using the calibrated parameters.

[0141] Applied optimizations: Folding batch normalization, combining weights and biases, and fusing the convolutional layer and the subsequent ReLU activation layer into a single convolution operation.

[0142] Preferably, both the first model and the second model use externally adjustable hyperparameters. The hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression. The adjustable hyperparameters are set as variables and assigned initial values, so that the system can adjust the hyperparameter values ​​by modifying the variables according to specific needs. It has higher flexibility and applicability, and can be adjusted and optimized according to the needs of different scenarios.

[0143] Preferably, the image information is in RGB565 image format, and image information preprocessing includes image format processing and image compression.

[0144] The image format processing is as follows: the data format collected by the camera OV2640 is RGB565 or RGB16, but the input data format of the low-power embedded people detection system 18 based on the ARM architecture designed deep learning model in this embodiment is RGB888 or RGB24, so a conversion program needs to be designed. The whole program can be described as follows:

[0145] (1) Extract the channel data in RGB565.

[0146] (2) Expand the 5-bit red data to 8 bits by shifting it left by 3 bits.

[0147] (3) Expand the 6-bit green data to 8 bits by shifting it left by 2 bits and filling the lowest two bits with 0.

[0148] (4) Expand the 5-bit blue data to 8 bits by shifting it left by 3 bits.

[0149] (5) Combine the expanded red, green, and blue data into pixel values ​​in RGB888 format.

[0150] (6) Repeat the above steps to traverse each pixel value.

[0151] Image compression: In people counting systems, whether using Visua Wake Word or Tiny YOLO V2LC, the input image must be compressed into a specific shape for neural network processing. This is done using the nearest neighbor interpolation (NNI) algorithm, a KD-tree-based interpolation algorithm often used for image processing and other problems.

[0152] The advantages of the nearest neighbor interpolation method are that it is simple to implement and has a fast calculation speed. It is particularly suitable for embedded systems with limited resources or when processing speed is more important than image quality.

[0153] Furthermore, the data type obtained by the OV2640 camera is Uint16, while the data required by Visual Wake Word is int8, and the data required by TinyYolo V2 LC is also Uint8, so data type conversion is required. Converting from Uint16 to Uint8 is relatively simple. During image format conversion, the color channels of each pixel are expanded from 16 bits (5 bits of red, 6 bits of green, and 5 bits of blue) to 24 bits (8 bits of red, 8 bits of green, and 8 bits of blue). In this case, a Uint16 array storing RGB888 format image data is already allocated as Uint8 for each channel. Simply convert the array to Uint8 and increase the array length accordingly. Converting from Uint16 to int8 requires an additional mapping step. Because the range of int8 is -127 to 127, a 128-bit offset from the original data is sufficient.

[0154] Preferably, in step S24, the final data type of the Visual Wake Word model is int8. In order to express the probability of the current category, these integer data need to be dequantized into floating-point numbers. The presence detection is based on the output of the first model, and the output of the first model is dequantized. The dequantized output result is used as the class confidence. When the class confidence is greater than the class confidence threshold, it is determined that a person exists. When the presence of a person is detected, the second model is activated. When the class confidence is less than the class confidence threshold, it is determined that no one exists, and step S2 is repeated.

[0155] Among them, the inverse quantization adopts the formula:

[0156] real_value_output=(q_output-(-128))*0.003906250;

[0157] real_value_output refers to the dequantized output result, and q_output refers to the model output.

[0158] After extracting the output of the Tiny YOLO V2 LC model, the output data must be decoded to extract the required information. The decoding method is prior art and will not be described here.

[0159] The core idea of ​​the non-maximum suppression (NMS) algorithm, also known as Non-Maximum Suppression, is to find local maxima while suppressing non-maximum elements. During target detection, many overlapping candidate bounding boxes may be generated for the same target. In order to screen the optimal target bounding box and eliminate redundant boxes, non-maximum suppression technology is used. The implementation of this algorithm relies on the intersection over union (IOU), which measures the degree of overlap between anchor boxes. Based on the scores of the anchor boxes, redundant anchor boxes with low scores and high overlap with high-scoring anchor boxes can be removed in turn. By simply setting an appropriate IOU threshold, accurate anchor box prediction results can be obtained.

[0160] Preferably, in step S34, the number of people is determined by the number of anchor frames, and the number of anchor frames is determined by the anchor frame score threshold and non-maximum suppression screening after model detection; the output of the second model is decoded and non-maximum suppression is used, and when it is detected that the person disappears, the first model is activated.

[0161] An embedded people counting system based on a dual-model architecture, comprising:

[0162] Data acquisition module, using a camera to capture real-time image information;

[0163] The data preprocessing module performs image format conversion and image compression on the captured image information to make the image information adapt to the input standard of the model;

[0164] The model inference module uses the microcontroller unit to run the first model or the second model; the data post-processing module dequantizes the output of the first model to determine the presence of a person, and decodes and performs non-maximum suppression on the output of the second model to obtain number of people information.

[0165] Information display module, which displays the original image, processed image information and the current hyperparameter information of the system on the screen;

[0166] The hyperparameter change module triggers external interrupts by pressing buttons to modify the system hyperparameters.

[0167] In order to verify the credibility and effectiveness of the above-mentioned embedded people counting method and system based on the dual-model architecture, an experimental process was designed for verification:

[0168] To verify the effect of the first model in the present invention, the Visual Wake Word model based on the Resnet-14 network was selected for comparative verification. This model uses the same dataset and has the same function as the first model in the present invention. The dataset adopts the COCO2017 dataset, and Accuracy, MACC, Activation RAM, and Weights FLASH are used as performance evaluation indicators. Accuracy represents the accuracy of the model. MACC refers to the multiplication and accumulation operations in the neural network, indicating the model complexity and computing resource consumption. Activation RAM represents the memory size occupied during model inference. Weights FLASH represents the memory size occupied by the model weights.

[0169] Table 5 Performance comparison between the first model and the Visual Wake Word model

[0170]

[0171] The data in Table 5 demonstrates that the first model of our invention significantly outperforms the VisualWake Word model in both MACC and Activation RAM, making it more suitable for deploying inference on resource-constrained embedded platforms. To validate the effectiveness of the second model of our invention, we selected the original model, Tiny YOLO V2, for comparison and verification. mAP, MACC, Activation RAM, and WeightsFLASH were used as performance evaluation metrics, with mAP representing the model's average precision.

[0172] Table 6 Performance comparison between the second model and Tiny YOLO V2 model

[0173]

[0174] The data in Table 6 shows that the second model of the present invention outperforms Tiny YOLO v2 in all four indicators and is more suitable for deploying reasoning on resource-constrained embedded platforms.

[0175] To verify the effect of full integer quantization in the present invention, the first model and the second model are compared before and after quantization, and Accuracy, mAP, MACC, Activation RAM, and Weights FLASH are used as performance evaluation indicators.

[0176] Table 7 Quantitative comparison of the first model

[0177]

[0178] Table 8 Quantitative comparison of the second model

[0179]

[0180] The data in Tables 7 and 8 show that both models significantly reduce the storage and computing resource requirements after quantization, while the decrease in accuracy is very small, which is more suitable for deploying inference on resource-constrained embedded platforms.

[0181] To verify the effectiveness of the embedded people counting system based on the dual-model architecture proposed in this paper, the first and second models were deployed on an ARM-based MCU, specifically the STM32H743XIT6. The CubeAI model verification tool was used to determine the model's validity. Independent tests of the L2 relative error were performed on the first and second models. The test dataset used random tensors, and the experiment was repeated three times.

[0182] Table 9 System model verification

[0183]

[0184]

[0185] The data in Table 9 shows that the deployed model meets the accuracy requirements. All L2 relative errors are controlled below 0.01, which is consistent with the data of the original model.

[0186] In order to verify the performance of the embedded people counting system based on the dual-model architecture in the present invention, the first model and the second model are deployed in an ARM architecture MCU, specifically STM32H743XIH6, with an operating frequency of 400Mhz, and the inference time, operating voltage, operating current, and system power consumption are used as performance evaluation indicators.

[0187] Table 10 System performance test

[0188]

[0189] The inference time of the first model is 29 milliseconds (ms). To calculate the frames per second (FPS), use the following formula:

[0190]

[0191] The percentage of power consumption reduction of the first model compared to the second model is calculated using the following formula:

[0192]

[0193] The data in Table 10 demonstrates that the dual-model architecture not only provides rapid detection speed but also lowers energy consumption. In the absence of an operator, the system can detect approximately 34 frames per second, while reducing power consumption by approximately 7.25% compared to a single-model architecture.

[0194] Through the verification of the above embodiments, it can be seen that the embedded people counting method and system based on the dual-model architecture proposed in the present invention can realize real-time people counting while ensuring data security; since the network structure adopts a lightweight design, the model has the advantages of fast computing speed and low resource occupation; since embedded devices are used, the system has the advantages of low cost and low power consumption.

[0195] Both the first model and the second model process and judge the image information collected by the current frame. Taking the inference time of the first model as 29ms as an example, the first model receives the collected image information of the current frame and performs preprocessing, inference and post-processing operations, which takes 29ms. When the presence of a person is detected, the second model is activated. At this time, the second model receives the collected image information of the current frame. At this time, the received current frame image information is already a real-time image after 29 milliseconds. The second model also preprocesses, infers and post-processes the image. Because the movement speed of the person in the camera monitoring screen is limited, after the first model completes the 29-millisecond judgment cycle, the second model re-collects the image from the camera, and the person is still within the monitoring screen range. The present invention adopts the first model preprocessing, which can save resources and speed up the judgment of the existence of the person. The inference time of the first model is related to the performance of the ARM chip. In this embodiment, the inference time of the stm32H743XIH6 with an operating frequency of 480Mhz is 29ms, that is, the image frame acquisition interval is 29ms. If the gd32H757ZMT6 with an operating frequency of 600Mhz is used, the image frame acquisition interval can reach 23ms.

[0196] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An embedded people counting method based on a dual-model architecture, characterized in that: The following steps are involved: S1. Construction of the first model and the second model: S11. Use the COCO2017 dataset to extract portrait images to create a dataset; S12. Use the dataset to train a first model and a second model through TensorFlow, where the first model adopts the MobileNetV1 model and the second model adopts the improved Tiny YOLO V2 model. The second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure: a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a first depth-separable convolution block, a second depth-separable convolution block, a third depth-separable convolution block, a fourth depth-separable convolution block, and a last convolution layer; S13, deploying the first model and the second model in an ARM architecture microcontroller unit; S2. Person detection of the first model: S21, receiving the collected current image information; S22, pre-processing the collected image information and entering the first model; S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information; S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and the process proceeds to step S3; S3. People counting of the second model: S31, receiving the collected current image information; S32, pre-processing the collected image information and entering the second model; S33, reasoning of the second model: the second model uses the improved Tiny YOLO V2 model to output graphic information; S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

2. The embedded people counting method based on dual-model architecture according to claim 1, characterized in that: Both the first model and the second model use externally adjustable hyperparameters. The hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression.

3. The embedded people counting method based on a dual-model architecture as claimed in claim 1, characterized in that: Both the first model and the second model adopt full integer quantization.

4. The embedded people counting method based on a dual-model architecture as claimed in claim 1, characterized in that: The deployment of the first model and the second model both adopts model conversion technology, using CMSIS-NN to convert the model into a format that can be run on embedded devices.

5. The embedded people counting method based on dual-model architecture according to claim 1, characterized in that: The image information preprocessing includes image format processing and image compression.

6. The method for embedded people counting based on dual-model architecture according to claim 1, wherein: The second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure: The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; a ReLU activation function; The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; a ReLU activation function; The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; and a ReLU activation function. The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy and no bias; a batch normalization layer; and a ReLU activation function. The first depth-wise separable convolutional block consists of: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The second depth-wise separable convolutional block includes: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'paddingsame' strategy. The third depth-wise separable convolutional block includes: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 128 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The fourth depthwise separable convolutional block includes: a depthwise separable convolutional layer with 3×3 filters, a stride of 1×1, a 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, a 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, a 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

7. An embedded people counting system based on a dual-model architecture, used to implement an embedded people counting method based on a dual-model architecture as claimed in any one of claims 1 to 6, characterized in that: include: Data acquisition module, using a camera to capture real-time image information; The data preprocessing module performs image format conversion and image compression on the captured image information to make the image information adapt to the input standard of the model; A model inference module, utilizing a microcontroller unit to run the first model or the second model; The data post-processing module dequantizes the output of the first model to determine the presence of people, and decodes and performs non-maximum suppression on the output of the second model to obtain the number of people information.

8. The embedded people counting system based on dual-model architecture as claimed in claim 7, characterized in that: Also includes: Information display module, which displays the original image, processed image information and the current hyperparameter information of the system on the screen; The hyperparameter change module triggers external interrupts by pressing buttons to modify the system hyperparameters.

Images