Embedded people counting method and system based on double-model architecture

Abstract

The invention discloses an embedded people counting method and system based on a double-model architecture, and the method comprises the following steps: S1, building a first model and a second model, S2, carrying out the detection of people through the first model, and S3, carrying out the counting of people through the second model. By designing a lightweight model and deploying the lightweight model in the ARM architecture MCU, low-power-consumption and low-cost embedded system edge computing is realized, the system completes data processing and computing locally, and data is prevented from being uploaded to a cloud server, so that the data security is effectively improved; in addition, double-model construction is adopted, so that the system performs figure detection by using the first model in an unmanned state, rapid reasoning is realized, invalid consumption of computing resources is remarkably reduced, and the overall power consumption is further reduced.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to an embedded people counting method and system based on a dual-model architecture. Background Art

[0002] With the development of society, security management and personnel flow control in public places are becoming more and more important. In crowded places such as shopping malls, stations, and airports, real-time and accurate headcount is of great significance for security monitoring, resource allocation, and data analysis. Traditional statistical methods are mainly divided into two categories: hardware-based methods and computer vision-based methods.

[0003] Methods based on hardware devices: mainly use infrared sensors, ultrasonic sensors and other devices to count people. This type of method is more complicated in installation and maintenance, and is easily affected by environmental factors, resulting in low statistical accuracy.

[0004] Computer vision-based methods: Use cameras to capture video images and perform crowd counting through image processing technology. This type of method has the following advantages: no contact, easy installation, and adaptability to different scenarios, but this method requires high-performance computing equipment and faces problems such as data security, high cost, and high power consumption. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an embedded people counting method and system based on a dual-model architecture, which can effectively overcome the limitations of traditional hardware equipment methods and at the same time solve the problems of data security, cost and power consumption based on computer vision methods, and provide an efficient, reliable and economical solution for counting people in public places.

[0006] To solve the above technical problems, the technical solution of the present invention is: an embedded people counting method based on a dual-model architecture, comprising the following steps:

[0007] S1. Construction of the first model and the second model:

[0008] S11. Use the COCO2017 dataset to extract portrait images to create a dataset;

[0009] S12. Using the data set to train a first model and a second model through TensorFlow, the first model adopts the MobileNet V1 model, and the second model adopts the improved Tiny YOLO V2 model;

[0010] S13, deploying the first model and the second model in an ARM architecture microcontroller unit;

[0011] S2. Person detection of the first model:

[0012] S21, receiving the collected current image information;

[0013] S22, pre-processing the collected image information and entering the first model;

[0014] S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information;

[0015] S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and step S3 is entered;

[0016] S3. Crowd counting of the second model:

[0017] S31, receiving the collected current image information;

[0018] S32, pre-processing the collected image information and entering the second model;

[0019] S33, reasoning of the second model: the second model uses an improved Tiny YOLO V2 model to output graphic information;

[0020] S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

[0021] As a preferred technical solution, both the first model and the second model adopt externally adjustable hyperparameters, the hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression.

[0022] As a preferred technical solution, both the first model and the second model use full integer quantization to reduce model size and computational complexity.

[0023] As a preferred technical solution, the deployment of the first model and the second model both adopts model conversion technology, using CMSIS-NN to convert the model into a format that can be run on an embedded device.

[0024] As a preferred technical solution, the image information preprocessing includes image format processing and image compression.

[0025] As a preferred technical solution, the second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure:

[0026] The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'padding same' strategy and no bias; a batch normalization layer; and a ReLU activation function.

[0027] The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0028] The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0029] The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0030] The first depth-wise separable convolutional block includes: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0031] The second depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'padding same' strategy.

[0032] The third depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 128 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0033] The fourth depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0034] The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, using the 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

[0035] Another preferred technical solution is an embedded people counting system based on a dual-model architecture, comprising: a data acquisition module, which uses a camera to capture real-time image information; a data preprocessing module, which performs image format conversion and image compression on the captured image information so that the image information adapts to the input standard of the model; a model reasoning module, which uses a microcontroller unit to run the first model or the second model; a data post-processing module, which dequantizes the output of the first model to determine the presence of a person, and decodes and non-maximum suppresses the output of the second model to obtain number of people information.

[0036] As a preferred technical solution, it also includes: an information display module, which displays the original image, processed image information and the current hyperparameter information of the system through the screen; a hyperparameter change module, which triggers an external interrupt by pressing a button to modify the system hyperparameters.

[0037] Due to the adoption of the above technical scheme, the beneficial effects of the present invention are as follows: the present invention realizes low-power, low-cost embedded system edge computing by designing a lightweight model and deploying it in an ARM architecture MCU. The system completes data processing and calculation locally, avoiding uploading data to a cloud server, thereby effectively improving data security; in addition, the present invention adopts a dual-model construction, so that the system uses the first model to perform person detection in an unmanned state, realizes rapid reasoning, significantly reduces the ineffective consumption of computing resources, and further reduces the overall power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following drawings are intended only to illustrate and explain the present invention, and are not intended to limit the scope of the present invention.

[0039] Figure 1 is a method flow chart of an embodiment of the present invention; DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and examples. In the following detailed description, certain exemplary embodiments of the present invention are described only by way of illustration. Needless to say, those of ordinary skill in the art will recognize that the described embodiments may be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the drawings and description are illustrative in nature and are not intended to limit the scope of protection of the claims.

[0041] like Figure 1 As shown, an embedded people counting method based on a dual-model architecture includes the following steps:

[0042] S1. Construction of the first model and the second model. This step serves as the basis for model operation.

[0043] S11. Use the COCO2017 dataset to extract portrait images to create a dataset;

[0044] S12. Using the data set to train a first model and a second model through TensorFlow, the first model adopts the MobileNet V1 model, and the second model adopts the improved Tiny YOLO V2 model;

[0045] S13, deploying the first model and the second model in an ARM architecture microcontroller unit;

[0046] S2. Person detection of the first model

[0047] S21, receiving the collected current image information;

[0048] S22, pre-processing the collected image information to meet the model input requirements and enter the first model;

[0049] S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information;

[0050] S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and step S3 is entered;

[0051] S3. Crowd Counting of the Second Model

[0052] S31, receiving the collected current image information;

[0053] S32, pre-processing the collected image information to meet the model input requirements and enter the second model;

[0054] S33, reasoning of the second model: the second model uses an improved Tiny YOLO V2 model to output graphic information;

[0055] S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

[0056] The COCO2017 dataset is used for training. The COCO dataset is a large database widely used for object detection, image segmentation, and image annotation. It belongs to the existing technology and will not be described here.

[0057] Preferably, the steps of preparing the data set in step S11 are specifically:

[0058] S111: Open and read the COCO annotation file, and load its content into the variable coco_data;

[0059] S112: traverse the images list in coco_data, and for each image, initialize a Boolean variable copy_image to mark whether the current image needs to be copied, construct the image file name image_file_name, and construct the corresponding label file name label_file_name;

[0060] S113: For each picture, traverse the annotations list and check whether the annotation belongs to the current picture through image_id matching. If so, obtain the category_id of the annotation;

[0061] S114: Find the corresponding category name class_name in the categories list. If class_name is in the provided classes list, set copy_image to True, indicating that the image needs to be copied.

[0062] S115: Get the index class_id of the category, and get the bounding box (bbox) information from the annotation. The bounding box format of COCO is [x, y, width, height], and the YOLO format is [class_id, x_center, y_center, width, height], where x_center and y_center are the normalized center coordinates relative to the width and height of the image. The bounding box coordinates need to be converted to the YOLO format and the result is saved in the corresponding label file.

[0063] S116: If copy_image is True, the image is copied from coco_images_dir to export_folder.

[0064] For the second model, the training mainly involves filtering out the human annotation information in the COCO dataset and converting the COCO format to the YOLO format. For the training of the first model, a Visual WakeWords dataset needs to be built from the COCO dataset. The specific steps are as follows: For each picture in the dataset, a label of 1 or 0 is assigned based on whether there is an object of interest (such as a person). If at least one bounding box in the picture corresponds to the object of interest and the area of ​​the bounding box exceeds a certain threshold (such as 0.5% of the image area), the picture is marked as 1; if these conditions are not met, it is marked as 0.

[0065] In the Tiny YOLO V2 model, each grid cell is responsible for detecting objects in the image and outputting a bounding box for each object. The specific location and size of these bounding boxes are determined by the parameters predicted by the corresponding grid cell. Prior boxes are a set of bounding boxes pre-defined before training, which are used to help the network learn the location and size of objects.

[0066] In Tiny YOLO V2, the prior box is obtained by clustering analysis of the training data set. The K-means clustering algorithm is applied to cluster the bounding boxes in the training data, and the distance formula used is: d = 1-IOU, where IOU stands for intersection over union. Through the K-means algorithm, the five optimal cluster centers, i.e., the prior boxes, are found, as shown in the following table.

[0067] Table 1 Prior box parameters

[0068]

[0069] Preferably, the parameters for training the first model in step S12 include three categories: image preprocessing, image augmentation, and training hyperparameters. Image preprocessing and image augmentation are processing of the data set, while training parameters are for the model and affect the recognition effect of the model. The training parameters of the first model are as follows:

[0070] A. Image Preprocessing

[0071] S121: The scale factor is 1 / 127.5 and the offset is -1 to rescale the image;

[0072] S122: using a nearest neighbor interpolation method to ensure that the adjusted image conforms to the aspect ratio of the original image;

[0073] B. Image Augmentation

[0074] S123: random contrast value is 0.4, random brightness value is 0.05;

[0075] S124: Random horizontal inversion;

[0076] S125: random translation width and height adjustment factors are 0.25 respectively, using reflection fill mode and nearest neighbor interpolation;

[0077] S126: random rotation factor of 0.125, using reflection fill mode and nearest neighbor interpolation;

[0078] S127: Randomly scale width and height with adjustment factors of 0.25, using reflection fill mode and nearest neighbor interpolation;

[0079] S128: random distortion factor of 0.0515, using wrap-around fill mode and nearest neighbor interpolation;

[0080] C. Training Hyperparameters

[0081] S129: Use the MobileNet V1 model, the alpha parameter is 0.25, and the input shape is (96,96,3);

[0082] S1210: Pre-trained weights use ImageNet pre-trained weights;

[0083] S1211: Dropout layer dropout rate is 0.3;

[0084] S1212: batch size is 64 samples;

[0085] S1213: Train for 1000 cycles in total;

[0086] S1214: Use Adam optimizer with a learning rate of 0.001;

[0087] S1215: Use the ReduceLROnPlateau callback function to reduce the learning rate when the validation set accuracy no longer improves, and the learning rate decay factor is 0.5; the number of tolerance cycles is 40, and the lower limit of the learning rate is 1.0e-05;

[0088] S1216: Use the EarlyStopping callback function to stop training and restore the best weights when the validation set accuracy no longer improves, with a tolerance of 60 cycles.

[0089] Preferably, the parameters for training the second model in step S12 include four categories: image preprocessing, image augmentation, training hyperparameters, and post-processing parameters. Image preprocessing, image augmentation, and training hyperparameters have the same functions as those in the first model. Post-processing parameters do not participate in model training and are only called during model evaluation, such as when calculating mAP. The training parameters of the second model are as follows:

[0090] A. Image preprocessing:

[0091] S1217: Scale ratio 1 / 255, no offset, to rescale the image;

[0092] S1218: using a nearest neighbor interpolation method to ensure that the adjusted image conforms to the aspect ratio of the original image;

[0093] B. Image Augmentation

[0094] S1219: The random rotation angle is set to 5 degrees;

[0095] S1220: Translation is set to 0.1;

[0096] S1221: Horizontal flip is set to 1.0;

[0097] S1222: Gaussian blur set to 3.0;

[0098] S1223: Linear contrast set to a random value between 0.75 and 1.5;

[0099] C. Training hyperparameters:

[0100] S1224: Batch size is set to 64;

[0101] S1225: The number of training rounds is set to 1000;

[0102] S1226: Use Adam optimizer and set the learning rate to 0.01;

[0103] S1227: Use ReduceLROnPlateau callback function, learning rate attenuation factor is 0.05, tolerance is 50 cycles, and learning rate lower limit is 1.0e-06;

[0104] S1228: Use EarlyStopping callback function, with a tolerance of 80 cycles;

[0105] D. Post-processing parameters:

[0106] S1229: The confidence threshold is set to 0.001;

[0107] S1230: The non-maximum suppression (NMS) threshold is set to 0.5;

[0108] S1231: The IoU evaluation threshold is set to 0.4;

[0109] S1232: The maximum number of detection frames is set to 980.

[0110] Preferably, the first model in step S12 adopts the MobileNet V1 model. In order to deploy the deep learning network model on the microcontroller unit (MCU) of the ARM architecture, the algorithm needs to be optimized. The Visual Wake Word model based on the Resnet-14 network is an open source model, which belongs to the prior art and is not repeated here. The ResNet-14 deep residual network solves the gradient vanishing and gradient explosion problems in deep network training by introducing residual blocks, but the traditional convolution operation has a large amount of calculation, and the use of residual blocks also increases the calculation burden, resulting in an increase in weight information. Considering the requirements of embedded systems for real-time and lightweight, the Visual Wake Word algorithm is optimized, and the MobileNet V1 model is used to replace the ResNet-14 network model. See Table 2 for the MobileNet V1 model. The MobileNet V1 model uses the design of deep separable convolution to significantly reduce the number of parameters and calculation requirements of the model, making the model more lightweight and more suitable for mobile devices and embedded systems with limited resources.

[0111] Table 2 MobileNet V1 model network structure

[0112]

[0113]

[0114] The first model uses the MobileNet V1 architecture, which is a lightweight network model that replaces ordinary convolution with depthwise separable convolution. Compared with ordinary convolution, it has fewer parameters and computational complexity.

[0115] Preferably, the second model in step S12 adopts an improved Tiny YOLO V2 model. Tiny YOLO V2 is a simplified, lightweight variant of the YOLO target detection algorithm, and its network structure consists of only 9 convolutional layers. Tiny YOLOV2 is a simplified, lightweight variant of the YOLO target detection algorithm, and its network structure consists of only 9 convolutional layers. The purpose of this design is to maintain relatively excellent detection performance in a resource-constrained environment, while significantly reducing the complexity and number of parameters of the network to adapt to embedded devices or platforms with weak computing power. Thanks to this design, Tiny YOLO V2 can achieve real-time target detection under these conditions, opening up broader application prospects for object recognition applications. The specific structure of its network model is shown in Table 3.

[0116] Table 3 Tiny YOLO V2 network structure

[0117]

[0118] In order to improve the performance of the Tiny YOLO V2 model, the second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure, see Table 4 for synchronization:

[0119] The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'padding same' strategy and no bias; a batch normalization layer; and a ReLU activation function.

[0120] The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0121] The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0122] The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function.

[0123] The first depth-wise separable convolutional block includes: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0124] The second depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'padding same' strategy.

[0125] The third depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 128 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0126] The fourth depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function.

[0127] The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, using the 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

[0128] The length of the detection vector is determined according to the number of object categories and the number of bounding box parameters in the actual application scenario. In this embodiment, the length of the detection vector is set to 30.

[0129] Table 4 Improved Tiny YOLO V2 model backbone network structure

[0130]

[0131] The second model uses the improved Tiny YOLO V2 architecture, which is defined as Tiny YOLO V2 LC. First, four convolution layers with 3×3 convolution kernels with a stride of 2 are used to gradually reduce the size of the feature map, mainly to extract features of different scales and better capture the information of the target in the subsequent target detection task. Then, four depthwise separable convolution layers with 3×3 convolution kernels with a stride of 3 are used. Each convolution layer is followed by BatchNormalization and ReLU activation functions. By flexibly using depthwise separable convolutions and reducing the number of parameters, the design of a low-power embedded people detection system based on the ARM architecture ensures that the model is more lightweight while maintaining a high target detection accuracy.

[0132] In the present invention, the first model performs person presence detection, which is essentially an image classification model, and the model architecture all uses deep separable convolution; the second model performs person number counting, which is essentially a target detection model, and ordinary convolution is still used in the early stage of reasoning to better extract features; since the second model only partially uses deep separable convolution, and the target detection task is more complex than the image classification task, compared with the first model, the first model has smaller computational complexity and parameter quantity, faster reasoning speed, and occupies less resources.

[0133] Preferably, the deployment of step S13 includes:

[0134] S131: Convolution and depth-wise separable convolution are built using the NNFunctions API;

[0135] S132: The ReLU activation function is built using the NNSupportFunctions API.

[0136] To port the deep model written in Python to C language so that it can run on the ARM core, the deployment of the first model and the second model both adopts model conversion technology, and uses CMSIS-NN to convert the model into a format that can run on embedded devices. This is a well-known method in the prior art and will not be repeated here.

[0137] Preferably, the first model and the second model both use full integer quantization, which is to replace the common 32-bit or 64-bit floating-point bits for network parameter storage with a smaller number of bits to reduce the space occupied by model parameter storage, and ultimately achieve the purpose of model compression. Full integer quantization uses TensorFlow Lite to convert both weights and activations into integers, including the following steps:

[0138] Calibration: Run the model and record the range of values ​​activated to determine the appropriate scaling factor and zero point;

[0139] Quantize weights: Convert weights from floating point numbers to integers using calibrated parameters;

[0140] Quantize activations: Convert activations from floating point numbers to integers using calibrated parameters;

[0141] Applied optimizations: Folding batch normalization, combining weights and biases, and fusing the convolutional layer and the subsequent ReLU activation layer into a single convolution operation.

[0142] Preferably, both the first model and the second model use externally adjustable hyperparameters, the hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression. The adjustable hyperparameters are set as variables and assigned initial values, so that the system can adjust the hyperparameter values ​​by modifying the variables according to specific needs, which has higher flexibility and applicability, and can be adjusted and optimized according to the needs of different scenarios.

[0143] Preferably, the image information is in RGB565 image format, and image information preprocessing includes image format processing and image compression.

[0144] The image format processing is specifically as follows: the data format collected by the camera OV2640 is RGB565, that is, RGB16, but the input data format of the low-power embedded people detection system 18 based on the ARM architecture of the deep learning model designed in this embodiment is RGB888, that is, RGB24, so a conversion program needs to be designed. The whole program can be described as follows:

[0145] (1) Extract the channel data in RGB565.

[0146] (2) Expand the 5-bit red data to 8 bits by shifting left by 3 bits.

[0147] (3) Expand the 6-bit green data to 8 bits by shifting left by 2 bits and padding the lowest two bits with 0.

[0148] (4) Expand the 5-bit blue data to 8 bits by shifting left by 3 bits.

[0149] (5) Combine the expanded red, green, and blue data into a pixel value in RGB888 format.

[0150] (6) Repeat the above steps to traverse each pixel value.

[0151] Image compression: In the people counting system, whether using Visua Wake Word or Tiny YOLO V2LC, their input needs to compress the image into a specific shape for neural network processing. The nearest neighbor interpolation algorithm is used to compress image information. Nearest neighbor interpolation (NNI) is an interpolation algorithm based on KD tree, which is often used in image processing and other problems.

[0152] The advantages of the nearest neighbor interpolation method are that it is simple to implement and has a fast calculation speed. It is particularly suitable for embedded systems with limited resources or when processing speed is more important than image quality.

[0153] In addition, the data type obtained by the camera OV2640 is Uint16, the data required by Visual Wake Word is int8, and the data required by TinyYolo V2 LC is Uint8, so data type conversion is also required. It is relatively simple to convert Uint16 to Uint8. When the image format is converted, the color channel of each pixel will be expanded from 16 bits (5 bits of red, 6 bits of green, and 5 bits of blue) to 24 bits (8 bits of red, 8 bits of green, and 8 bits of blue). In this case, there is already a Uint16 array that stores RGB888 format image data, and each channel has been allocated as Uint8. You only need to force the array to be converted to Uint8 and expand the array length accordingly. The conversion of Uint16 to int8 is based on this, with an additional mapping step. Because the range of int8 is -127 to 127, it is sufficient to offset 128 units on the original data.

[0154] Preferably, in step S24, the final data data type of the Visual Wake Word model is int8. In order to express the probability of the current category, these integer data need to be dequantized into floating point numbers. The existence detection is based on the output judgment of the first model, and the output of the first model is dequantized. The dequantized output result is used as the class confidence. When the class confidence is greater than the class confidence threshold, it is judged that a person exists. When the person is detected, the second model is activated. When the class confidence is less than the class confidence threshold, it is judged that no one exists, and step S2 is repeated;

[0155] Among them, the inverse quantization adopts the formula:

[0156] real_value_output=(q_output-(-128))*0.003906250;

[0157] real_value_output refers to the dequantized output result, and q_output refers to the model output.

[0158] After extracting the output of the Tiny YOLO V2 LC model, the output data must be decoded to extract the required information. The decoding method belongs to the prior art and will not be described here.

[0159] The core idea of ​​the non-maximum suppression (NMS) algorithm is to find local maxima and suppress non-maximum elements. In the process of target detection, many overlapping candidate bounding boxes may be generated for the same target. In order to screen out the optimal target bounding box and eliminate redundant boxes, the non-maximum suppression technology is used. The implementation of this algorithm depends on the intersection over union (IOU), which measures the degree of overlap between anchor boxes. Through the score of the anchor box, the redundant anchor boxes with low scores and high overlap with the high-scoring anchor boxes can be removed one by one. Just set an appropriate IOU threshold to obtain accurate anchor box prediction results.

[0160] Preferably, in step S34, the number of characters is counted based on the number of anchor frames, which is determined by the anchor frame score threshold and non-maximum suppression screening after model detection; the output of the second model is decoded and non-maximum suppression is used, and when it is detected that the character disappears, the first model is activated.

[0161] An embedded people counting system based on a dual-model architecture, comprising:

[0162] The data acquisition module uses a camera to capture real-time image information;

[0163] The data preprocessing module performs image format conversion and image compression on the captured image information so that the image information adapts to the input standard of the model;

[0164] The model inference module uses the microcontroller unit to run the first model or the second model; the data post-processing module dequantizes the output of the first model to determine the existence of a person, and decodes and non-maximum suppresses the output of the second model to obtain the number of people information.

[0165] Information display module, which displays the original image, processed image information and current hyperparameter information of the system on the screen;

[0166] The hyperparameter change module triggers external interrupts by pressing buttons to modify the system hyperparameters.

[0167] In order to verify the credibility and effectiveness of the above-mentioned embedded people counting method and system based on the dual-model architecture, an experimental process is designed for verification:

[0168] In order to verify the effect of the first model in the present invention, the Visual Wake Word model based on the Resnet-14 network was selected for comparative verification. The model has the same data set and the same function as the first model in the present invention. The data set adopts the COCO2017 data set, and Accuracy, MACC, Activation RAM, and Weights FLASH are used as performance evaluation indicators. Accuracy represents the accuracy of the model. MACC refers to the multiplication and accumulation operations in the neural network, indicating the model complexity and computing resource consumption. Activation RAM represents the memory size occupied during model inference, and Weights FLASH represents the memory size occupied by the model weights.

[0169] Table 5 Performance comparison between the first model and the Visual Wake Word model

[0170]

[0171] The data in Table 5 reflect that the first model of the present invention is much higher than the VisualWake Word model in the MACC and Activation RAM indicators, and is more suitable for deploying reasoning on resource-constrained embedded platforms. In order to verify the effect of the second model in the present invention, the original model Tiny YOLO V2 is selected for comparative verification, and mAP, MACC, Activation RAM, and WeightsFLASH are used as performance evaluation indicators. mAP represents the average accuracy of the model.

[0172] Table 6 Performance comparison between the second model and Tiny YOLO V2 model

[0173]

[0174] The data in Table 6 show that the second model of the present invention is superior to Tiny YOLO v2 in all four indicators and is more suitable for deploying reasoning on resource-constrained embedded platforms.

[0175] In order to verify the effect of full integer quantization in the present invention, the first model and the second model are compared before and after quantization, and Accuracy, mAP, MACC, Activation RAM, and Weights FLASH are used as performance evaluation indicators.

[0176] Table 7 Quantitative comparison of the first model

[0177]

[0178] Table 8 Quantitative comparison of the second model

[0179]

[0180] The data in Table 7 and Table 8 show that both models significantly reduce the requirements for storage and computing resources after quantization, while the decrease in accuracy is very small, which is more suitable for deploying reasoning on resource-constrained embedded platforms.

[0181] In order to verify the effectiveness of the embedded people counting system based on the dual-model architecture in the present invention, the first model and the second model are deployed in an ARM architecture MCU, specifically STM32H743XIT6, and the CubeAI model verification tool is used to determine whether the model is valid. The L2 relative error of the first model and the second model are independently tested. The test data set uses random tensors, and the experiment is repeated three times.

[0182] Table 9 System model verification

[0183]

[0184]

[0185] The data in Table 9 show that the deployed model meets the accuracy requirements, and all L2 relative errors are controlled below 0.01, which is consistent with the data of the original model.

[0186] In order to verify the performance of the embedded people counting system based on the dual-model architecture in the present invention, the first model and the second model are deployed in an ARM architecture MCU, specifically STM32H743XIH6, with an operating frequency of 400Mhz, and the inference time, operating voltage, operating current, and system power consumption are used as performance evaluation indicators.

[0187] Table 10 System performance test

[0188]

[0189] The inference time of the first model is 29 milliseconds (ms). To calculate the frames per second (FPS), use the following formula:

[0190]

[0191] The percentage of power consumption reduction of the first model compared to the second model is calculated using the following formula:

[0192]

[0193] The data in Table 10 shows that the dual-model architecture not only provides fast detection speed, but also has low energy consumption. In the absence of human intervention, the system can achieve data detection of approximately 34 frames per second, while the power consumption is reduced by approximately 7.25% compared to the single-model architecture.

[0194] It can be seen from the verification of the above embodiments that the embedded people counting method and system based on the dual-model architecture proposed in the present invention can realize real-time people counting while ensuring data security; since the network structure adopts a lightweight design, the model has the advantages of fast calculation speed and less resource occupation; since embedded devices are used, the system has the advantages of low cost and low power consumption.

[0195] Both the first model and the second model process and judge the image information collected by the current frame. Taking the reasoning time of the first model as 29ms as an example, the first model receives the collected image information of the current frame and performs preprocessing, reasoning and post-processing operations, which takes 29ms. When the presence of a person is detected, the second model is activated. At this time, the second model receives the collected image information of the current frame. At this time, the received current frame image information is already a real-time image after 29 milliseconds. The second model also preprocesses, reasoned and post-processes the image. Because the movement speed of the person in the camera monitoring screen is limited, after the first model completes the 29-millisecond judgment cycle, the second model re-collects the image from the camera, and the person is still within the monitoring screen. The present invention adopts the first model for preprocessing, which can save resources and speed up the judgment of the presence of people. The inference time of the first model is related to the performance of the ARM chip. In this embodiment, the inference time of the stm32H743XIH6 with an operating frequency of 480Mhz is 29ms, that is, the image frame acquisition interval is 29ms. If the gd32H757ZMT6 with an operating frequency of 600Mhz is used, the image frame acquisition interval can reach 23ms.

[0196] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An embedded people counting method based on a dual-model architecture, characterized in that: The following steps are involved: S1. Construction of the first model and the second model: S11. Use the COCO2017 dataset to extract portrait images to create a dataset; S12. Use the data set to train the first model and the second model through TensorFlow, where the first model adopts the MobileNetV1 model and the second model adopts the improved Tiny YOLO V2 model; S13, deploying the first model and the second model in an ARM architecture microcontroller unit; S2. Person detection of the first model: S21, receiving the collected current image information; S22, pre-processing the collected image information and entering the first model; S23, reasoning of the first model: the first model uses the MobileNet V1 model to output image information; S24, post-processing of the first model: performing inverse quantization processing on the output of the first model to determine whether the person exists. When it is detected that the person disappears, step S2 is repeated; when it is detected that the person exists, the second model is activated and step S3 is entered; S3. Crowd counting of the second model: S31, receiving the collected current image information; S32, pre-processing the collected image information and entering the second model; S33, reasoning of the second model: the second model uses an improved Tiny YOLO V2 model to output graphic information; S34, post-processing of the second model: decode and perform non-maximum suppression processing on the output of the second model to obtain the number of people information. When the presence of a person is detected, repeat step S3; when the disappearance of the person is detected, activate the first model and return to step S2.

2. The embedded people counting method based on dual model architecture as claimed in claim 1, characterized in that: Both the first model and the second model use externally adjustable hyperparameters, the hyperparameters of the first model include a class confidence threshold, and the hyperparameters of the second model include an anchor box score threshold and an IOU threshold for non-maximum suppression.

3. The embedded people counting method based on dual model architecture as claimed in claim 1, characterized in that: Both the first model and the second model adopt full integer quantization.

4. The embedded people counting method based on dual model architecture as claimed in claim 1, characterized in that: The deployment of the first model and the second model both adopts model conversion technology, using CMSIS-NN to convert the model into a format that can be run on embedded devices.

5. The embedded people counting method based on dual model architecture as claimed in claim 1, characterized in that: The image information preprocessing includes image format processing and image compression.

6. The embedded people counting method based on dual model architecture as claimed in claim 1, characterized in that: The second model replaces the backbone network of Tiny YOLO V2 to form an improved Tiny YOLO V2, which specifically includes the following structure: The first convolutional block includes: a convolutional layer with 16 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; and a ReLU activation function. The second convolutional block includes: a convolutional layer with 32 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; and a ReLU activation function. The third convolutional block includes: a convolutional layer with 64 3×3 filters, a stride of 2×2, using the 'paddingsame' strategy, and no bias term; a batch normalization layer; and a ReLU activation function. The fourth convolutional block includes: a convolutional layer with 128 3×3 filters, a stride of 2×2, using the 'padding same' strategy, and no bias term; a batch normalization layer; and a ReLU activation function. The first depth-wise separable convolutional block includes: a depth-wise separable convolutional layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolutional layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The second depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; a ReLU activation function; a max pooling layer with a 2×2 pooling window, a stride of 1×1, and a 'paddingsame' strategy. The third depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 128 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The fourth depth-wise separable convolution block includes: a depth-wise separable convolution layer with 3×3 filters, a stride of 1×1, using the 'padding same' strategy, and no bias; a batch normalization layer; a 1×1 convolution layer with 256 filters, a stride of 1, using the 'padding same' strategy, and no bias; a batch normalization layer; and a ReLU activation function. The last convolutional layer includes: a 1×1 convolutional layer with the number of filters equal to the length of the detection vector, a stride of 1×1, using the 'padding same' strategy, and no bias term; a batch normalization layer as the output of the model.

7. An embedded people counting system based on a dual-model architecture, used to implement an embedded people counting method based on a dual-model architecture as claimed in any one of claims 1 to 6, characterized in that: include: The data acquisition module uses a camera to capture real-time image information; The data preprocessing module performs image format conversion and image compression on the captured image information so that the image information adapts to the input standard of the model; A model reasoning module, utilizing a microcontroller unit to run the first model or the second model; The data post-processing module dequantizes the output of the first model to determine the presence of a person, and decodes and performs non-maximum suppression on the output of the second model to obtain information about the number of people.

8. The embedded people counting system based on dual model architecture as claimed in claim 7, characterized in that: Also includes: Information display module, which displays the original image, processed image information and current hyperparameter information of the system on the screen; The hyperparameter change module triggers external interrupts by pressing buttons to modify the system hyperparameters.

Images