A design method for an unmanned trash can system based on machine vision

Abstract

A design method for an unmanned trash bin system based on machine vision belongs to the field of machine vision technology. The system utilizes a combined end-to-end cloud architecture and uses edge computing to process video and other information in real time, improving system responsiveness, reducing data transmission load, and lowering platform operating costs. The system can be deployed at centralized community waste collection points to assist in the correct placement and sorting of waste. It provides real-time monitoring and statistics on residents' waste placement and trash bin status, ensuring that the waste collection process is supervised and traceable. This system can replace existing manual monitoring methods, effectively reducing labor costs and improving the implementation of waste sorting policies.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and in particular relates to a behavior discrimination method based on deep learning. Background Art

[0002] With the rapid development of smart cities and the continuous improvement of residents' living standards, people's demand for a high-quality living environment and a green, environmentally friendly society is growing stronger. To meet people's growing needs, a wide variety of goods and daily necessities are flooding the market. This has brought with it an increasing amount of domestic waste, which has placed tremendous pressure on the environment. Some cities have even become surrounded by garbage. Waste sorting is a major factor in environmental pollution and the difficulty of resource recycling, and has become an urgent issue that needs to be addressed in my country.

[0003] To achieve a preliminary classification of domestic waste, sortable trash cans are currently available on the market, but they still require manual placement of the trash into the corresponding sorting slots. However, residents generally lack awareness and knowledge of waste sorting, resulting in incomplete waste sorting. Therefore, the government has implemented a system where volunteers are stationed at designated garbage stations to educate residents who come to dispose of their waste, monitor their actions, and perform secondary sorting and classification of the waste in the bins. While this volunteer stationing system can reduce misclassification and improve the accuracy of waste sorting, it requires a significant investment of human resources, resulting in a waste of human resources.

[0004] In response to the above problems, the present invention has designed an unmanned garbage sorting system based on machine vision. This system can be deployed at centralized community garbage disposal points to assist in the correct placement and classification of garbage. It also monitors and compiles statistics on residents' garbage placement and the status of garbage bins in real time, ensuring that the garbage placement process is supervised and traceable. The system can replace the original manual on-duty method, effectively reducing labor costs and improving the implementation of garbage sorting policies. The system adopts an end-to-end cloud architecture and processes video and other information in real time through edge computing, improving the system's response timeliness, reducing the data transmission load, and reducing the platform's operating costs. Summary of the Invention

[0005] The present invention discloses an end-to-end cloud-integrated unmanned trash can system based on machine vision. The terminal uses a face recognition algorithm to identify the person who puts the garbage this time, providing a basis for tracing the wrong dumping behavior; uses a target detection algorithm to identify the type of garbage to be put; uses a multi-target tracking algorithm to track the detected garbage to be put; uses an expert system to match the garbage put this time with the garbage can area of ​​a specific category divided in advance to determine whether the placement behavior is correct; uses dual cameras to jointly measure the height of the garbage stack in the garbage can, and when it exceeds the threshold height, the garbage can is considered to be overflowing. Finally, the above information is summarized to the cloud management platform to realize the tracing of the wrong dumping behavior of garbage in the unmanned area and the real-time monitoring and data aggregation of the overflowing status of the garbage can.

[0006] The main steps of this method are as follows:

[0007] Step 1: Use the face detection algorithm to obtain the face detection boxes in the camera image and the facial key points landmarks corresponding to each detection box; send the boxes and their corresponding landmarks to the alignment network to output the aligned face image Image face ; Align the face image Image face Send it to the face recognition network to generate an n-dimensional feature vector embedding, which satisfies:

[0008] embedding=(x1,x2,x3…x n )

[0009] As the basis for face comparison; finally, the generated facial feature vector embedding is compared with the registered facial feature vector set in the database For comparison, e i satisfy:

[0010] e i =(y1,y2,y3…y n )

[0011] Calculate its Euclidean distance d:

[0012]

[0013] Select the value that satisfies the Euclidean distance d less than the preset threshold d threshold The name corresponding to the face information is taken as the final recognition result, where d threshold =1.24.

[0014] Step 2 uses a deep neural network model to detect the target of the garbage to be placed and obtain information about the type of garbage to be placed. This is divided into the following four stages:

[0015] (1) In the preprocessing stage, the input image is sliced, the sliced ​​image is convolved and reduced in dimension, and then sent to the backbone network for feature extraction;

[0016] (2) In the feature map extraction stage, the backbone network consists of a deep separable convolution (DSConv) module and a ghost convolution bottleneck layer (SE-GhostBottleneck) module with a channel attention mechanism.

[0017] The input of the SE-GhostBottleneck module goes through ghost convolution dimensionality reduction GhostConv – group convolution DWConv – channel weight adjustment SELayer – ghost convolution dimensionality increase GhostConv operation to output feature map.

[0018] The specific structure of the backbone network is as follows:

[0019] (2-1) The input first passes through the DSConv module for convolution downsampling operation, and then is sent to the SE-GhostBottleneck module for feature extraction to obtain a shallow feature map;

[0020] (2-2) The shallow feature map is sent to the DSConv module for convolution downsampling operation, and then sent to the 3-layer cascade SE-GhostBottleneck module for feature extraction to obtain a deeper feature map.

[0021] (2-3) The deeper feature map is sent to the DSConv module for convolution downsampling operation, and then sent to the 3-layer cascade SE-GhostBottleneck module for feature extraction to obtain the deep feature map.

[0022] (2-4) The deep feature map is sent to the spatial pyramid pooling layer SPP module for multiple receptive field fusion to improve the robustness of the model for detecting objects of different sizes, and the output feature map is sent to the channel attention module SELayer for channel weight adjustment, and its output result is used as the feature extraction result of the backbone network.

[0023] (3) Feature fusion stage: For the feature maps extracted by the backbone network, based on the strong semantic features of the deep feature maps and the strong positioning features of the shallow feature maps, parameters of different detection layers are aggregated from different backbone layers to better fuse the features extracted by the backbone network. The specific structure is as follows:

[0024] (3-1) First, the backbone network output feature map is sent to the GhostBottleneck module of the ghost convolution bottleneck layer to output the feature map f1;

[0025] (3-2) Send the feature map f1 to the GhostConv module and output the feature map f2;

[0026] (3-3) Upsample the feature map f2 and stack it with the feature map mentioned in (2-3) in the channel dimension through a skip connection layer to output the feature map f3;

[0027] (3-4) Send the feature map f3 to the GhostBottleneck module of the ghost convolution bottleneck layer and output the feature map f4;

[0028] (3-5) Send the feature map f4 to the GhostConv module and output the feature map f5;

[0029] (3-6) Upsample the feature map f5 and stack it with the feature map mentioned in (2-2) in the channel dimension through a skip connection layer to output the feature map f6;

[0030] (3-7) The feature map f6 is fed into the GhostBottleneck module of the ghost convolution bottleneck layer, and the feature map FM1 is output;

[0031] (3-8) Send the feature map FM1 to the depth-wise separable convolution DSConv module and output the feature map f7;

[0032] (3-9) The feature map f7 is stacked with the feature map mentioned in step (3-4) in the channel dimension through a skip connection layer to output the feature map f8;

[0033] (3-10) Send the feature map f8 to the GhostBottleneck module of the ghost convolution bottleneck layer and output the feature map FM2;

[0034] (3-11) Send the feature map FM2 to the depth-wise separable convolution DSConv module and output the feature map f9;

[0035] (3-12) The feature map f9 is stacked with the feature map mentioned in (3-2) in the channel dimension through a skip connection layer, and the feature map f10 is output;

[0036] (3-13) The feature f10 map is fed into the GhostBottleneck module of the ghost convolution bottleneck layer, and the feature map FM3 is output;

[0037] (4) In the target detection stage, a three-level detection head is set up, corresponding to the detection targets of small, medium, and large scales respectively. The feature maps FM1, FM2, and FM3 are fed into the detection head, and the bounding box is regressed according to the preset anchor box to achieve target detection.

[0038] Step 3 uses a multi-target tracking algorithm to track and predict the target trajectory. In order to solve the problem of target sequence jump caused by the original occlusion, a tracker dual-input parallel structure is designed to combine multi-angle feature information to ensure the stability of the tracking effect and reduce the misjudgment rate. The specific process is as follows:

[0039] (1) Combine the tracking frames at each moment to form a set of trajectories;

[0040] (2) Predict a tracking frame for the current frame through filtering operation. The tracking frame has two states: a certain state and an uncertain state.

[0041] (3) Detect the current frame, and then associate the detection frame with the tracking frame. At this time, two situations may occur:

[0042] (a) Matching is successful, and the tracking frame is updated;

[0043] (b) If the matching fails, perform IOU matching on the tracking box and the detection box that failed to match. If the matching succeeds, update it and continue the prediction-observation-update tracking process; if it still fails, match the feature vector of the original tracked target with the feature vector of the auxiliary view detection target:

[0044] If the match is successful, the tracking target is considered to be not lost and the tracking information is retained. Otherwise, a new tracking frame is established for it and initialized to an uncertain state. IOU matching is performed on the next few frames. If the match is successful, the tracking frame is set to a confirmed state and the prediction-observation-update tracking process continues.

[0045] (4) Execute the above steps (1) to (3) for the next frame until the task is completed.

[0046] Step 4: The expert system matches the coordinates of the end point of the garbage dumping trajectory with the type of garbage bin in the area. A mismatch is considered as misplaced garbage, and failure to enter the area is considered as garbage spillage. Specifically, the following operations are performed for each frame of the image:

[0047] (1) Traverse all detection frames. If the hand or target initial coordinates are detected within the delivery domain, no processing is done and the next frame is entered. Otherwise, execute (2).

[0048] (2) If the tracking end coordinates are not within the delivery area, the delivery is determined to be garbage spillage and the next frame is entered. Otherwise, execute (3)

[0049] (3) If the detection target type does not match the delivery area type, the delivery error information is added to the judgment list; otherwise, the delivery correct information is added to the judgment list, and the alarm display duration is set to 0.

[0050] (4) Set the termination display duration to 2s and traverse the judgment list. If the alarm display duration is longer than the termination display duration, remove the delivery information from the judgment list and send the alarm information to the cloud server. Otherwise, keep the alarm information display until it is longer than the termination display duration.

[0051] In step 5, the head-up camera C1 obtains the current height h of the dropped object from the ground, and predefines a specific height H (slightly higher than the trash can opening). The head-down camera C2 obtains the pixel width s′ of the detection target frame and divides it by the pixel width s of the demarcated trash can area to obtain the proportional coefficient k. The formula is as follows:

[0052]

[0053] Where k∈(0,1), when the height h of the center point of the object detected by the horizontal camera coincides with the preset height H, that is, h=H, the side length W of the trash can is known. t Calculate the actual width W of the dropped object using the following formula:

[0054] W=W t ×k

[0055] The distance D between the object and the camera is calculated using the camera monocular ranging principle. The formula is as follows:

[0056]

[0057] Where D is the distance from the detection object to the camera, W is the actual width of the detection object, F is the focal length of the camera, and P is the pixel value of the width of the detection object in the camera.

[0058] Finally, by calculating the height H of the camera C2 from the ground at a known viewing angle c The distance D to the target closest to the camera min The maximum garbage stacking height H in the current bucket is calculated by the difference max , the formula is as follows:

[0059] H max =H c -D min

[0060] And set the threshold height H threshold satisfy:

[0061] H threshold =H can ×α

[0062] Among them H can is the total height of the trash can, α is the overflow coefficient, take α=0.7, when H max Exceeding the threshold height H threshold , the trash can is considered overflowing.

[0063] In step 6, the edge computing device starts two processes—a main process and a subprocess—and communicates through shared memory S. The subprocess uses the head-up camera C1 from step 5 as input to the facial recognition module, outputs the name of the person delivering the object, N, and obtains the height information h of the currently delivered object. The subprocess stores these information in shared memory S for the main process to access. The main process uses the head-down camera C2 from step 5 as input for target detection and feeds the results to the tracker, which determines the type CL and trajectory information T of the delivered object. These information is then fed into the expert system to determine the result Code for the delivery action and obtain the current timestamp T. Finally, the aforementioned information—the name of the person delivering the object, the delivery result Code, the current timestamp T, and the stack height h within the bucket—is sent to the cloud platform for data aggregation. During this time, the main process streams the data to the cloud-based Nginx server via RTMP to enable real-time monitoring of the process.

[0064] In summary, the present invention addresses the burden of manual on-duty tasks at garbage stations at this stage and proposes a garbage disposal behavior recognition and monitoring system based on deep learning. The system can be deployed at centralized garbage disposal points in the community to assist in the correct placement and classification of garbage, and conduct real-time monitoring and statistics on residents' garbage disposal and garbage bin status information to ensure that the garbage disposal process is supervised and traceable. The system can replace the original manual supervision, effectively reduce labor costs, and improve the implementation effect of garbage classification policies. In addition, it should be noted that the part of the invention patent related to user information collection needs to be carried out under the premise of user knowledge and authorization, and the relevant information can only be used for tracing the source of incorrect placement and is prohibited from being disclosed to the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for use in the embodiments or descriptions of the prior art. The drawings described below are only representative of some embodiments of the present invention, but the algorithms of the present invention are generally applicable to related tasks.

[0066] Figure 1 The overall flow chart of the system design provided by the present invention;

[0067] Figure 2 Improved YOLOv5-Ghost network structure diagram in the embodiment provided by the present invention;

[0068] Figure 3 This is a module structure diagram of the SE-GhostBottleneck layer of the ghost network that adds a channel attention mechanism in the embodiment provided by the present invention;

[0069] Figure 4 A schematic diagram of the flow of a parallel dual-camera multi-target tracking algorithm in an embodiment of the present invention;

[0070] Figure 5 A flowchart of the expert system in the embodiment provided by the present invention;

[0071] Figure 6 This is a schematic diagram of the principle of dual-camera joint ranging in the embodiment provided by the present invention;

[0072] Figure 7 A schematic diagram of the multi-process communication principle based on middleware in an embodiment provided by the present invention;

[0073] Figure 8 This is a diagram of the vertical camera output result in the embodiment provided by the present invention;

[0074] Figure 9 This is a diagram of the horizontal camera output result in the embodiment provided by the present invention;

[0075] Figure 10 This is a rendering of the information management platform in the embodiment provided by the present invention. DETAILED DESCRIPTION

[0076] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings and relevant precautions.

[0077] The present invention will be described below from multiple different aspects, including algorithms, parameters, and mathematical principles. For ordinary people who want to use the present invention, if they do not want to make further improvements and developments, they can only use some or all of the structures or processes of the present invention to implement the present invention. For a clearer explanation, specific contents including model optimization, parameter configuration, and processing sequence are specifically explained, but the relevant parameters and configurations can be flexibly changed according to the required tasks. At the same time, some well-known methods will not be elaborated in detail below.

[0078] The current volunteer-based approach to responding to and maintaining waste sorting policies consumes significant human resources. Therefore, a behavioral recognition solution combining machine vision and deep learning can significantly improve this situation. The specific implementation process is as follows:

[0079] 1. Set up camera C1 at a horizontal angle and connect it to edge computing device D with built-in face detection and face recognition algorithms. Use the face detection algorithm to capture the face area of ​​the person being placed. Then use the face recognition algorithm to reduce the dimension of the face area to obtain low-dimensional feature information. Compare the feature information with the face library data, return the matching result, and complete the identification and authentication of the person's identity information.

[0080] 2. Set up a bird's-eye view camera C2 at a vertical angle and connect it to the edge computing device D with built-in target detection and target tracking algorithms. Combine cameras C1 and C2 to locate and classify the dropped objects from two angles using the target detection algorithm. The detection results are sent to the tracking network, and multi-target tracking is performed by combining the feature information of horizontal and vertical angles.

[0081] 3. The expert system matches the end point of the object's trajectory with the type of trash can in the area. An incorrect match is considered an incorrect placement, and failure to enter the area is considered as garbage spillage.

[0082] 4. The height of the garbage stack in the trash can is calculated through the joint ranging of dual cameras. When the stacking height exceeds the threshold height, the trash can is considered to be overflowing.

[0083] 5. The above-mentioned identity information of the person who placed the garbage, the results of the placement behavior, and the garbage bin overflow status information are uniformly sent to the cloud information platform for data aggregation, statistics and display.

[0084] Example

[0085] Figure 1 The flowchart of the machine vision-based end-cloud integration system in this embodiment is shown. The method includes the following steps:

[0086] 1. Use the RetinaFace face detection algorithm with MobileNet0.25 as the feature backbone network as the face recognition module, obtain the face detection boxes in the camera image and the 5 facial key points landmarks corresponding to each detection box; send the boxes and their corresponding landmarks to the alignment network to output the aligned face image Image face ; Align the face image Image face Send it to the InsightFace network to generate a 512-dimensional feature vector embedding, which satisfies:

[0087] embedding=(x1,x2,x3…x n )

[0088] As the basis for face comparison; finally, the generated facial feature vector embedding is compared with the registered facial feature vector set in the database For comparison, e i satisfy:

[0089] e i =(y1,y2,y3…y n )

[0090] Calculate its Euclidean distance d:

[0091]

[0092] Select the value that satisfies the Euclidean distance d less than the preset threshold d threshold The name corresponding to the face information is taken as the final recognition result, where d threshold =1.24.

[0093] 2. Use the improved YOLOV5-Ghost deep neural network model to detect the target garbage and obtain information about the type of garbage to be placed. Considering the computing power limitations of terminal mobile devices, a series of lightweight processing is performed on the original network.

[0094] 2.1 There are a large number of conventional convolution operations in the original network, which bring high parameter and computational complexity to the model. Consider replacing them with depthwise separable convolution and ghost convolution.

[0095] Assuming that the input feature map size is Win×Hin×M, the output feature map size is Wout×Hout×N, and the convolution kernel size is K×K, the calculation formulas for the computational amount and parameter amount of the three convolution operations are as follows:

[0096] Conventional convolution calculation amount

[0097] K×K×M×Wout×Hout×N

[0098] Depthwise separable convolution computational complexity

[0099] (K×K+N)×M×Wout×Hout

[0100] Ghost convolution computational complexity

[0101]

[0102] Conventional convolution parameter

[0103] K×K×M×N

[0104] Depthwise separable convolution parameters

[0105] K×K×M+M×N

[0106] Ghost convolution parameters

[0107]

[0108] Take the convolution layer with input parameters of (160, 160, 64), output parameters of (160, 160, 64), and convolution kernel sizes of K = 1 and K = 3 as an example:

[0109] K=1 / K=3 Regular convolution Depthwise Separable Convolution Ghostly Convolution Parameter quantity 4096 / 36864 4160 / 4672 2080 / 18720 Computational Amount 105M / 944M 106M / 120M 53M / 479M

[0110] Combined with the above table, we choose to replace the regular convolution with a convolution kernel size of 1 with ghost convolution, and replace the regular convolution with a convolution kernel size of 3 with depthwise separable convolution.

[0111] This network is lightweight based on the original YOLOv5s network structure. The network structure parameter adjustment process is shown in the following table:

[0112]

[0113]

[0114] GFLOPS (Giga Floating-point Operations Per Second) represents 1 billion floating-point operations per second, P (Precision) represents accuracy, R (Recall) represents recall, mAP@0.5 (mean Average Precision IOU = 0.5) represents the average accuracy of the dataset when the IOU is set to 0.5, and takes the average, mAP@0.5:0.95 (IOU threshold ranges from 0.5 to 0.95, with a step size of 0.05) represents the average mAP at different IOU thresholds, and FPS (Frames Per Second) represents the number of detection frames of the final trained model in actual applications.

[0115] 2.2 Backbone Network The Backbone module conducts an experimental comparison of two mainstream structures. The results are shown in the following table:

[0116]

[0117] Considering the accuracy guarantee issue, the hourglass-shaped SE-GhostBottleneck structure containing the attention module is used to replace the C3 structure in the original network, in order to exchange the detection speed for the lowest possible accuracy cost to meet the real-time requirements of the system.

[0118] The final network structure is as follows:

[0119] The entire network is divided into three parts. The first part is the lightweight backbone network of GhostNet with an attention mechanism, which is mainly used to extract features from the input data. The second part is the bottleneck layer module of GhostNet, which integrates depthwise separable convolution. It adopts the FPN+PAN structure, combining the top-down transmission of strong semantic features of FPN and the bottom-up transmission of strong positioning features of PAN to better integrate the features extracted by the backbone. The third part is the detection head, which makes predictions for detection targets of three different sizes and dimensions. Specifically, take the input of the convolutional network as an example: a 3*640*640 RGB three-channel color image:

[0120] (a) After the Focus module, the feature map with a size of 12*320*320 is obtained after slicing. After 32 convolution layers with a kernel size of 3 and a stride of 1, the feature map with a size of 32*320*320 is obtained after dimensionality reduction.

[0121] (b) The 32*320*320 feature map is fed into 64 depth-wise separable convolution (DSConv) modules with a kernel size of 3 and a stride of 2 to obtain a feature map of size 64*160*160.

[0122] (c) The 64*160*160 feature map is fed into 64 ghost convolution bottleneck layers with a convolution kernel of 3 and a stride of 1, with the SE channel attention mechanism added.

[0123] SE-GhostBottleneck module, obtains a feature map of size 64*160*160;

[0124] (d) The 64*160*160 feature map is fed into 128 depth-wise separable convolution (DSConv) modules with a kernel size of 3 and a stride of 2, and then fed into 128 SE-GhostBottleneck modules with the same structure as in (c) for three times to obtain a feature map of size 128*80*80.

[0125] (e) The 128*80*80 feature map is fed into 256 depthwise separable convolution (DSConv) modules with a kernel size of 3 and a stride of 2, and then fed into 256 SE-GhostBottleneck modules with the same structure as in (c) for three times to obtain a feature map of size 256*40*40.

[0126] (f) The 256*40*40 feature map is fed into a depthwise separable convolution (DSConv) module with 512 convolution kernels of size 3 and stride 2 to obtain a feature map of size 512*20*20.

[0127] (g) In order to improve the robustness of the model for detecting objects of different sizes, multiple receptive fields are fused and the 512*20*20 feature map is fed into the spatial pyramid pooling layer (SPP) module to obtain a feature map of size 512*20*20.

[0128] (h) The 512*20*20 feature map is fed into the GhostBottleneck module with 512 convolution kernels of 3 and a stride of 1 to obtain a feature map of size 512*20*20;

[0129] (i) The 512*20*20 feature map is fed into a GhostConv module with 256 convolution kernels of size 1 and stride 1 to obtain a feature map of size 256*20*20;

[0130] (k) Upsample the 256*20*20 feature map to obtain a feature map of size 256*40*40, and then stack it with the 256*40*40 feature map mentioned in step (e) in the channel dimension through a skip connection layer to obtain a feature map of size 512*40*40;

[0131] (l) The 512*40*40 feature map is fed into the GhostBottleneck module with 256 convolution kernels of size 1 and stride 1 to obtain a feature map of size 256*40*40;

[0132] (m) The 256*40*40 feature map is fed into a GhostConv module with 128 convolution kernels of size 1 and stride 1 to obtain a 128*40*40 feature map.

[0133] (n) Upsampling the 128*40*40 feature map to obtain a feature map of size 128*80*80, and then stacking it with the 128*80*80 feature map in step (d) in the channel dimension through a skip connection layer to obtain a feature map of size 256*80*80;

[0134] (o) The 256*80*80 feature map is fed into the GhostBottleneck module with 128 convolution kernels of size 3 and stride 1 to obtain a feature map FM1 of size 128*80*80;

[0135] (p) The 128*80*80 feature map is fed into 128 depth-wise separable convolution (DSConv) modules with a kernel size of 3 and a stride of 2 to obtain a feature map of size 128*40*40.

[0136] (q) The 128*40*40 feature map is stacked with the 128*40*40 feature map mentioned in step (m) in the channel dimension through a skip connection layer to obtain a feature map of size 256*40*40;

[0137] (r) The 256*40*40 feature map is fed into the GhostBottleneck module with 256 convolution kernels of size 3 and stride 1 to obtain a feature map FM2 of size 256*40*40;

[0138] (s) The 256*40*40 feature map is fed into 256 depthwise separable convolution (DSConv) modules with a kernel size of 3 and a stride of 2 to obtain a feature map of size 256*20*20.

[0139] (t) The 256*20*20 feature map is stacked with the 256*20*20 feature map in step (i) in the channel dimension through a skip connection layer to obtain a feature map of size 512*20*20;

[0140] (u) The 512*20*20 feature map is fed into the GhostBottleneck module with 512 convolution kernels of size 3 and stride 1 to obtain a feature map FM3 of size 512*20*20;

[0141] (v) The feature map FM1 with a size of 128*80*80 proposed in step (o), the feature map FM2 with a size of 256*40*40 proposed in step (r), and the feature map FM3 with a size of 512*20*20 proposed in step (u) are fed into the detection head, corresponding to large target, medium target, and small target detection, respectively.

[0142] After testing, the mAP on the self-made dataset dropped from 0.805 to 0.764, and the FPS increased from 20 frames to 48 frames.

[0143] 3 The target trajectory is tracked and predicted through the improved P-DeepSORT multi-target tracking algorithm with parallel dual tracking. In order to solve the problem of target sequence jump caused by the original occlusion, a tracker dual-input parallel structure is designed to combine multi-angle feature information to ensure the stability of the tracking effect and reduce the misjudgment rate.

[0144] 3.1 Starting from prediction (Tracks), since the track Bbox at each moment is grouped together, it is equivalent to forming a set of tracks;

[0145] 3.2 After Kalman filter prediction, a trajectory Bbox is predicted for the current frame. The Bbox has two states: confirmed / unconfirmed, indicating that the trajectory Bbox is a confirmed target / unconfirmed target respectively;

[0146] 3.3 Detect the current frame, and then associate the detection Bboxs results with the predicted confirmedtrack Bbox (matched tracks);

[0147] 3.4.1 If the match is successful, update the tracking Bbox;

[0148] 3.4.2 If the match fails, consider the following situations:

[0149] (a) Track matching fails. Detection may have missed a target. The predicted track is still there at a certain moment, but the detector does not detect the corresponding target. Occlusion often causes this problem, so a parallel ReID module is used to search for possible matching targets from multiple angles.

[0150] (b) Detection matching fails. It is possible that an object has just entered the shot at a certain moment, and detection will fail to match tracks because the object is new and there is no track for prediction before. Another case is that the object is occluded for a long time, resulting in no matching track for the detected object.

[0151] To address the aforementioned matching failure issue, we perform an IOU match on the failed tracks and detections. If a match is successful, we update the track and continue with the prediction-observation-update tracking process. If the match still fails, we create a new track, initialize it to unconfirmed, and inspect it three times. If it is the actual target, we modify it to confirmed and then continue with the prediction-observation-update process.

[0152] 3.5 Repeat steps 3.1 to 3.4 for the next frame until the task is completed.

[0153] 4. The expert system matches the coordinates of the end point of the garbage disposal trajectory with the category of the garbage bin in the area. If the match is incorrect, it will be considered as the garbage being misplaced, and if the garbage is not placed in the area, it will be considered as garbage being spilled. Specifically, the following variables are set:

[0154] a.Objects.Hand indicates whether there is a hand in all detection boxes in the current frame;

[0155] b.ItemO .point birth Indicates the initial coordinates of the detection box;

[0156] c.Item O .class indicates the category of the detection box;

[0157] d.Areas bin It means demarcating the trash can area;

[0158] e.Area.class indicates the category of the current trash bin area;

[0159] f.Text False / Text True Indicates the prompt text of delivery error / correctness respectively;

[0160] g.List Show Indicates the text list to be displayed;

[0161] h.Item L .frame indicates the number of frames in which the text is displayed;

[0162] i.Frame max Indicates the maximum number of frames that the displayed text can exist;

[0163] j.Item L .json represents the data packet information sent to the server

[0164] Perform the following operations on each frame:

[0165] (1) Traverse all detection frames. If the hand or target initial coordinates are detected within the delivery domain, no processing is done and the next frame is entered. Otherwise, execute (2).

[0166] (2) If the tracking end coordinates are not within the delivery area, the delivery is determined to be garbage spillage and the next frame is entered. Otherwise, execute (3)

[0167] (3) If the detection target type does not match the delivery area type, the delivery error information is added to the judgment list; otherwise, the delivery correct information is added to the judgment list, and the alarm display duration is set to 0.

[0168] (4) Set the termination display duration to 2s and traverse the judgment list. If the alarm display duration is longer than the termination display duration, remove the delivery information from the judgment list and send the alarm information to the cloud server. Otherwise, keep the alarm information display until it is longer than the termination display duration.

[0169] The pseudo code of the process is as follows:

[0170]

[0171]

[0172] 5. The head-up camera C1 obtains the current height information h of the dropped object from the ground, and a specific height H (slightly higher than the trash can opening) is defined in advance. The downward camera C2 obtains the pixel width s′ of the detection target frame and divides it by the pixel width s of the defined trash can area to obtain the proportional coefficient k. The formula is as follows:

[0173]

[0174] Where k∈(0,1), when the height h of the center point of the object detected by the horizontal camera coincides with the preset height H, that is, h=H, the side length W of the trash can is known. t Calculate the actual width W of the dropped object using the following formula:

[0175] W=W t ×k

[0176] The distance D between the object and the camera is calculated using the camera monocular ranging principle. The formula is as follows:

[0177]

[0178] Where D is the distance from the detection object to the camera, W is the actual width of the detection object, F is the focal length of the camera, and P is the pixel value of the width of the detection object in the camera.

[0179] Finally, by calculating the height Hc of the camera C2 from the ground and the distance D of the target closest to the camera, the min The maximum garbage stacking height H in the current bucket is calculated by the difference max , the formula is as follows:

[0180] H max =H c -D min

[0181] When H max Exceeding the preset threshold H threshold When the trash can is considered to be overflowing, the threshold H threshold Set it to 70% of the trash can height.

[0182] 6 The edge computing device starts two processes: a main process and a subprocess, communicating through shared memory S. The subprocess uses the head-up camera C1 from step 5 as the input to the facial recognition module, outputs the name of the person delivering the object, N, and simultaneously obtains the height information h of the currently delivered object. The subprocess stores these information in shared memory S for the main process to access. The main process uses the head-down camera C2 from step 5 as the input for target detection and feeds the results into the tracker to obtain the type CL and trajectory information T of the delivered object. These information is then fed into the expert system to determine the result Code for the delivery action and the current timestamp T. Finally, the name of the person delivering the object, the delivery result Code, the current timestamp T, and the stack height h in the bucket are sent to the cloud platform for data aggregation.

[0183] During this period, the main process pushes the stream to the cloud Nginx server through the RTMP protocol to achieve real-time monitoring requirements of the process.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, this does not mean that the present invention is limited to this. Its users can still modify the technical solutions recorded in the above embodiments or replace some or all of the technical features therein by equivalents, and can also make further improvements and optimizations on the algorithm on this basis. However, it is worth noting that the basic framework and structure of the present invention will not deviate from the scope of the technical solutions of the various embodiments of the present invention with these modifications, replacements and innovations.

Claims

1. A method for designing an unmanned trash can system based on machine vision, characterized by: Step 1: Use the face detection algorithm to obtain the face detection boxes in the camera image and the facial key points landmarks corresponding to each detection box; send the boxes and their corresponding landmarks to the alignment network to output the aligned face image Image face ; Align the face image Image face Send it to the face recognition network to generate an n-dimensional feature vector embedding, which satisfies: embedding=(x1,x2,x3…x n ) As the basis for face comparison; finally, the generated facial feature vector embedding is compared with the registered facial feature vector set in the database For comparison, e i satisfy: <h2 style=";text-align:left;direction:ltr">e<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (y1,y2,y3…y)<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ) Calculate its Euclidean distance d: Select the value that satisfies the Euclidean distance d less than the preset threshold d threshold The name corresponding to the face information is taken as the final recognition result; threshold =1.24; Step 2 uses a deep neural network model to detect the target of the garbage to be placed and obtain information about the type of garbage to be placed. This is divided into the following four stages: (1) In the preprocessing stage, the input image is sliced, the sliced ​​image is convolved and reduced in dimension, and then sent to the backbone network for feature extraction; (2) In the feature map extraction stage, the backbone network consists of a deep separable convolution (DSConv) module and a ghost convolution bottleneck layer (SE-GhostBottleneck) module with a channel attention mechanism. The input of the SE-GhostBottleneck module goes through ghost convolution dimensionality reduction GhostConv – group convolution DWConv – channel weight adjustment SELayer – ghost convolution dimensionality increase GhostConv operation to output feature maps; The specific structure of the backbone network is as follows: (2-1) The input first passes through the DSConv module for convolution downsampling operation, and then is sent to the SE-GhostBottleneck module for feature extraction to obtain a shallow feature map; (2-2) The shallow feature map is sent to the DSConv module for convolution downsampling operation, and then sent to the 3-layer cascade SE-GhostBottleneck module for feature extraction to obtain a deeper feature map; (2-3) The deeper feature map is sent to the DSConv module for convolution downsampling, and then sent to the 3-layer cascade SE-GhostBottleneck module for feature extraction to obtain the deep feature map; (2-4) The deep feature map is sent to the spatial pyramid pooling layer SPP module for multiple receptive field fusion to improve the robustness of the model for detecting objects of different sizes, and the output feature map is sent to the channel attention module SELayer for channel weight adjustment, and its output result is used as the feature extraction result of the backbone network; (3) Feature fusion stage: For the feature maps extracted by the backbone network, based on the strong semantic features of the deep feature maps and the strong positioning features of the shallow feature maps, parameters of different detection layers are aggregated from different backbone layers to better fuse the features extracted by the backbone network. The specific structure is as follows: (3-1) First, the backbone network output feature map is sent to the GhostBottleneck module of the ghost convolution bottleneck layer to output the feature map f1; (3-2) Send the feature map f1 to the GhostConv module and output the feature map f2; (3-3) Upsample the feature map f2 and stack it with the deep feature map mentioned in (2-3) in the channel dimension through a skip connection layer to output the feature map f3; (3-4) Send the feature map f3 to the GhostBottleneck module of the ghost convolution bottleneck layer and output the feature map f4; (3-5) Send the feature map f4 to the GhostConv module and output the feature map f5; (3-6) Upsample the feature map f5 and stack it with the deeper feature map mentioned in (2-2) in the channel dimension through a skip connection layer to output the feature map f6; (3-7) The feature map f6 is fed into the GhostBottleneck module of the ghost convolution bottleneck layer, and the feature map FM1 is output; (3-8) Send the feature map FM1 to the depth-wise separable convolution DSConv module and output the feature map f7; (3-9) The feature map f7 is stacked with the feature map f4 mentioned in step (3-4) in the channel dimension through a skip connection layer to output the feature map f8; (3-10) Send the feature map f8 to the GhostBottleneck module of the ghost convolution bottleneck layer and output the feature map FM2; (3-11) Send the feature map FM2 to the depth-wise separable convolution DSConv module and output the feature map f9; (3-12) The feature map f9 is stacked with the feature map f2 mentioned in (3-2) in the channel dimension through a skip connection layer, and the feature map f10 is output; (3-13) The feature f10 map is fed into the GhostBottleneck module of the ghost convolution bottleneck layer, and the feature map FM3 is output; (4) In the target detection stage, a three-level detection head is set up, corresponding to the detection targets of small, medium and large scales respectively; the feature maps FM1, FM2 and FM3 are sent to the detection head, and the bounding box is regressed according to the preset anchor box to achieve target detection; Step 3 uses a multi-target tracking algorithm to track and predict the target trajectory. In order to solve the problem of target sequence jump caused by the original occlusion, a tracker dual-input parallel structure is designed to combine multi-angle feature information to ensure the stability of the tracking effect and reduce the misjudgment rate. The specific process is as follows: (1) Combine the tracking frames at each moment to form a set of trajectories; (2) Predict a tracking frame for the current frame through filtering operation. The tracking frame has two states: a certain state and an uncertain state. (3) Detect the current frame and then associate the detection frame with the tracking frame. Two situations may occur at this time: (a) Matching is successful, and the tracking frame is updated; (b) If the matching fails, the tracking box and the detection box that failed to match are matched by IOU. If the matching succeeds, they are updated and the prediction-observation-update tracking process is continued. If it still fails, the feature vector of the original tracked target is matched with the feature vector of the auxiliary view detection target: If the match is successful, the tracking target is considered to be not lost and the tracking information is retained. Otherwise, a new tracking frame is established for it and initialized to an uncertain state. IOU matching is performed on the next few frames. If the match is successful, the tracking frame is set to a confirmed state and the prediction-observation-update tracking process continues. (4) Execute the above steps (1) to (3) for the next frame until the task is completed; In step 4, the expert system matches the coordinates of the end point of the garbage trajectory with the type of garbage bin in the area. An incorrect match is considered as misplaced garbage, and failure to enter the area is considered as garbage spillage. Specifically, the following operations are performed for each frame of the image: (4.1) Traverse all detection frames. If the hand or target initial coordinates are detected within the delivery domain, no processing is done and the next frame is entered. Otherwise, execute (4.2). (4.2) If the tracking end coordinates are not within the delivery area, the delivery is determined to be garbage spillage and the next frame is entered. Otherwise, execute (4.3) (4.3) If the detection target type does not match the delivery area type, the delivery error information is added to the judgment list; otherwise, the delivery correct information is added to the judgment list and the alarm display duration is set to 0; (4.4) Set the termination display duration to 2s and traverse the judgment list. If the alarm display duration is longer than the termination display duration, remove the delivery information from the judgment list and send the alarm information to the cloud server. Otherwise, keep the alarm information display until it exceeds the termination display duration. In step 5, the head-up camera C1 obtains the current height h of the dropped object from the ground and defines a specific height H in advance. The head-down camera C2 obtains the pixel width s′ of the detection target frame and divides it by the pixel width s of the defined trash can area to obtain the proportional coefficient k. The formula is as follows: Where k∈(0,1), when the height h of the center point of the object detected by the horizontal camera coincides with the preset height H, that is, h=H, the side length W of the trash can is known. t Calculate the actual width W of the dropped object using the following formula: W=W t ×k The distance D between the object and the camera is calculated using the camera monocular ranging principle. The formula is as follows: Where D is the distance from the detection object to the camera, W is the actual width of the detection object, F is the focal length of the camera, and P is the pixel value of the width of the detection object in the camera; Finally, by calculating the height H of the camera C2 from the ground at a known viewing angle c The distance D to the target closest to the camera min The maximum garbage stacking height H in the current bucket is calculated by the difference max , the formula is as follows: H max =H c -D min And set the threshold height H threshold satisfy: H threshold =H can ×α Among them H can is the total height of the trash can, α is the overflow coefficient, take α=0.7, when H max Exceeding the threshold height H threshold When , the trash can is considered to be overflowing; In step 6, the edge computing device starts two processes, a main process and a sub-process, and implements process communication through shared memory S. The sub-process uses the head-up camera C1 in step 5 as the input of the face recognition module, outputs the name N of the person who placed the object, and obtains the height information h of the current object. Finally, the name N and height information h are stored in the shared memory S for the main process to call. The main process uses the overhead camera C2 in step 5 as the input for target detection and sends the result to the tracker to obtain the type CL and trajectory information T of the dropped object. CL and T are then input into the expert system to obtain the judgment result Code for this drop action and the current timestamp T. Finally, the name N of the dropper, the drop result Code, the current timestamp T, and the stack height h in the bucket are sent to the cloud platform for data aggregation. During this period, the main process pushes the stream to the cloud Nginx server via the RTMP protocol to achieve real-time monitoring requirements of the process.

Images