A DeepSort garden waste statistics method based on drone motion status
By introducing Kalman filter and feature fusion matching in the DeepSort algorithm, the problem of low accuracy of garbage tracking statistics caused by changes in the motion state of the drone is solved, and the accurate statistics and assessment standards for garden garbage are improved.
Patent Information
- Application Number
- CN202210323197.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-29
AI Technical Summary
When the drone's movement state changes, the existing DeepSort algorithm leads to low accuracy in garden garbage tracking statistics, and it is impossible to accurately assess the inspection results.
Using the DeepSort garden garbage statistics method based on the motion state of the drone, the Kalman filter and cost matrix matching is used, and the adaptive fusion of SIFT and CNN features is improved, and the IOU cost matrix and fuzzy evaluation index are improved, and the matching between the prediction box and the detection box is improved. accuracy.
It has realized the accurate tracking and accurate statistics of garbage in the videos taken by drone, and improved the assessment standards for garden management systems and cleaning personnel.
Smart Images

Figure CN114821364B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of target tracking and computer vision, and in particular relates to a DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle. Background Art
[0002] With the construction of spiritual civilization, the overall quality of our citizens has been continuously improved, and environmental awareness has gradually increased. However, there are also some phenomena of littering due to non-compliance with public order and lack of awareness of urban norms. Therefore, in order to ensure the environment of the garden and improve the tourists' experience in the garden, it is necessary to maintain and manage the garden. However, there are too many links in the existing garden management system that rely on human operation. For example, for the disposal of garbage in the garden, the cleaning staff generally conduct uninterrupted inspections of the responsible area, clean up the garbage, and ensure the cleanliness of the garden environment. However, manual maintenance also has disadvantages. Due to the large area of the garden, a lot of manpower and material resources are required to ensure that the garden environment is always clean; and the current assessment standards for the inspection results of garden cleaning staff still need to be further improved; most of the smart gardens now use drones to dynamically monitor the situation in the garden, and then use algorithms such as target tracking to intelligently analyze the monitoring information. With the rapid development of deep learning in the field of computer vision, target tracking technology as a part of the field of computer vision has also made breakthrough developments. However, drones are controlled by humans to collect information, so they may suddenly accelerate, decelerate, rise, or fall. Such sudden acceleration, deceleration, and rise will cause blurred images, sudden changes in target states, and changes in target sizes, which will lead to inaccurate prediction of garbage prediction frames and errors in matching prediction frames with detection frames. If the DeepSort tracking algorithm is used for garden garbage statistics, its accuracy is often limited by the effectiveness of feature extraction and the accuracy of garbage frame prediction. Therefore, if the original DeepSort tracking method is used, garbage tracking statistics will be inaccurate, and the inspection results cannot be accurately evaluated. Summary of the invention
[0003] In order to solve the technical problems mentioned in the above background technology, the present invention proposes a novel DeepSort garbage statistics method based on the motion state of a drone to solve the problem of low accuracy of DeepSort in tracking and counting garden garbage.
[0004] In order to achieve the above technical objectives, the technical solution of the present invention is:
[0005] A DeepSort garden waste statistics method based on the motion state of a drone includes the following steps:
[0006] (1) Use drones to collect video information of the garden and process it frame by frame;
[0007] (2) Input the video frame generated in step 1 into the object recognition network to obtain the garbage detection box information;
[0008] (3) Initialize the Kalman filter. The tracker predicts the position of the detection frame of the next frame based on the position of the detection frame of the current frame, generates a prediction frame, and then matches the prediction frame with the detection frame of the next frame according to the cost matrix. When the matching result is a tracker mismatch, if the tracker is in an uncertain state, the tracker is directly deleted. If the tracker is in a certain state and reaches a certain number of times in a row, the tracker is deleted. When the matching result is a detection frame mismatch, the detection frame is initialized as a new tracker. When the matching result is a successful pairing of the detection frame and the prediction frame, the detection frame corresponding to the successfully matched prediction frame is updated through the Kalman filter to its corresponding tracker. Repeat step (3) until a confirmed tracker appears or the video frame ends.
[0009] (4) When the uncertain tracker in step (3) successfully matches three consecutive frames, the tracker changes to the certain state, and the prediction box predicted by the certain state tracker is cascade matched with the detection box. When the matching result is that the tracker matches, the corresponding tracker is updated through Kalman filtering. When the matching result is that the detection box and the tracker do not match, the previous uncertain state tracker and the mismatched tracker are matched together with the mismatched detector according to the cost matrix;
[0010] (5) When the result of the cost matrix matching in step (4) is a tracker mismatch, if the tracker is in an uncertain state, the tracker is directly deleted; if the tracker is in a certain state, it is deleted only after a certain number of consecutive times; when the matching result is a detection frame mismatch, the detection frame is initialized as a new tracker; when the matching result is a successful pairing of the detection frame and the prediction frame, the corresponding detection frame is updated through the Kalman filter to its corresponding tracker; repeat steps (4) and (5) until the end of the video.
[0011] Preferably, the step (1) further comprises filtering each frame of the image.
[0012] Preferably, the target recognition network in step (2) adopts the YOLOv5 network model.
[0013] Preferably, in step (3), the predicted frame is matched with the detection frame of the next frame according to a cost matrix, and the cost matrix formula is as follows:
[0014]
[0015] Among them, P(x P ,yP,w P ,l P ,h P ,v xP ,vyP ,v hP ) represents the prediction box information,
[0016] T(x T ,yT,w T ,l T ,h T ,v xT ,v yT ,v hT ) represents the detection box information; S area Indicates the overlapping area between the prediction box and the detection box; w P 、w T Represent the width of the prediction box and the detection box respectively; l P , l T Respectively represent the length of the prediction box and the detection box; v xP 、v yP They respectively represent the flight speed in the horizontal and vertical directions of the drone when it collects the previous frame of the image where the detection frame is located; v xT 、v yT The horizontal and vertical flight speeds of the drone when collecting the image where the detection frame is located; t0 represents the time difference between the previous frame image of the drone when collecting the image where the detection frame is located and the time difference between the drone when collecting the image where the detection frame is located; (x P ,y P )、(x T ,y T ) represent the center coordinates of the prediction box and the detection box respectively.
[0017] Preferably, in step (4), the appearance information in the cascade matching is obtained by adaptively fusing CNN features and SIFT features;
[0018] Perform feature point detection on the prediction box and the detection box, and fuse the SIFT feature and CNN feature according to the adaptive fusion coefficient. The formula of the adaptive fusion coefficient is as follows:
[0019]
[0020] in, Indicates the absolute value of the altitude change rate of the drone;
[0021] The coordinates of a feature point detected by the SIFT algorithm are (x, y), the SIFT feature descriptor is calculated, and then a 64×64 image block is intercepted with the feature point as the center. The CNN model is used to extract the CNN feature descriptor of the image block, and finally the Euclidean distance is used to calculate a feature point P of the detection box. t and a feature point P of the prediction box p The Euclidean distance between them, that is, the similarity, is as follows:
[0022]
[0023] Among them, L s , L c Represent the dimensions of SIFT and CNN feature descriptors respectively. The SIFT feature descriptor of a feature point in the detection box is The SIFT descriptor of a feature point in the prediction box is The CNN feature descriptor of a feature point in the detection box is The CNN descriptor of a feature point in the prediction box is
[0024] A method for obtaining a prediction frame of a blurred image comprises the following steps:
[0025] S1, dividing the current frame image into Z local images of size N×N, and performing fuzzy judgment on the local images;
[0026] S2, according to the local image and area D area The intersection-and-union ratio is used for judgment. If the intersection-and-union ratio is greater than 0, the prediction box information of the next frame is determined according to the motion state of the previous frame and the position state of the current frame; if the intersection-and-union ratio is less than 0, the prediction box is predicted according to the steps of the DeepSort algorithm.
[0027] Preferably, the fuzzy judgment method in step S1 is as follows:
[0028] The current frame image being tracked is divided into Z local images of size N×N. The set of local images after segmentation is assumed to be V = {v1,v1,...,v Z}, calculate the maximum gradient value of each local image and compare it with the threshold, the maximum gradient value G xy The formula is:
[0029]
[0030]
[0031]
[0032] Where x and y represent the position of a pixel in the local image; o is an image channel and belongs to one of the three channels {R, G, B}; I is the pixel set of the local image of N×N; represents the gradient operator; when the maximum gradient value is less than or equal to the threshold, the local image is considered to be blurred; let the set of local images judged to be blurred, that is, the blurred area of the current frame image, be U = {u1,u2,...,u n},u nrepresents the nth blurred partial image.
[0033] Preferably, the method for determining whether the blurred local image set has an impact on target recognition in step S2 is as follows:
[0034] Assume the center of the prediction box is the center of the circle, R o The circular area with radius D area , using D area Traverse the set U and calculate u i With D area The intersection-over-union ratio, R o The calculation formula is as follows:
[0035]
[0036] Among them, l p 、w p They are the length and width of the prediction box respectively;
[0037] Traverse the set U. If the intersection-and-union ratio calculated in S2 is greater than 0, it is considered that the blurred area in the frame image has an impact on target recognition. The predicted frame obtained by the Kalman filter prediction of the previous frame tracker is used as the detection frame of the frame image. The Kalman filter updates the target state, and then the updated target state is used to predict the prediction frame information of the next frame. The motion state of the Kalman filter is the same as that of the previous frame, and the position state is determined by the detection frame of the frame. If both are equal to 0, it is considered that the blurred area in the frame image has no impact on target recognition, and normal tracking is performed according to the steps of the DeepSort algorithm.
[0038] The beneficial effects brought by adopting the above technical solution are:
[0039] In the smart garden scene, the garbage in the video information taken by the drone can be accurately tracked, thereby achieving accurate statistics of the garbage and improving the assessment standards of the garden management system and cleaning staff. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is the flow chart of the tracking algorithm;
[0041] Figure 2 Confirmation state tracker matching process DETAILED DESCRIPTION
[0042] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
[0043] The tracking algorithm flow chart proposed by the present invention is as follows: Figure 1As shown, the method proposed in the present invention mainly includes three contents: first, to address the problem that the target size changes when the drone rises and descends, resulting in inaccurate matching between the prediction frame and the detection frame, a SIFT feature matching fusion based on the height change rate is proposed; second, to address the problem of sudden acceleration and deceleration of the drone, the IOU cost matrix is improved, the accuracy of matching between the detection frame and the prediction frame is improved, and the situation of multiple inspections is reduced; third, when image blur affects the matching, a prediction frame acquisition method based on an improved blur evaluation index is proposed.
[0044] 1. SIFT feature matching fusion based on height change rate
[0045] In cascade matching, appearance information is obtained by adaptively fusing CNN features with SIFT features. The flowchart is as follows: Figure 2 As shown, firstly, feature point detection is performed on the prediction box and the detection box, and SIFT features and CNN features are fused according to the adaptive fusion coefficient. The formula of the adaptive fusion coefficient is as follows:
[0046]
[0047] in, It represents the altitude change rate of the UAV. The greater the change in the UAV's flight altitude, the greater the change in the target's scale. The smaller the change in the flight altitude, the smaller the change in the target's scale.
[0048] The coordinates of a feature point detected by the SIFT algorithm are (x, y), the SIFT feature descriptor is calculated, and then a 64×64 image block is intercepted with the feature point as the center. The CNN model is used to extract the CNN feature descriptor of the image block, and finally the Euclidean distance is used to calculate a feature point P of the detection box. t and a feature point P of the prediction box p The Euclidean distance between them, that is, the similarity, is as follows:
[0049]
[0050] Among them, L s , L c Represent the dimensions of SIFT and CNN feature descriptors respectively. The SIFT feature descriptor of a feature point in the detection box is The SIFT descriptor of a feature point in the prediction box is The CNN feature descriptor of a feature point in the detection box is The CNN descriptor of a feature point in the prediction box is
[0051] The smaller d is, the closer the descriptors of the two feature points are, and the more similar the features are. Traverse the feature points of the detection frame and the prediction frame, and calculate the Euclidean distance between the descriptor of a feature point in the detection frame and the descriptor of a feature point in the prediction frame.
[0052] 2. Improve the IOU cost matrix
[0053] The improved cost matrix formula is as follows:
[0054]
[0055] Among them, using P(x P ,y P , wP ,l P ,h P ,v xP ,v yP ,v hP ) represents the prediction box information, using T(x T ,y T ,w T ,l T ,h T ,v xT ,v yT ,v hT ) represents the detection box information. S area Indicates the overlapping area between the prediction box and the detection box; w P , w T Represent the width of the prediction box and the detection box respectively; l P , l T Respectively represent the length of the prediction box and the detection box; v xP , v yP They represent the forward flight speed and the vertical flight speed of the drone when collecting the previous frame of the image where the detection frame is located; v xT , v yT The forward flight speed and the vertical forward flight speed of the drone when collecting the image where the detection frame is located are respectively; t0 represents the time difference between the previous frame image of the drone when collecting the image where the detection frame is located and the time difference between the drone when collecting the image where the detection frame is located; (x P ,y P ), (x T ,y T ) represent the center coordinates of the prediction box and the detection box respectively.
[0056] 3. Prediction box acquisition method based on improved fuzzy evaluation index
[0057] DeepSort uses Kalman filtering to predict the position of the detection frame of the next frame. However, due to the blurriness of some areas of the image, some small targets may not be recognized successfully, and there is no detection frame matching the predicted frame, which leads to tracking failure and multiple detections. To address the above problems, this paper proposes to pre-check the image before tracking to determine whether the image blur affects the subsequent target tracking. The pre-check steps are as follows:
[0058] Step 1: Divide the image into Z local images of size N×N. Let the set of local images after segmentation be V = {v1,v1,...,v Z}, and calculate the maximum gradient value of each local image according to the following formula and compare it with the threshold to determine whether the local image is blurred. Let the set of local images determined to be blurred be U = {u1,u2,...,u n}.
[0059]
[0060]
[0061]
[0062] Step 2: Set the center position of the prediction box of the previous frame image as the center of the circle, and o The circular area with radius D area , where R o As shown in the following formula, traverse the set U and calculate u i With D area The intersection and ratio of .
[0063]
[0064] Among them, l p , w p are the length and width of the prediction box respectively.
[0065] Step 3: Traverse the image set U. If the intersection-and-union ratio calculated in step 2 is greater than 0, it is considered that the blurred area in the frame image affects the target recognition, so this prediction box is used as the detection box of the frame image, the Kalman filter updates the target state, and then the updated target state is used to predict the prediction box information of the next frame, where the motion state of the Kalman filter is the same as the previous frame, and the position state is determined by the detection box of the frame. If they are all equal to 0, it is considered that the blurred area in the frame image does not affect the target recognition, and the tracking is normal according to the steps of the DeepSort algorithm.
[0066] The embodiments are only for illustrating the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.
Claims
1. A DeepSort garden waste statistics method based on the motion state of drones, characterized in that: The following steps are involved: (1) Use drones to collect video information of the garden and process it frame by frame; (2) Input the video frame generated in step 1 into the object recognition network to obtain the garbage detection box information; (3) Initialize the Kalman filter. The tracker predicts the position of the detection frame of the next frame based on the position of the detection frame of the current frame, generates a prediction frame, and then matches the prediction frame with the detection frame of the next frame according to the cost matrix. When the matching result is a tracker mismatch, if the tracker is in an uncertain state, the tracker is directly deleted. If the tracker is in a certain state and reaches a certain number of times in a row, the tracker is deleted. When the matching result is a detection frame mismatch, the detection frame is initialized as a new tracker. When the matching result is a successful pairing of the detection frame and the prediction frame, the detection frame corresponding to the successfully matched prediction frame is updated through the Kalman filter to its corresponding tracker. Repeat step (3) until a confirmed tracker appears or the video frame ends. (4) When the uncertain tracker in step (3) successfully matches three consecutive frames, the tracker changes to the certain state, and the prediction box predicted by the certain state tracker is cascade matched with the detection box. When the matching result is that the tracker matches, the corresponding tracker is updated through Kalman filtering. When the matching result is that the detection box and the tracker do not match, the previous uncertain state tracker and the mismatched tracker are matched together with the mismatched detector according to the cost matrix; (5) When the result of the cost matrix matching in step (4) is a tracker mismatch, if the tracker is in an uncertain state, the tracker is directly deleted; if the tracker is in a certain state, it is deleted only after a certain number of consecutive times; when the matching result is a detection frame mismatch, the detection frame is initialized as a new tracker; when the matching result is a successful pairing of the detection frame and the prediction frame, the corresponding detection frame is updated through the Kalman filter to its corresponding tracker; repeat steps (4) and (5) until the end of the video.
2. According to claim 1, a DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle is characterized in that: The step (1) also includes filtering each frame of the image.
3. According to the DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle according to claim 1, it is characterized in that: In the step (2), the target recognition network adopts the YOLOv5 network model.
4. According to the DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle according to claim 1, it is characterized in that: In step (3), the predicted frame is matched with the detection frame of the next frame according to the cost matrix, and the cost matrix formula is as follows: Among them, P(x P ,y P ,w P ,l P ,h P ,v xP ,v yP ,v hP ) represents the prediction box information, T(x T ,y T ,w T ,l T ,h T ,v xT ,v yT ,v hT ) represents the detection box information; S area Indicates the overlapping area between the prediction box and the detection box; w P 、w T Represent the width of the prediction box and the detection box respectively; l P , l T Respectively represent the length of the prediction box and the detection box; v xP 、v yP They respectively represent the flight speed in the horizontal and vertical directions of the drone when it collects the previous frame of the image where the detection frame is located; v xT 、v yT The horizontal and vertical flight speeds of the drone when collecting the image where the detection frame is located; t0 represents the time difference between the previous frame image of the drone when collecting the image where the detection frame is located and the time difference between the drone when collecting the image where the detection frame is located; (x P ,y P )、(x T ,y T ) represent the center coordinates of the prediction box and the detection box respectively.
5. The DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle according to claim 1 is characterized in that: In the step (4), the appearance information in the cascade matching is obtained by adaptively fusing the CNN features and the SIFT features; Perform feature point detection on the prediction box and the detection box, and fuse the SIFT feature and CNN feature according to the adaptive fusion coefficient. The formula of the adaptive fusion coefficient is as follows: in, Indicates the absolute value of the altitude change rate of the drone; The coordinates of a feature point detected by the SIFT algorithm are (x, y), the SIFT feature descriptor is calculated, and then a 64×64 image block is intercepted with the feature point as the center. The CNN model is used to extract the CNN feature descriptor of the image block, and finally the Euclidean distance is used to calculate a feature point P of the detection box. t and a feature point P of the prediction box p The Euclidean distance between them, that is, the similarity, is as follows: Among them, L s , L c Represent the dimensions of SIFT and CNN feature descriptors respectively. The SIFT feature descriptor of a feature point in the detection box is The SIFT descriptor of a feature point in the prediction box is The CNN feature descriptor of a feature point in the detection box is The CNN descriptor of a feature point in the prediction box is 6. A method for obtaining a fuzzy image prediction frame of the DeepSort garden waste statistics method based on the motion state of an unmanned aerial vehicle according to claim 1, characterized in that: The following steps are involved: S1, dividing the current frame image into Z local images of size N×N, and performing fuzzy judgment on the local images; S2, according to the local image and area D area The intersection-and-union ratio is used for judgment. If the intersection-and-union ratio is greater than 0, the prediction box information of the next frame is determined according to the motion state of the previous frame and the position state of the current frame; if the intersection-and-union ratio is less than 0, the prediction box is predicted according to the steps of the DeepSort algorithm.
7. The method for obtaining a blurred image prediction frame according to claim 6, characterized in that: The fuzzy judgment method in step S1 is as follows: The current frame image being tracked is divided into Z local images of size N×N. The set of local images after segmentation is assumed to be V = {v1,v1,...,v Z }, calculate the maximum gradient value of each local image and compare it with the threshold, the maximum gradient value G xy The formula is: Where x and y represent the position of a pixel in the local image; o is an image channel and belongs to one of the three channels {R, G, B}; I is the pixel set of the local image of N×N; represents the gradient operator; when the maximum gradient value is less than or equal to the threshold, the local image is considered to be blurred; let the set of local images judged to be blurred, that is, the blurred area of the current frame image, be U = {u1,u2,...,u n },u n represents the nth blurred partial image.
8. The method for obtaining a blurred image prediction frame according to claim 7, characterized in that: The method for determining whether the blurred local image set has an impact on target recognition in step S2 is as follows: Assume that the center position of the prediction box is the center of the circle, R o The circular area with radius D area , using D area Traverse the set U and calculate u i With D area The intersection-over-union ratio, R o The calculation formula is as follows: Among them, l p 、w p They are the length and width of the prediction box respectively; Traverse the set U. If the intersection-and-union ratio calculated in S2 is greater than 0, it is considered that the blurred area in the frame image has an impact on target recognition. The predicted frame obtained by the Kalman filter prediction of the previous frame tracker is used as the detection frame of the frame image. The Kalman filter updates the target state, and then the updated target state is used to predict the prediction frame information of the next frame. The motion state of the Kalman filter is the same as that of the previous frame, and the position state is determined by the detection frame of the frame. If both are equal to 0, it is considered that the blurred area in the frame image has no impact on target recognition, and normal tracking is performed according to the steps of the DeepSort algorithm.
