A vision-based littering detection method and system

Abstract

The present invention relates to the field of urban management technology, specifically a method and system for detecting littering based on vision, comprising the following steps: a video acquisition step: performing video acquisition through a video acquisition device, and dividing the video data into a previous video frame f1 and a subsequent video frame f2 in chronological order, and transmitting them to a target detector for detection; a step of identifying newly added garbage: the target detector detects garbage on pedestrians' hands and garbage on the ground in f1 and f2, and after detecting the garbage on pedestrians' hands in f1 and not detecting the garbage on pedestrians' hands in f2, it is determined that there is newly added garbage. The present invention determines whether there is littering behavior by comparing the vector features of the garbage on the hands and the garbage on the ground, so that the detection accuracy is higher, it is applicable to real life, the detection process is simple, the implementation cost is low, and the face detection network can record face information and can identify people who have repeatedly littered, so that they can be educated in a targeted manner.

Description

Technical Field

[0001] The present invention relates to the technical field of urban management, and in particular to a vision-based littering detection method and system. Background Art

[0002] At present, my country is promoting the construction of civilized cities and beautifying the urban living environment. This is what the government has been doing. However, with the deepening of urbanization, people's quality problems cannot keep up with the pace of urbanization. Littering is still everywhere, which not only affects the appearance of the city, but also endangers our daily living environment. Urban environmental issues have become an important issue that the government needs to solve. In the past, the more common practice was that managers posted warm reminders in public places, or relied on people's consciousness, but these were not restrictive; the only restrictive and common practice was to send special people to supervise places where garbage was commonly thrown and educate people who littered. However, this method is too labor-intensive, inefficient and has no unified standards; now that the city is being promoted in the digitalization process, this human-based approach to solving the problem of littering is obviously not in line with the needs of urban digital development. The real need for urban development is to monitor littering based on artificial intelligence automation.

[0003] Chinese Patent No. CN202010485156.9 provides a method for detecting littering based on urban management surveillance video. First, the urban management surveillance video is checked for road garbage at regular intervals. If road garbage is found, the surveillance video in the previous period is checked back and analyzed to find the pedestrians who littered. If no pedestrians who littered are found, the pedestrians who littered are searched in the videos in the previous period of time of the cameras around them. In this way, littering in the surveillance video is detected.

[0004] The existing technology uses cameras to detect littering with low accuracy, resulting in an inability to accurately identify littering behavior. Therefore, it is urgent to design a vision-based littering detection method and system to solve the above problem. Summary of the invention

[0005] The purpose of the present invention is to provide a vision-based littering detection method and system to solve the problem of detection accuracy raised in the above background technology.

[0006] The technical solution of the present invention is: a method for detecting littering based on vision, comprising the following steps:

[0007] Video acquisition step: Video is acquired through a video acquisition device, and the video data is divided into a previous video frame f1 and a next video frame f2 in chronological order and transmitted to the target detector for detection;

[0008] Identify new garbage step: The object detector detects garbage on the hands of pedestrians and garbage on the ground in f1 and f2, and when garbage on the hands of pedestrians is detected in f1 and garbage on the hands of pedestrians is not detected in f2, it is confirmed that there is new garbage;

[0009] Confirming the newly added garbage step: Process and compare the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 to confirm the newly added ground garbage;

[0010] Garbage feature extraction step: extract feature vectors of garbage on hand and garbage on the ground respectively through garbage feature extractor;

[0011] Feature comparison step: The feature vectors of the garbage on hand and the garbage on the ground are normalized by the garbage feature extractor, and the cosine similarity of the two vectors is calculated. When the similarity is greater than the set threshold, it is determined that there is littering behavior;

[0012] Face detection step: After detecting littering behavior, the face is detected by the face detector and the face image is cropped;

[0013] Face feature extraction step: extract the feature vector of the face through the face feature extractor and upload it to the background face database;

[0014] Face data search steps: Calculate the cosine similarity between the uploaded face feature vector and the background face database. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times.

[0015] Furthermore, in the face data search step, when the cosine similarity of the face feature vector is less than or equal to the set threshold, it is determined that this person is littering for the first time. In the step of confirming the new garbage, when the garbage on the ground in f1 is zero, the coordinate information of the garbage on the ground is empty.

[0016] Furthermore, in the step of identifying newly added garbage, the target detector detects garbage on the hands of pedestrians and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, and the detection result in f2 is tracked, after no garbage on the hands is detected in f2 and newly added garbage is detected on the ground, the step of confirming newly added garbage is entered.

[0017] Furthermore, in the step of confirming the newly added garbage, assuming that there are N garbage on the ground in f1, then the garbage on the ground in f1 is g1, g2, ..., gn, and the garbage on the ground in f2 is g1, g2, ..., gn+1, and n is a natural number.

[0018] Furthermore, in the step of extracting garbage features, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted according to the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

[0019] Furthermore, in the step of extracting garbage features, the feature vectors of garbage on hands and garbage on the ground are 256-dimensional feature vectors, and in the step of extracting face features, the feature vector extracted from the face is a 256-dimensional feature vector.

[0020] Furthermore, after detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When the littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

[0021] Furthermore, after no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t is continued and the next frame is entered to continue the step of identifying new garbage.

[0022] Furthermore, in the feature comparison step, the feature vector obtained in the step of extracting garbage features is normalized, and then the cosine similarity of the two vectors is calculated. Assuming that the feature vector is g = (x1, x2, x3, .., x256), the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

[0023] Furthermore, the cosine similarity formula of the garbage on hand and the garbage on the ground is: cos <g hand ^,g add ^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

[0024] A vision-based litter detection system comprising:

[0025] Video acquisition module: Video is acquired through a video acquisition device, and the video data is divided into the previous video frame f1 and the next video frame f2 according to the time sequence and transmitted to the target detector for detection;

[0026] New garbage identification module: The object detector detects garbage on pedestrians’ hands and garbage on the ground in f1 and f2. When garbage on pedestrians’ hands is detected in f1 and garbage on pedestrians’ hands is not detected in f2, it is confirmed that there is new garbage.

[0027] Confirmation of newly added garbage module: the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 are processed and compared to confirm the newly added ground garbage;

[0028] Garbage feature extraction module: The garbage feature extractor is used to extract the feature vectors of garbage on the hand and garbage on the ground respectively;

[0029] Feature comparison module: The garbage feature extractor processes the feature vectors of garbage on hand and garbage on the ground into normalized feature vectors, calculates the cosine similarity of the two vectors, and identifies littering when the similarity is greater than the set threshold;

[0030] Face detection module: After detecting littering, the face detector detects the face and crops the face image;

[0031] Face feature extraction module: extracts the feature vector of the face through the face feature extractor and uploads it to the background face database;

[0032] Face data search module: calculate the cosine similarity between the uploaded face feature vector and the background face database. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times.

[0033] Furthermore, in the face data search module, when the cosine similarity of the face feature vector is less than or equal to the set threshold, it is determined that this person is littering for the first time. In the new garbage confirmation module, when the ground garbage in f1 is zero, the coordinate information of the garbage on the ground is empty.

[0034] Furthermore, in the module for identifying newly added garbage, the target detector detects garbage on the hands of pedestrians and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, and the detection result in f2 is tracked, when no garbage on the hands is detected in f2 and newly added garbage is detected on the ground, the module for confirming newly added garbage is entered.

[0035] Furthermore, in the module for confirming the newly added garbage, assuming that there are N garbage on the ground in f1, then the garbage on the ground in f1 is g1, g2, ..., gn, and the garbage on the ground in f2 is g1, g2, ..., gn+1, and n is a natural number.

[0036] Furthermore, in the garbage feature extraction module, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted according to the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

[0037] Furthermore, in the garbage feature extraction module, the feature vectors of garbage on hands and garbage on the ground are 256-dimensional feature vectors, and in the face feature extraction module, the feature vector of the extracted face is a 256-dimensional feature vector.

[0038] Furthermore, after detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When the littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

[0039] Furthermore, after no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t is continued and the next frame is entered to continue to identify the new garbage module.

[0040] Furthermore, in the feature comparison module, the feature vector obtained in the garbage feature extraction module is normalized, and then the cosine similarity of the two vectors is calculated. Assuming that the feature vector is g = (x1, x2, x3, .., x256), the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

[0041] Furthermore, the cosine similarity formula of the garbage on hand and the garbage on the ground is: cos <g hand ^,g add ^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

[0042] The present invention provides a visual-based litter detection method and system through improvements, which have the following improvements and advantages compared with the prior art:

[0043] (1) The present invention determines whether there is littering by comparing the vector features of garbage on the hand and garbage on the ground, so that the detection accuracy is higher, it is applicable to real life, the detection process is simple, and the implementation cost is low.

[0044] (2) The present invention can record facial information and identify people who litter multiple times through a face detection network, and can provide targeted education to them.

[0045] (3) The present invention identifies the newly added garbage on the ground and then identifies the person who threw the garbage, making the identification of the litterer more accurate and preventing misidentification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments:

[0047] Figure 1 is a flow chart of an embodiment of the present invention;

[0048] Figure 2 yoloV5 target detection network of an embodiment of the present invention, respectively detecting garbage on the hand and garbage on the ground;

[0049] Figure 3 Schematic diagram of feature extraction of hand garbage and ground garbage detected by the EfficientNet-b3 feature extraction network of an embodiment of the present invention;

[0050] Figure 4 This is a flowchart of facial information recognition for multiple littering cases according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the attached Figures 1 to 4 The present invention is described in detail, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0053] A method for detecting littering based on vision in an embodiment of the present invention is as follows: Figure 1 to Figure 4 As shown, the following steps are included:

[0054] Video acquisition step: Video is acquired through a video acquisition device, and the video data is divided into a previous video frame f1 and a next video frame f2 in chronological order and transmitted to the target detector for detection;

[0055] Identify new garbage step: The object detector detects garbage on the hands of pedestrians and garbage on the ground in f1 and f2, and when garbage on the hands of pedestrians is detected in f1 and garbage on the hands of pedestrians is not detected in f2, it is confirmed that there is new garbage;

[0056] Confirming the newly added garbage step: Process and compare the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 to confirm the newly added ground garbage;

[0057] Garbage feature extraction step: extract feature vectors of garbage on hand and garbage on the ground respectively through garbage feature extractor;

[0058] Feature comparison step: The feature vectors of the garbage on the hand and the garbage on the ground are normalized by the garbage feature extractor. The feature vectors of the garbage on the hand and the garbage on the ground are compared by normalized feature vector processing, and the cosine similarity of the two vectors is calculated. When the similarity is greater than the set threshold, it is determined that there is littering behavior. When littering behavior is identified, an alarm can be issued in time;

[0059] Face detection step: After detecting littering behavior, the face is detected by the face detector and the face image is cropped;

[0060] Face feature extraction step: extract the feature vector of the face through the face feature extractor and upload it to the background face database;

[0061] Face data search steps: calculate the cosine similarity of the uploaded face feature vector and the background face database, and use the cosine similarity to confirm whether the person has littered. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times. When a person who has littered multiple times is detected, the background management staff will provide targeted education for him.

[0062] Furthermore, in the face data search step, when the cosine similarity of the face feature vector is less than or equal to the set threshold, it is determined that this person is littering for the first time. In the step of confirming the new garbage, when the garbage on the ground in f1 is zero, the coordinate information of the garbage on the ground is empty.

[0063] Furthermore, in the step of identifying newly added garbage, the object detector detects garbage on the hands of pedestrians and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, and the detection result in f2 is tracked, when no garbage on the hands is detected in f2 and newly added garbage is detected on the ground, the step of confirming newly added garbage is entered, such as Figure 2 , is to detect garbage on the hands and garbage on the ground through the yoloV5 target detection network; in the network training stage, the annotations are divided into two categories, one category is garbage on the hands, and the other category is garbage on the ground; the annotated rectangular box is (x, y, w, h) and the category it belongs to, where x, y are the coordinates of the center point of the rectangular box, w, h are the width and height of the rectangular box. When reading the training data, these annotated data are normalized, that is, divided by the width and height of the corresponding original image to ensure that they are all in the range of 0 to 1; in the yoloV5 reasoning stage, the target category, the confidence score of whether it is a target, and the target box coordinate information are output.

[0064] Furthermore, in the step of confirming the newly added garbage, assuming that there are N pieces of ground garbage in f1, then the ground garbage in f1 is g1, g2, …, gn, and the ground garbage in f2 is g1, g2, …, gn+1, where n is a natural number. According to the coordinate information of the ground garbage in f1 and f2, if the ground garbage g exists in f1, the corresponding g can be found in f2 according to the coordinate information, thereby excluding g as newly added ground garbage. Based on this feature, the newly added ground garbage can be determined.

[0065] Furthermore, in the step of extracting garbage features, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted according to the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

[0066] Furthermore, in the step of extracting garbage features, the feature vectors of garbage on the hand and garbage on the ground are 256-dimensional feature vectors, and in the step of extracting face features, the feature vector of the face extracted is a 256-dimensional feature vector. Figure 3 The process can extract a 256-dimensional vector, which represents the characteristics of the image block and can be used for subsequent similarity comparison.

[0067] Furthermore, after detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When the littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

[0068] Furthermore, after no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t is continued and the next frame is entered to continue the step of identifying new garbage.

[0069] Furthermore, in the feature comparison step, the feature vector obtained in the step of extracting garbage features is normalized, and then the cosine similarity of the two vectors is calculated. Assuming that the feature vector is g = (x1, x2, x3, .., x256), the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

[0070] Furthermore, the cosine similarity formula for the garbage on the hand and the garbage on the ground is: cos <g hand ^,g add ^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

[0071] A vision-based litter detection system comprising:

[0072] Video acquisition module: Video is acquired through a video acquisition device, and the video data is divided into the previous video frame f1 and the next video frame f2 according to the time sequence and transmitted to the target detector for detection;

[0073] New garbage identification module: The object detector detects garbage on pedestrians’ hands and garbage on the ground in f1 and f2. When garbage on pedestrians’ hands is detected in f1 and garbage on pedestrians’ hands is not detected in f2, it is confirmed that there is new garbage.

[0074] Confirmation of newly added garbage module: the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 are processed and compared to confirm the newly added ground garbage;

[0075] Garbage feature extraction module: The garbage feature extractor is used to extract the feature vectors of garbage on the hand and garbage on the ground respectively;

[0076] Feature comparison module: The garbage feature extractor performs normalized feature vector processing on the feature vectors of the garbage on hand and the garbage on the ground. Through normalized feature vector processing, the features of the garbage on hand and the garbage on the ground are compared, and the cosine similarity of the two vectors is calculated. When the similarity is greater than the set threshold, it is determined that there is littering behavior. When littering behavior is identified, an alarm can be issued in time;

[0077] Face detection module: After detecting littering, the face detector detects the face and crops the face image;

[0078] Face feature extraction module: extracts the feature vector of the face through the face feature extractor and uploads it to the background face database;

[0079] Face data search module: calculates the cosine similarity between the uploaded face feature vector and the background face database, and uses the cosine similarity to confirm whether the person has littered. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times. When a person who has littered multiple times is detected, the background management staff will provide targeted education to him.

[0080] Furthermore, in the face data search module, when the cosine similarity of the face feature vector is less than or equal to the set threshold, the person is deemed to be littering for the first time. In the module for confirming new garbage, when the garbage on the ground in f1 is zero, the coordinate information of the garbage on the ground is empty.

[0081] Furthermore, in the module for identifying newly added garbage, the object detector detects garbage on the hands of pedestrians and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, the detection result in f2 is tracked. When no garbage on the hands is detected in f2 and newly added garbage is detected on the ground, the module for confirming newly added garbage is entered, such as Figure 2 , is to detect garbage on the hands and garbage on the ground through the yoloV5 target detection network; in the network training stage, the annotations are divided into two categories, one category is garbage on the hands, and the other category is garbage on the ground; the annotated rectangular box is (x, y, w, h) and the category it belongs to, where x, y are the coordinates of the center point of the rectangular box, w, h are the width and height of the rectangular box. When reading the training data, these annotated data are normalized, that is, divided by the width and height of the corresponding original image to ensure that they are all in the range of 0 to 1; in the yoloV5 reasoning stage, the target category, the confidence score of whether it is a target, and the target box coordinate information are output.

[0082] Furthermore, in the module for confirming newly added garbage, assuming that there are N pieces of ground garbage in f1, the ground garbage in f1 is g1, g2, …, gn, and the ground garbage in f2 is g1, g2, …, gn+1, where n is a natural number. According to the coordinate information of the ground garbage in f1 and f2, if the ground garbage g exists in f1, the corresponding g can be found in f2 according to the coordinate information, thereby excluding g as newly added ground garbage. Based on this feature, the newly added ground garbage can be determined.

[0083] Furthermore, in the garbage feature extraction module, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted according to the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

[0084] Furthermore, in the garbage feature extraction module, the feature vectors of the garbage on the hand and the garbage on the ground are 256-dimensional feature vectors, and in the face feature extraction module, the feature vector of the face extracted is a 256-dimensional feature vector. Figure 3 The process can extract a 256-dimensional vector, which represents the characteristics of the image block and can be used for subsequent similarity comparison.

[0085] Furthermore, after detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When the littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

[0086] Furthermore, after no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t is continued and the next frame is entered to continue to identify the new garbage module.

[0087] Furthermore, in the feature comparison module, the feature vector obtained in the garbage feature extraction module is normalized, and then the cosine similarity of the two vectors is calculated. Assuming the feature vector is g = (x1, x2, x3, .., x256), the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

[0088] Furthermore, the cosine similarity formula for the garbage on hand and the garbage on the ground is:

[0089] cos <g hand ^,g add^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

[0090] The working principle of the present invention is as follows: video acquisition step: video acquisition is performed by a video acquisition device, and the video data is divided into a previous video frame f1 and a subsequent video frame f2 in chronological order and transmitted to a target detector for detection; a newly added garbage identification step: the target detector detects garbage on the hands of pedestrians and garbage on the ground in f1 and f2, and after the garbage on the hands of pedestrians is detected in f1 and when the garbage on the hands of pedestrians is not detected in f2, it is confirmed that there is newly added garbage; a newly added garbage confirmation step: the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 are processed and compared to confirm the newly added ground garbage; a garbage feature extraction step: the garbage feature extractor is used to extract the newly added garbage on the ground and the newly added garbage on the ground respectively. Take the feature vectors of the garbage on the hands and the garbage on the ground; feature comparison step: use the garbage feature extractor to normalize the feature vectors of the garbage on the hands and the garbage on the ground, calculate the cosine similarity of the two vectors, and when the similarity is greater than the set threshold, it is determined that there is littering; face detection step: after detecting the littering behavior, detect the face through the face detector and crop the face image; face feature extraction step: extract the feature vector of the face through the face feature extractor and upload it to the background face database; face data search step: calculate the cosine similarity of the uploaded face feature vector and the background face database, and when the similarity is greater than the set threshold, it is considered that this person has littered many times.

[0091] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting littering based on vision, characterized in that: The following steps are involved: Video acquisition step: Video is acquired through a video acquisition device, and the video data is divided into a previous video frame f1 and a next video frame f2 in chronological order and transmitted to the target detector for detection; Identify new garbage step: The object detector detects garbage on the hands of pedestrians and garbage on the ground in f1 and f2, and when garbage on the hands of pedestrians is detected in f1 and garbage on the hands of pedestrians is not detected in f2, it is confirmed that there is new garbage; Confirming the newly added garbage step: Process and compare the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 to confirm the newly added ground garbage; Garbage feature extraction step: extract feature vectors of garbage on hand and garbage on the ground respectively through garbage feature extractor; Feature comparison step: The feature vectors of the garbage on hand and the garbage on the ground are normalized by the garbage feature extractor, and the cosine similarity of the two vectors is calculated. When the similarity is greater than the set threshold, it is determined that there is littering behavior; Face detection step: After detecting littering behavior, the face is detected by the face detector and the face image is cropped; Face feature extraction step: extract the feature vector of the face through the face feature extractor and upload it to the background face database; Face data search steps: Calculate the cosine similarity between the uploaded face feature vector and the background face database. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times.

2. The visual-based littering detection method of claim 1, wherein: In the face data search step, when the cosine similarity of the face feature vector is less than or equal to the set threshold, it is determined that this person is littering for the first time. In the step of confirming the new garbage, when the garbage on the ground in f1 is zero, the coordinate information of the garbage on the ground is empty.

3. The visual-based littering detection method of claim 1, wherein: In the step of identifying new garbage, the target detector detects garbage on the hands of pedestrians and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, the detection result in f2 is tracked. When no garbage on the hands is detected in f2 and new garbage is detected on the ground, the step of confirming new garbage is entered.

4. The visual-based littering detection method of claim 3, wherein: In the step of confirming the newly added garbage, suppose there are N pieces of garbage on the ground in f1, then the garbage on the ground in f1 is g1, g2, …, gn, and the garbage on the ground in f2 is g1, g2, …, gn+1, and n is a natural number.

5. The visual-based littering detection method of claim 3, wherein: In the step of extracting garbage features, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted based on the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

6. The visual-based littering detection method of claim 1, wherein: In the step of extracting garbage features, the feature vectors of garbage on hands and garbage on the ground are 256-dimensional feature vectors. In the step of extracting face features, the feature vector of the extracted face is a 256-dimensional feature vector.

7. The visual-based littering detection method of claim 1, wherein: After detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

8. The visual-based littering detection method of claim 3, wherein: After no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t is continued and the next frame is entered to continue the step of identifying new garbage.

9. The visual-based littering detection method of claim 6, wherein: In the feature comparison step, the feature vector obtained in the step of extracting garbage features is normalized, and then the cosine similarity of the two vectors is calculated. Assuming that the feature vector is g = (x1, x2, x3, .., x256), the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

10. The visual-based littering detection method of claim 9, wherein: The cosine similarity formula for the garbage on the hand and the garbage on the ground is: <g hand ^,g add ^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

11. A vision-based litter detection system, characterized in that: include: Video acquisition module: Video is acquired through a video acquisition device, and the video data is divided into the previous video frame f1 and the next video frame f2 according to the time sequence and transmitted to the target detector for detection; New garbage identification module: The object detector detects garbage on pedestrians’ hands and garbage on the ground in f1 and f2. When garbage on pedestrians’ hands is detected in f1 and garbage on pedestrians’ hands is not detected in f2, it is confirmed that there is new garbage. Confirmation of newly added garbage module: the coordinate information of the ground garbage in f1 and the coordinate information of the newly added ground garbage in f2 are processed and compared to confirm the newly added ground garbage; Garbage feature extraction module: The garbage feature extractor is used to extract the feature vectors of garbage on the hand and garbage on the ground respectively; Feature comparison module: The garbage feature extractor processes the feature vectors of garbage on hand and garbage on the ground into normalized feature vectors, calculates the cosine similarity of the two vectors, and identifies littering when the similarity is greater than the set threshold; Face detection module: After detecting littering, the face detector detects the face and crops the face image; Face feature extraction module: extracts the feature vector of the face through the face feature extractor and uploads it to the background face database; Face data search module: calculate the cosine similarity between the uploaded face feature vector and the background face database. If the similarity is greater than the set threshold, it is considered that the person has littered multiple times.

12. The vision-based litter detection system of claim 11, wherein: In the face data search module, when the cosine similarity of the face feature vector is less than or equal to the set threshold, it is determined that this person is littering for the first time. In the new garbage confirmation module, when the ground garbage in f1 is zero, the coordinate information of the garbage on the ground is empty.

13. The vision-based litter detection system of claim 11, wherein: In the module for identifying newly added garbage, the target detector detects garbage on pedestrians' hands and garbage on the ground in two video frames f1 and f2 with a set interval time t. When garbage on the hands is detected in f1, the detection result in f2 is tracked. When no garbage on the hands is detected in f2 and newly added garbage is detected on the ground, the module for confirming newly added garbage is entered.

14. The vision-based litter detection system of claim 13, wherein: In the module for confirming newly added garbage, suppose there are N pieces of garbage on the ground in f1, then the garbage on the ground in f1 is g1, g2, …, gn, and the garbage on the ground in f2 is g1, g2, …, gn+1, where n is a natural number.

15. The vision-based litter detection system of claim 13, wherein: In the garbage feature extraction module, the coordinate frame information of the garbage on the hand and the garbage on the ground is detected by the object detector, and the image block of the garbage is intercepted based on the coordinate frame information, and the feature vector is extracted through the image block of the garbage.

16. The vision-based litter detection system of claim 11, wherein: In the garbage feature extraction module, the feature vectors of garbage on hands and garbage on the ground are 256-dimensional feature vectors, and in the face feature extraction module, the feature vector of the face extracted is a 256-dimensional feature vector.

17. The vision-based litter detection system of claim 11, wherein: After detecting the littering behavior of pedestrians, face detection is performed and compared with the background database. When littering behavior is identified, the face is detected, the face is cropped, and the feature information of the face is extracted to generate a 256-dimensional feature vector. The face feature vector is compared with the background database for similarity, the face information is stored in the database and the face information of multiple littering behaviors is sent to the management terminal.

18. The vision-based litter detection system of claim 13, wherein: After no garbage on the hands is detected in f2 and no new garbage on the ground is detected, the interval time t continues and the next frame continues to identify the new garbage module.

19. The vision-based litter detection system of claim 16, wherein: In the feature comparison module, the feature vector obtained in the garbage feature extraction module is normalized, and then the cosine similarity of the two vectors is calculated. Suppose the feature vector is g = (x1, x2, x3, .., x256), then the vector normalization formula is: g^ = g / |g|, where |g| is the modulus of the vector g, |g| = sqrt(x1 2 +x2 2 +x3 2 +…+x256 2 ).

20. The vision-based litter detection system of claim 19, wherein: The cosine similarity formula for the garbage on the hand and the garbage on the ground is: <g hand ^,g add ^>=g hand ^.g add ^ / |g hand ^|.| gadd ^|, where g hand Indicates garbage on hand, g add Indicates garbage on the ground.

Images