Garbage throwing detection method, device and equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a garbage throwing detection method, device and equipment and a storage medium, the method comprises the following steps: determining the position information of a staying pedestrian and a garbage can in a garbage throwing area; acquiring the skeleton key point information of the staying pedestrian, and determining the arm swing information according to the skeleton key point information; and performing garbage throwing detection according to the garbage can position information and the arm swing information. Compared with the prior art, the administrator needs to manually capture garbage throwing images, which leads to low detection efficiency of garbage throwing. According to the garbage throwing detection method, the garbage can position information and the arm swing information are used for garbage throwing detection, so that the detection efficiency of garbage throwing is improved, and the false positive rate of garbage throwing is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a garbage throwing detection method and device, equipment and a storage medium. BACKGROUND

[0002] In the face of increasing garbage production, through garbage classification management, garbage resource utilization is maximized, and the number of garbage disposal is reduced, which is a very important measure. At present, many cities such as Beijing, Shenzhen and Nanjing have formulated garbage classification management regulations and compulsorily implemented garbage classification. In the prior art, due to the installation position of the camera and the body posture of the person throwing garbage, the hands throwing garbage cannot be correctly captured, which is easy to misreport, thereby causing a high false positive rate of garbage throwing.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a garbage throwing detection method, device, equipment and storage medium, which aims to solve the technical problem of how to reduce the false positive rate of garbage throwing.

[0005] To achieve the above purpose, the present application provides a garbage throwing detection method, which comprises:

[0006] determining the position information of the garbage can and the person staying in the garbage throwing area;

[0007] obtaining the skeletal key point information of the person staying, and determining the arm swing information according to the skeletal key point information;

[0008] performing garbage throwing detection according to the position information of the garbage can and the arm swing information.

[0009] Optionally, the step of obtaining the skeletal key point information of the person staying and determining the arm swing information according to the skeletal key point information comprises:

[0010] obtaining the skeletal key point information of the person staying, and obtaining the coordinates of each key point and the confidence corresponding to each key point according to the skeletal key point information;

[0011] selecting a plurality of target key point coordinates from each key point according to the confidence;

[0012] determining the arm swing information according to the plurality of target key point coordinates.

[0013] Optionally, the step of determining the arm swing information according to the plurality of target key point coordinates comprises:

[0014] select palm key point coordinates and arm key point coordinates from the plurality of target key point coordinates;

[0015] determine arm swing information according to the arm key point coordinates and the palm key point coordinates.

[0016] Optionally, the step of selecting palm key point coordinates and arm key point coordinates from the plurality of target key point coordinates comprises:

[0017] select palm key point coordinates from the plurality of target key point coordinates, and determine garbage can coordinates according to the garbage can position information;

[0018] determine whether the palm key point coordinates satisfy a preset throwing condition according to the garbage can coordinates;

[0019] select arm key point coordinates from the target key point coordinates when the palm key point coordinates satisfy the preset throwing condition.

[0020] Optionally, the step of determining arm swing information according to the arm key point coordinates and the palm key point coordinates comprises:

[0021] determine an arm swing angle according to the arm key point coordinates and the palm key point coordinates;

[0022] determine whether the arm swing angle is greater than or equal to a preset swing threshold value;

[0023] determine arm swing information according to the arm swing angle when the arm swing angle is greater than or equal to the preset swing threshold value.

[0024] Optionally, the step of determining arm swing information according to the arm swing angle comprises:

[0025] obtain an arm swing time corresponding to the stopped pedestrian;

[0026] determine an arm swing angular velocity according to the arm swing angle and the arm swing time;

[0027] determine whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold value;

[0028] determine arm swing information according to the arm swing angle and the arm swing angular velocity when the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold value.

[0029] Optionally, after the step of determining whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold value, the method further comprises:

[0030] When the arm swing angular velocity is less than the preset swing angular velocity threshold, returning to the step of determining the staying pedestrian in the garbage throwing area.

[0031] Optionally, the step of detecting garbage throwing according to the garbage can position information and the arm swing information comprises:

[0032] Obtaining the staying duration of the staying pedestrian in the garbage throwing area;

[0033] Determining whether the staying duration is less than or equal to a preset staying threshold;

[0034] When the staying duration is less than or equal to the preset staying threshold, detecting garbage throwing according to the garbage can position information and the arm swing information.

[0035] Optionally, the step of detecting garbage throwing according to the garbage can position information and the arm swing information further comprises:

[0036] When detecting that the staying pedestrian throws garbage, obtaining a garbage image;

[0037] Determining garbage classification information according to the garbage image, and giving a garbage throwing prompt to the staying pedestrian according to the garbage classification information.

[0038] Optionally, the step of determining garbage classification information according to the garbage image comprises:

[0039] Preprocessing the garbage image to obtain a to-be-processed garbage image;

[0040] Convolving the to-be-processed garbage image to obtain a convolved garbage image;

[0041] Pooling the convolved garbage image to obtain a garbage feature image;

[0042] Determining garbage classification information according to the garbage feature image.

[0043] Optionally, the step of determining garbage classification information according to the garbage feature image comprises:

[0044] Determining garbage feature information according to the garbage feature image;

[0045] Classifying the garbage feature information to obtain garbage classification information.

[0046] Optionally, the step of determining the staying pedestrian in the garbage throwing area and the garbage can position information further comprises:

[0047] Acquire a panoramic image of the trash can, and calibrate the position of the trash can based on the panoramic image to obtain the vertex pixel coordinates;

[0048] The trash can marking area is determined based on the vertex pixel coordinates, and the trash disposal area is determined based on the trash can marking area.

[0049] Furthermore, to achieve the above objectives, the present invention also proposes a waste disposal detection device, the waste disposal detection device comprising:

[0050] The determination module is used to determine the location information of pedestrians and trash cans within the garbage disposal area;

[0051] The acquisition module is used to acquire the skeletal key point information of the stopped pedestrian and determine the arm swing information based on the skeletal key point information;

[0052] The detection module is used to detect garbage disposal based on the location information of the trash can and the swing amplitude information of the arm.

[0053] Optionally, the acquisition module is further configured to acquire the skeletal key point information of the stopped pedestrian, and obtain the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information;

[0054] The acquisition module is further configured to select multiple target key point coordinates from each key point based on the confidence level;

[0055] The acquisition module is also used to determine the arm swing amplitude information based on the coordinates of multiple target key points.

[0056] Optionally, the acquisition module is further configured to select the coordinates of the palm key point and the coordinates of the arm key point from multiple target key point coordinates;

[0057] The acquisition module is also used to determine arm swing information based on the coordinates of the key points of the arm and the key points of the palm.

[0058] Optionally, the acquisition module is further configured to select the coordinates of a hand key point from multiple target key point coordinates, and determine the coordinates of the trash can based on the trash can location information;

[0059] The acquisition module is also used to determine whether the coordinates of the key points of the hand meet the preset disposal conditions based on the coordinates of the trash can.

[0060] The acquisition module is further configured to select the coordinates of the arm key points from the coordinates of each target key point when the coordinates of the palm key points meet the preset deployment conditions.

[0061] Optionally, the acquisition module is further configured to determine the arm swing angle based on the coordinates of the key points of the arm and the coordinates of the key points of the palm;

[0062] The acquisition module is also used to determine whether the arm swing angle is greater than or equal to a preset swing threshold.

[0063] The acquisition module is further configured to determine arm swing information based on the arm swing angle when the arm swing angle is greater than or equal to the preset swing threshold.

[0064] Optionally, the acquisition module is further configured to acquire the arm swing time corresponding to the stopped pedestrian;

[0065] The acquisition module is also used to determine the arm swing angular velocity based on the arm swing angle and the arm swing time;

[0066] The acquisition module is also used to determine whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold.

[0067] The acquisition module is further configured to determine arm swing information based on the arm swing angle and the arm swing angular velocity when the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold.

[0068] Furthermore, to achieve the above objectives, the present invention also proposes a waste disposal detection device, the device comprising: a memory, a processor, and a waste disposal detection program stored in the memory and executable on the processor, the waste disposal detection program being configured to implement the steps of the waste disposal detection method described above.

[0069] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a waste disposal detection program, which, when executed by a processor, implements the steps of the waste disposal detection method described above.

[0070] This invention first determines the location information of pedestrians and trash cans within the waste disposal area, then acquires the skeletal key point information of the pedestrians, and determines the arm swing information based on the skeletal key point information. Finally, it uses the arm swing information based on the trash can location information and the skeletal key point information to detect waste disposal. Compared to existing technologies that require manually capturing images of waste disposal, resulting in low detection efficiency, this invention improves detection efficiency and reduces false alarm rates by using trash can location information and arm swing information for waste disposal detection. Attached Figure Description

[0071] Figure 1This is a schematic diagram of the structure of the waste disposal detection device in the hardware operating environment involved in the embodiments of the present invention;

[0072] Figure 2 This is a flowchart illustrating the first embodiment of the waste disposal detection method of the present invention;

[0073] Figure 3 This is a schematic diagram of key points of the human skeleton in the first embodiment of the waste disposal detection method of the present invention;

[0074] Figure 4 This is a graph showing the arm swing amplitude and angular velocity of the first embodiment of the waste disposal detection method of the present invention;

[0075] Figure 5 This is a flowchart illustrating the second embodiment of the waste disposal detection method of the present invention;

[0076] Figure 6 This is a structural block diagram of the first embodiment of the waste disposal detection device of the present invention.

[0077] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0078] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0079] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a waste disposal detection device in the hardware operating environment of an embodiment of the present invention.

[0080] like Figure 1As shown, the waste disposal detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0081] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the waste disposal detection equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0082] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a waste disposal detection program.

[0083] exist Figure 1 In the waste disposal detection device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the waste disposal detection device of the present invention can be set in the waste disposal detection device. The waste disposal detection device calls the waste disposal detection program stored in the memory 1005 through the processor 1001 and executes the waste disposal detection method provided in the embodiment of the present invention.

[0084] This invention provides a method for detecting waste disposal, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the waste disposal detection method of the present invention.

[0085] In this embodiment, the waste disposal detection method includes the following steps:

[0086] Step S10: Determine the location information of pedestrians and trash cans in the garbage disposal area.

[0087] It is easy to understand that the executing entity of this embodiment can be a garbage disposal detection device with functions such as data processing, network communication and program operation, or other computer devices with similar functions. This embodiment does not limit it.

[0088] It is understood that the garbage disposal area can be understood as the garbage disposal entry area and / or garbage disposal exit area, etc., and the garbage bin location information can be the spatial coordinate information of the garbage bin, etc., which are not limited in this embodiment.

[0089] Furthermore, in order to accurately determine the garbage disposal area, before determining the location information of pedestrians and garbage cans within the garbage disposal area, a panoramic image of the garbage can can be acquired, and the location of the garbage can can be calibrated based on the panoramic image to obtain vertex pixel coordinates. Then, the calibrated area of ​​the garbage can can be determined based on the vertex pixel coordinates, and the garbage disposal area can be determined based on the calibrated area of ​​the garbage can.

[0090] Another method for defining waste disposal areas is to use a quadrilateral to mark the location of the waste bins at the disposal point, obtaining the pixel coordinates of the four vertices, denoted as Area_bin. Then, the quadrilateral is used to delineate the waste disposal entry area and the waste disposal exit area, denoted as Area_in and Area_out respectively. The size of the waste disposal entry area and the waste disposal exit area can be selected based on the site conditions.

[0091] Assuming the width of the garbage disposal area is 2 to 4 times the width of Area_bin and the height is 1.2 times the height of Area_bin, then the width of the garbage disposal area can be 3 to 5 times the width of Area_bin and the height can be 1.4 times the height of Area_bin, etc.

[0092] It should be noted that the garbage disposal entry area and the garbage disposal exit area can be the same area. However, due to image detection errors or slight changes in the pedestrian's position at the boundary, the detection results may show the pedestrian repeatedly entering and exiting within a short period of time. This jitter or switching causes many inconveniences for subsequent processing. Therefore, the area set for the garbage disposal exit area needs to be larger than the garbage disposal entry detection area, which can reduce image detection errors, etc.

[0093] It should also be noted that the pedestrians who are standing still are people waiting to be detected in the garbage disposal area. Since many pedestrians can walk back and forth in the garbage disposal area, in order to accurately identify the pedestrians who need to dispose of garbage, an appropriate intelligent model can be selected according to the on-site resource conditions. The intelligent model can detect and track pedestrians in the disposal area.

[0094] Since the YOLO V5 object detection model can be used for pedestrian detection with an inference time of as little as 0.007 seconds (140 frames per second) and a weight file size of only 27MB, the intelligent model can be the YOLO V5 object detection model, which can then be used for pedestrian detection. For subsequent pedestrian tracking, the Deepsort multi-object tracking algorithm can be used, and data can be correlated using motion models and appearance information.

[0095] If a multi-object tracking algorithm detects pedestrian i, it will generate a unique identifier for it, denoted as ID. i And as long as there is no tracking loss, ID i The list remains unchanged. A dynamic list of pedestrian IDs, ID_list = [ID1, ID2, ...], is created. When a pedestrian is detected entering the detection area (Area_in), their ID is added to the list; when a pedestrian is detected leaving the detection area (Area_out), their ID is removed from the category. The time t is also recorded when pedestrian i enters and leaves the detection area. iin and t iout For storage, etc.

[0096] Step S20: Obtain the skeletal key point information of the stopped pedestrian, and determine the arm swing information based on the skeletal key point information.

[0097] The skeletal key point information can be the coordinate information of multiple joint points of the pedestrian, such as the palm coordinates, elbow coordinates, or shoulder coordinates, etc. The arm swing information can be the arm swing angle and arm swing angular velocity, etc., which are not limited in this embodiment.

[0098] Furthermore, in order to accurately obtain arm swing information, the steps of obtaining the skeletal key point information of the stopped pedestrian and determining the arm swing information based on the skeletal key point information include: obtaining the skeletal key point information of the stopped pedestrian, the skeletal key point information including the coordinates of each key point and the confidence level corresponding to each key point, selecting multiple target key point coordinates from each key point based on the confidence level, and determining the arm swing information based on the coordinates of the multiple target key points, etc.

[0099] Assuming the skeletal keypoint information includes palm coordinates, elbow coordinates, and shoulder coordinates, the confidence level for palm coordinates is 0.9, for elbow coordinates it is 0.1, and for shoulder coordinates it is 0.7. If the preset confidence threshold is 0.6, and the confidence levels for palm and shoulder coordinates are greater than the preset threshold, then palm and shoulder coordinates are used as target keypoint coordinates. The confidence level for elbow coordinates is less than the preset threshold, so the elbow coordinates need to be re-acquired using skeletal keypoint acquisition software. The preset confidence threshold can be user-defined and can be 0.6, 0.8, etc. This embodiment does not impose any limitations.

[0100] Another method for determining arm swing amplitude information based on the coordinates of multiple target key points is to select the coordinates of the palm key point and the arm key point from multiple target key point coordinates, and then determine the arm swing amplitude information based on the coordinates of the arm key point and the palm key point.

[0101] The processing method for selecting hand and arm key point coordinates from multiple target key point coordinates can be as follows: select hand key point coordinates from multiple target key point coordinates, determine trash can coordinates based on trash can location information, and determine whether the hand key point coordinates meet preset disposal conditions based on trash can coordinates; when the hand key point coordinates meet the preset disposal conditions, select arm key point coordinates from each target key point coordinate, etc.

[0102] The method for determining arm swing amplitude information based on the coordinates of key points on the arm and the key points on the palm can be as follows: determine the arm swing amplitude angle based on the coordinates of key points on the arm and the key points on the palm, determine whether the arm swing amplitude angle is greater than or equal to a preset swing amplitude threshold, and when the arm swing amplitude angle is greater than or equal to the preset swing amplitude threshold, determine the arm swing amplitude information based on the arm swing amplitude angle, etc. The preset swing amplitude threshold can be user-defined and can be 60 degrees, 80 degrees, 100 degrees, etc.

[0103] The steps for determining arm swing information based on the arm swing angle can be as follows: obtaining the arm swing time corresponding to the pedestrian; determining the arm swing angular velocity based on the arm swing angle and arm swing time; determining whether the arm swing angular velocity is greater than or equal to a preset angular velocity threshold; when the arm swing angular velocity is greater than or equal to the preset angular velocity threshold, determining the arm swing information based on the arm swing angle and the arm swing angular velocity; when the arm swing angular velocity is less than the preset angular velocity threshold, returning to the step of determining the pedestrian in the garbage disposal area. The preset angular velocity can be user-defined, such as 4 or 5.

[0104] Assuming the angle between the key coordinates of the arm and the key coordinates of the palm and the ground (i.e., the arm swing angle) is 100 degrees, it is determined whether the arm swing angle of 100 degrees is greater than or equal to a preset swing threshold of 90 degrees. When the arm swing angle of 100 degrees is greater than or equal to the preset swing threshold of 90 degrees, the arm swing time corresponding to the stopped pedestrian is obtained. If the arm swing time is 20 seconds, which is greater than the preset arm swing time threshold of 4 seconds, the arm swing angular velocity 5 is determined based on the arm swing angle of 100 degrees and the arm swing time of 20 seconds. It is then determined whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold of 4. When the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold, arm swing information is determined based on the arm swing angle and the arm swing angular velocity, etc.

[0105] Step S30: Detect garbage disposal based on the garbage bin location information and the arm swing amplitude information.

[0106] The steps for detecting garbage disposal based on trash can location information and arm swing information can be as follows: obtain the duration of a pedestrian's stay in the garbage disposal area, then determine whether the duration of stay is less than or equal to a preset stay threshold. If the duration of stay is less than or equal to the preset stay threshold, garbage disposal detection is performed based on the arm swing information according to the trash can location information and skeletal key point information. The preset stay threshold can be set by the user, such as 5 minutes or 6 minutes, etc.

[0107] In the specific implementation, for each pedestrian detected and tracked, the top-down human keypoint detection model AlphaPose outputs the coordinates of 17 key points of the human body, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of key points in the human skeleton of the first embodiment of the waste disposal detection method of the present invention. In the diagram, 1 is the neck key point, 2 is the right shoulder key point, 3 is the right elbow key point, 4 is the right wrist key point, 5 is the left shoulder key point, 6 is the left elbow key point, 7 is the left wrist key point, 8 is the right hip key point, 9 is the right knee key point, 10 is the right ankle key point, 11 is the left hip key point, 12 is the left knee key point, and 13 is the left ankle key point. Each point in the skeletal key points corresponds to a coordinate p. i =(x i ,y i ) and score c i∈[0,1],i=0,1,...,16, representing the pixel position in the skeletal keypoint diagram and the confidence level of the skeletal point position, respectively. Then, it is determined whether the pedestrian has littered and whether the hand skeletal points were ever above the trash can. If (x4,y4)∈Area_bin and (x7,y7)∈Area_bin is not true, it means that the hand skeletal points were not detected above the food waste can. Then, the swing amplitude (angle) of the pedestrian's arms is calculated in real time through the positions of the elbows and hand skeletal points, and it is determined that the maximum swing amplitude and the swing amplitude angular velocity exceed the threshold.

[0108] One way to calculate the swing angular velocity is to calculate and save each frame t at each time point. j vector The angle between the angle and the vertical y-axis is denoted as . When a pedestrian is detected leaving the detection area, the maximum swing amplitude of both hands is calculated. If max_degree <degr ee thresh If the maximum degree is reached, pedestrians within the perimeter will be detected and tracked; otherwise, the maximum degree will be determined by taking c values ​​on both sides from near to far. i >c thresh From the 10 points, the slopes k1 and k2 of the two straight lines, i.e., the angular velocities, are calculated through linear fitting, such as... Figure 4 As shown. Figure 4 This is a graph showing the arm swing amplitude and angular velocity curve of the first embodiment of the waste disposal detection method of the present invention. Points marked with "×" are shown in the graph. i <c thresh This is not involved in the fitting process; the x-axis represents the video frame time, i.e., the number of frames, to avoid the algorithmic time consumption introduced by calculating astronomical time. Let k = max(|k1|,|k2|), if k <k th resh If the pedestrian is detected within the perimeter, then pedestrian detection and tracking are performed; otherwise, the dwell time of the pedestrian in the garbage disposal area is calculated to determine if it is less than the threshold. Let t be the value of this value. i =t i,out -t i,in If t i >t thresh If the pedestrian ID is found to be within the perimeter, pedestrian detection and tracking are performed; otherwise, an alarm is triggered with the message "Please dispose of trash correctly," and the pedestrian ID is removed from the ID_list. The system checks if all video frames have been analyzed; otherwise, pedestrian detection and tracking within the perimeter are performed. Where t... i The unit is video frame rate, etc.

[0109] Furthermore, to reduce the false alarm rate of garbage disposal, after the garbage disposal detection step based on the garbage bin location information and arm swing information, the garbage image is preprocessed to obtain a garbage image to be processed. The garbage image to be processed is then convolved to obtain a convolutional garbage image. The convolutional garbage image is then pooled to obtain a garbage feature image. The garbage classification information is determined based on the garbage feature image. The garbage classification information can be dry waste, hazardous waste, wet waste, etc. Then, the garbage classification information is used to determine whether the pedestrian has disposed of the garbage correctly.

[0110] The steps for determining waste classification information based on waste feature images can include determining waste feature information based on waste feature images, then classifying the waste feature information to obtain waste classification information, etc.

[0111] Preprocessing refers to the enhancement of an image, which includes noise reduction, image smoothing, and image sharpening. This enhancement is implemented through an image recognition processor.

[0112] The feature image is obtained by the image recognition processor through training a neural network model. This neural network model is a pre-trained neural network model and includes the following 15 layers: four intersecting convolutional layers and four downsampling layers, three fully connected layers, three activation layers, and one classification layer. The fully connected layers are used to adjust parameters and make the network more stable, while the activation layers are used to improve the network speed. In this neural network model, after the preprocessed image undergoes intersecting convolution and downsampling, the image recognition processor extracts the weight parameters in the 10th activation layer as the image features.

[0113] Features should not only be able to better describe images and reduce overfitting, but more importantly, they should also be able to distinguish between different categories of images.

[0114] The convolutional neural network image training model constructed in this invention has a total of 15 layers. Its structure improves the recognition rate and provides better stability for garbage identification. The preset classification network used in this invention can be a variety of models. Depending on the user's needs and the number of images, different classifiers can be selected for comparison, such as support vector machines, radial basis function classifiers, and backpropagation neural networks. Experiments show that support vector machines perform best, are the fastest, and have the highest recognition rate.

[0115] Feature image recognition is achieved through a pre-set classification network model, which is a network model trained on feature images. After the feature image is acquired, it is directly input into the classification network model to complete the recognition of the feature image. Then the recognition result is returned to the image recognition processor, which determines whether it is recyclable and the category of recyclable waste based on the recognition result.

[0116] In this embodiment, the location information of pedestrians and trash cans within the waste disposal area is first determined. Then, the skeletal key point information of the pedestrians is acquired, and the arm swing information is determined based on the skeletal key point information. Finally, waste disposal detection is performed based on the trash can location information and arm swing information. Compared to existing technologies, which require manually capturing images of waste disposal, resulting in low detection efficiency, this invention improves detection efficiency and reduces false alarm rates by performing waste disposal detection based on trash can location information and arm swing information.

[0117] refer to Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the waste disposal detection method of the present invention.

[0118] Based on the first embodiment described above, in this embodiment, step S20 further includes:

[0119] Step S201: Obtain the skeletal key point information of the stopped pedestrian, and obtain the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information.

[0120] The skeletal key point information can be the coordinate information of multiple joint points of the pedestrian, such as the palm coordinates, elbow coordinates, or shoulder coordinates, etc. The arm swing information can be the arm swing angle and arm swing angular velocity, etc., which are not limited in this embodiment.

[0121] Assuming that the skeletal keypoint information includes palm coordinates, elbow coordinates, and shoulder coordinates, then the confidence level corresponding to the palm coordinates is 9, the confidence level corresponding to the elbow coordinates is 4, and the confidence level corresponding to the shoulder coordinates is 7, etc.

[0122] Step S202: Select multiple target key point coordinates from each key point based on the confidence level.

[0123] Furthermore, in order to accurately obtain arm swing information, obtain skeletal key point information of a stopped pedestrian, and determine arm swing information based on skeletal key point information, the steps include: obtaining skeletal key point information of a stopped pedestrian, determining the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information, and selecting multiple target key point coordinates from the key points based on the confidence level.

[0124] Assuming the skeletal keypoint information includes palm coordinates, elbow coordinates, and shoulder coordinates, the confidence level for palm coordinates is 0.9, for elbow coordinates it is 0.4, and for shoulder coordinates it is 0.7. If the preset confidence threshold is 0.6, and the confidence levels for palm and shoulder coordinates are greater than the preset threshold, then palm and shoulder coordinates are used as target keypoint coordinates. The confidence level for elbow coordinates is less than the preset threshold, so the elbow coordinates need to be re-acquired using skeletal keypoint acquisition software. The preset confidence threshold can be user-defined and can be 0.6, 0.8, etc. This embodiment does not impose any limitations.

[0125] Another method for determining arm swing amplitude information based on the coordinates of multiple target key points is to select the coordinates of the palm key point and the arm key point from multiple target key point coordinates, and then determine the arm swing amplitude information based on the coordinates of the arm key point and the palm key point.

[0126] The processing method for selecting hand and arm key point coordinates from multiple target key point coordinates can be as follows: select hand key point coordinates from multiple target key point coordinates, determine trash can coordinates based on trash can location information, and determine whether the hand key point coordinates meet preset disposal conditions based on trash can coordinates; when the hand key point coordinates meet the preset disposal conditions, select arm key point coordinates from each target key point coordinate, etc.

[0127] Step S203: Determine the arm swing amplitude information based on the coordinates of multiple target key points.

[0128] The target key point coordinates can be the arm key point coordinates and the palm key point coordinates. The arm swing amplitude information can be determined based on the arm key point coordinates and the palm key point coordinates by determining the arm swing amplitude angle, judging whether the arm swing amplitude angle is greater than or equal to a preset swing amplitude threshold, and determining the arm swing amplitude information based on the arm swing amplitude when the arm swing amplitude angle is greater than or equal to the preset swing amplitude threshold. The preset swing amplitude threshold can be user-defined and can be 60 degrees, 80 degrees, 100 degrees, etc.

[0129] The steps for determining arm swing information based on the arm swing angle can be as follows: obtaining the arm swing time corresponding to the pedestrian; determining the arm swing angular velocity based on the arm swing angle and arm swing time; determining whether the arm swing angular velocity is greater than or equal to a preset angular velocity threshold; when the arm swing angular velocity is greater than or equal to the preset angular velocity threshold, determining the arm swing information based on the arm swing angle and the arm swing angular velocity; when the arm swing angular velocity is less than the preset angular velocity threshold, returning to the step of determining the pedestrian in the garbage disposal area. The preset angular velocity can be user-defined, such as 4 or 5.

[0130] Assuming the angle between the key coordinates of the arm and the key coordinates of the palm and the ground (i.e., the arm swing angle) is 100 degrees, it is determined whether the arm swing angle of 100 degrees is greater than or equal to a preset swing threshold of 90 degrees. When the arm swing angle of 100 degrees is greater than or equal to the preset swing threshold of 90 degrees, the arm swing time corresponding to the stopped pedestrian is obtained. If the arm swing time is 20 seconds, which is greater than the preset arm swing time threshold of 4 seconds, the arm swing angular velocity 5 is determined based on the arm swing angle of 100 degrees and the arm swing time of 20 seconds. It is then determined whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold of 4. When the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold, arm swing information is determined based on the arm swing angle and the arm swing angular velocity, etc.

[0131] In the specific implementation, for each pedestrian detected and tracked, the top-down human keypoint detection model AlphaPose outputs the coordinates of 17 key points of the human body, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of key points of a human skeleton in the first embodiment of the waste disposal detection method of the present invention, wherein each point in the skeleton corresponds to a coordinate p. i =(x i ,y i ) and score c i ∈[0,1],i=0,1,...,16, representing the pixel position in the skeletal keypoint diagram and the confidence level of the skeletal point position, respectively. Then, it is determined whether the pedestrian has littered and whether the hand skeletal points were ever above the trash can. If (x4,y4)∈Area_bin and (x7,y7)∈Area_bin is not true, it means that the hand skeletal points were not detected above the food waste can. Then, the swing amplitude (angle) of the pedestrian's arms is calculated in real time through the positions of the elbows and hand skeletal points, and it is determined that the maximum swing amplitude and the swing amplitude angular velocity exceed the threshold.

[0132] One way to calculate the swing angular velocity is to calculate and save each frame t at each time point. j vector The angle between the angle and the vertical y-axis is denoted as . When a pedestrian is detected leaving the detection area, the maximum swing amplitude of both hands is calculated. If max_degre e <degree thresh If the maximum degree is reached, pedestrians within the perimeter will be detected and tracked; otherwise, the maximum degree will be determined by taking c values ​​on both sides from near to far. i >c thresh The slopes k1 and k2 of the two straight lines, i.e. angular velocities, are calculated from the 10 points using linear fitting, and the arm swing angle and arm swing angular velocity are used as arm swing information.

[0133] In this embodiment, the skeletal key point information of the stopped pedestrian is first obtained, including the coordinates of each key point and the confidence level corresponding to each key point. Then, multiple target key point coordinates are selected from each key point based on the confidence level. After that, the arm swing information is determined based on the coordinates of multiple target key points, so that the arm swing information can be accurately obtained.

[0134] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the waste disposal detection device of the present invention.

[0135] like Figure 6 As shown, the waste disposal detection device proposed in this embodiment of the invention includes:

[0136] The determination module 6001 is used to determine the location information of pedestrians and trash cans in the garbage disposal area;

[0137] The acquisition module 6002 is used to acquire the skeletal key point information of the stopped pedestrian and determine the arm swing information based on the skeletal key point information;

[0138] The detection module 6003 is used to detect garbage disposal based on the garbage bin location information and the arm swing amplitude information.

[0139] In this embodiment, the location information of pedestrians and trash cans within the waste disposal area is first determined. Then, the skeletal key point information of the pedestrians is acquired, and the arm swing information is determined based on the skeletal key point information. Finally, waste disposal detection is performed based on the trash can location information and arm swing information. Compared to existing technologies, which require manually capturing images of waste disposal, resulting in low detection efficiency, this invention improves detection efficiency and reduces false alarm rates by performing waste disposal detection based on trash can location information and arm swing information.

[0140] Furthermore, the acquisition module 6002 is also used to acquire the skeletal key point information of the stopped pedestrian, and obtain the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information.

[0141] The acquisition module 6002 is further configured to select multiple target key point coordinates from each key point based on the confidence level;

[0142] The acquisition module 6002 is also used to determine the arm swing amplitude information based on the coordinates of multiple target key points.

[0143] Furthermore, the acquisition module 6002 is also used to select the palm key point coordinates and the arm key point coordinates from multiple target key point coordinates;

[0144] The acquisition module 6002 is also used to determine arm swing information based on the coordinates of the key points of the arm and the coordinates of the key points of the palm.

[0145] Furthermore, the acquisition module 6002 is also used to select the coordinates of the hand key point from multiple target key point coordinates, and determine the coordinates of the trash can based on the trash can location information;

[0146] The acquisition module 6002 is also used to determine whether the coordinates of the key points of the hand meet the preset disposal conditions based on the coordinates of the trash can.

[0147] The acquisition module 6002 is further configured to select the coordinates of the arm key points from the coordinates of each target key point when the coordinates of the palm key points meet the preset deployment conditions.

[0148] Furthermore, the acquisition module 6002 is also used to determine the arm swing angle based on the coordinates of the key points of the arm and the coordinates of the key points of the palm;

[0149] The acquisition module 6002 is also used to determine whether the arm swing angle is greater than or equal to a preset swing threshold.

[0150] The acquisition module 6002 is further configured to determine arm swing information based on the arm swing angle when the arm swing angle is greater than or equal to the preset swing threshold.

[0151] Furthermore, the acquisition module 6002 is also used to acquire the arm swing time corresponding to the stopped pedestrian;

[0152] The acquisition module 6002 is further configured to determine the arm swing angular velocity based on the arm swing angle and the arm swing time;

[0153] The acquisition module 6002 is also used to determine whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold.

[0154] The acquisition module 6002 is further configured to determine arm swing information based on the arm swing angle and the arm swing angular velocity when the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold.

[0155] Furthermore, the acquisition module 6002 is also used to return the operation of determining the pedestrians staying in the garbage disposal area when the arm swing angular velocity is less than the preset swing angular velocity threshold.

[0156] Furthermore, the detection module 6003 is also used to obtain the duration of the pedestrian's stay in the garbage disposal area;

[0157] The detection module 6003 is also used to determine whether the dwell time is less than or equal to a preset dwell threshold;

[0158] The detection module 6003 is also used to detect garbage disposal based on the garbage bin location information and the arm swing information when the dwell time is less than or equal to the preset dwell threshold.

[0159] Furthermore, the waste disposal detection device also includes a prompting module;

[0160] The notification module is used to acquire an image of the trash when it detects a pedestrian stopping to dispose of trash;

[0161] The prompting module is also used to determine garbage classification information based on the garbage image, and to prompt the pedestrians to dispose of garbage based on the garbage classification information.

[0162] Furthermore, the prompting module is also used to preprocess the garbage image to obtain a garbage image to be processed;

[0163] The prompting module is also used to perform convolution processing on the garbage image to be processed to obtain a convolution garbage image;

[0164] The prompting module is also used to perform pooling processing on the convolutional garbage image to obtain a garbage feature image;

[0165] The prompting module is also used to determine waste classification information based on the waste feature image.

[0166] Furthermore, the prompting module is also used to determine garbage feature information based on the garbage feature image;

[0167] The prompting module is also used to classify the waste feature information to obtain waste classification information.

[0168] Furthermore, the determining module 6001 is also used to acquire a panoramic image of the trash can, and to calibrate the position of the trash can based on the panoramic image of the trash can to obtain vertex pixel coordinates;

[0169] The determining module 6001 is further configured to determine the trash can marking area based on the vertex pixel coordinates, and to determine the trash disposal area based on the trash can marking area.

[0170] Other embodiments or specific implementations of the waste disposal detection device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0172] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0174] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting waste disposal, characterized in that, The waste disposal detection method includes: Determine the location information of pedestrians and trash cans within the designated waste disposal area; Obtain the skeletal key point information of the stopped pedestrian, and determine the arm swing information based on the skeletal key point information; Garbage disposal detection is performed based on the location information of the trash can and the swing amplitude information of the arm.

2. The method as described in claim 1, characterized in that, The step of obtaining the skeletal key point information of the stopped pedestrian and determining the arm swing information based on the skeletal key point information includes: Obtain the skeletal key point information of the stopped pedestrian, and obtain the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information; Based on the confidence level, select the coordinates of multiple target key points from each key point; The arm swing amplitude information is determined based on the coordinates of multiple target key points.

3. The method as described in claim 2, characterized in that, The step of determining the arm swing information based on the coordinates of multiple target key points includes: Select the hand and arm key point coordinates from multiple target key point coordinates; The arm swing amplitude information is determined based on the coordinates of the key points of the arm and the key points of the palm.

4. The method as described in claim 3, characterized in that, The step of selecting the hand key point coordinates and arm key point coordinates from multiple target key point coordinates includes: Select the hand key point coordinates from multiple target key point coordinates, and determine the trash can coordinates based on the trash can location information; Based on the coordinates of the trash can, determine whether the coordinates of the key points on the palm meet the preset disposal conditions; When the coordinates of the key points on the palm meet the preset deployment conditions, the coordinates of the key points on the arm are selected from the coordinates of the key points on each target.

5. The method as described in claim 3, characterized in that, The step of determining the arm swing amplitude information based on the coordinates of the key points of the arm and the key points of the palm includes: The arm swing angle is determined based on the coordinates of the key points of the arm and the key points of the palm. Determine whether the arm swing angle is greater than or equal to a preset swing threshold; When the arm swing angle is greater than or equal to the preset swing threshold, the arm swing information is determined based on the arm swing angle.

6. The method as described in claim 5, characterized in that, The step of determining the arm swing information based on the arm swing angle includes: Obtain the arm swing duration corresponding to the stopped pedestrian; The arm swing angular velocity is determined based on the arm swing angle and the arm swing time. Determine whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold; When the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold, the arm swing information is determined based on the arm swing angle and the arm swing angular velocity.

7. The method as described in claim 6, characterized in that, After the step of determining whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold, the method further includes: When the arm swing angular velocity is less than the preset swing angular velocity threshold, return to the step of determining the pedestrians staying in the garbage disposal area.

8. The method according to any one of claims 1 to 7, characterized in that, The step of detecting garbage disposal based on the location information of the garbage bin and the arm swing information includes: The duration of the pedestrian's stay in the garbage disposal area is obtained; Determine whether the dwell time is less than or equal to a preset dwell threshold; When the dwell time is less than or equal to the preset dwell threshold, garbage disposal detection is performed based on the garbage bin location information and the arm swing information.

9. The method as described in claim 8, characterized in that, After the step of detecting garbage disposal based on the location information of the trash can and the arm swing information, the method further includes: When a pedestrian is detected disposing of trash, an image of the trash is acquired; Based on the garbage image, garbage classification information is determined, and garbage disposal prompts are given to the pedestrians who are standing still based on the garbage classification information.

10. The method as described in claim 9, characterized in that, The step of determining waste classification information based on the waste image includes: The garbage image is preprocessed to obtain the garbage image to be processed; The garbage image to be processed is subjected to convolution processing to obtain a convolutional garbage image; The convolutional garbage image is subjected to pooling processing to obtain a garbage feature image; Waste classification information is determined based on the waste feature image.

11. The method as described in claim 10, characterized in that, The step of determining waste classification information based on the waste feature image includes: Determine waste feature information based on the waste feature image; The waste feature information is classified to obtain waste classification information.

12. The method as described in claim 1, characterized in that, Before the step of determining the location information of pedestrians and trash cans within the garbage disposal area, the method further includes: Acquire a panoramic image of the trash can, and calibrate the position of the trash can based on the panoramic image to obtain the vertex pixel coordinates; The trash can marking area is determined based on the vertex pixel coordinates, and the trash disposal area is determined based on the trash can marking area.

13. A waste disposal detection device, characterized in that, The waste disposal detection device includes: The determination module is used to determine the location information of pedestrians and trash cans within the garbage disposal area; The acquisition module is used to acquire the skeletal key point information of the stopped pedestrian and determine the arm swing information based on the skeletal key point information; The detection module is used to detect garbage disposal based on the location information of the trash can and the swing amplitude information of the arm.

14. The apparatus as claimed in claim 13, characterized in that, The acquisition module is also used to acquire the skeletal key point information of the stopped pedestrian, and to obtain the coordinates of each key point and the confidence level corresponding to each key point based on the skeletal key point information. The acquisition module is further configured to select multiple target key point coordinates from each key point based on the confidence level; The acquisition module is also used to determine the arm swing amplitude information based on the coordinates of multiple target key points.

15. The apparatus as claimed in claim 14, characterized in that, The acquisition module is also used to select the coordinates of the palm key point and the coordinates of the arm key point from multiple target key point coordinates; The acquisition module is also used to determine arm swing information based on the coordinates of the key points of the arm and the key points of the palm.

16. The apparatus as claimed in claim 15, characterized in that, The acquisition module is also used to select the coordinates of the palm key point from multiple target key point coordinates, and determine the coordinates of the trash can based on the trash can location information; The acquisition module is also used to determine whether the coordinates of the key points of the hand meet the preset disposal conditions based on the coordinates of the trash can. The acquisition module is further configured to select the coordinates of the arm key points from the coordinates of each target key point when the coordinates of the palm key points meet the preset deployment conditions.

17. The apparatus as claimed in claim 15, characterized in that, The acquisition module is also used to determine the arm swing angle based on the coordinates of the key points of the arm and the coordinates of the key points of the palm. The acquisition module is also used to determine whether the arm swing angle is greater than or equal to a preset swing threshold. The acquisition module is further configured to determine arm swing information based on the arm swing angle when the arm swing angle is greater than or equal to the preset swing threshold.

18. The apparatus as claimed in claim 17, characterized in that, The acquisition module is also used to acquire the arm swing time corresponding to the stopped pedestrian; The acquisition module is also used to determine the arm swing angular velocity based on the arm swing angle and the arm swing time; The acquisition module is also used to determine whether the arm swing angular velocity is greater than or equal to a preset swing angular velocity threshold. The acquisition module is further configured to determine arm swing information based on the arm swing angle and the arm swing angular velocity when the arm swing angular velocity is greater than or equal to the preset swing angular velocity threshold.

19. A waste disposal detection device, characterized in that, The device includes: a memory, a processor, and a waste disposal detection program stored in the memory and executable on the processor, the waste disposal detection program being configured to implement the steps of the waste disposal detection method as described in any one of claims 1 to 12.

20. A storage medium, characterized in that, The storage medium stores a waste disposal detection program, which, when executed by a processor, implements the steps of the waste disposal detection method as described in any one of claims 1 to 12.

Images