A method, device and electronic equipment for automatically detecting garbage littering behavior

Abstract

The application provides a method and device for automatically detecting garbage littering behavior and electronic equipment, and relates to the field of behavior detection, which comprises the following steps: acquiring a current frame image and a previous frame image of the current frame image in real time; identifying a detection object in the current frame image; judging whether the detection object exists in the previous frame image; when the detection object does not exist in the previous frame image, marking the current frame image as a first frame image of the detection object; and determining that there is garbage littering behavior based on the detection object in the first frame image. The application can automatically detect fallen garbage, greatly improves the efficiency of processing littered garbage, and reduces the management cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of detection, in particular to a method and device for automatically detecting garbage littering behavior and electronic equipment. BACKGROUND

[0002] With the continuous improvement of people's living standards, in the big cycle of consumption and production, the amount of garbage increases, accompanied by the implementation of garbage classification related policies, garbage littering phenomenon is more serious, making garbage recycling efficiency is low, affecting the city appearance, if relying on a large number of volunteers to participate, increase management costs.

[0003] For the detection of fallen garbage, although the detection accuracy is good, but often need to rely on specific scene and relatively good light, weather conditions. It can not be applied to the real scene in the daytime, night, backlight, strong light changes, and changing weather interference such as rain, snow, dust, fog and other scenes. SUMMARY

[0004] The present application provides a method, device and electronic equipment for automatically detecting garbage littering behavior, which can automatically detect littering garbage and prompt the management personnel to dispose of the littering garbage in time, greatly improving the efficiency of handling littering garbage and reducing the management cost.

[0005] The method for automatically detecting garbage littering behavior provided by the present application adopts the following technical scheme, comprising:

[0006] real-time acquisition of a current frame image and a previous frame image of the current frame image;

[0007] identifying a detection object in the current frame image;

[0008] judging whether the detection object exists in the previous frame image;

[0009] when the detection object does not exist in the previous frame image, marking the current frame image as a first frame image of the detection object;

[0010] based on the detection object in the first frame image, determining that there is garbage littering behavior.

[0011] Optionally, after the real-time acquisition of the current frame image and the previous frame image of the current frame image, the method further comprises:

[0012] judging whether the current frame image and the previous frame image are the same;

[0013] when the current frame image and the previous frame image are not the same, performing quality evaluation on the current frame image to obtain a quality evaluation result;

[0014] adjust the current frame image based on the quality evaluation result.

[0015] Optionally, the adjusting the current frame image based on the quality evaluation result comprises:

[0016] the quality evaluation result comprises no difference, abnormal illumination, and abnormal interference;

[0017] when the quality evaluation result is no difference, the current frame image is not adjusted;

[0018] when the quality evaluation result is abnormal illumination, the current frame image is adjusted by an illumination adjustment model;

[0019] when the quality evaluation result is abnormal interference, the current frame image is adjusted by an interference adjustment model.

[0020] Optionally, the identifying the detection object in the current frame image comprises:

[0021] when the quality evaluation result is no difference, the detection object in the current frame image is identified by a first detection model;

[0022] when the quality evaluation result is abnormal illumination, the detection object in the adjusted current frame image is identified by the first detection model;

[0023] when the quality evaluation result is abnormal interference, the detection object in the adjusted current frame image is identified by a second detection model.

[0024] Optionally, the determining that there is a garbage littering behavior based on the detection object in the first frame image comprises:

[0025] generating a track chain of the detection object based on the position and frame number of the detection object in the first frame image and subsequent continuous frame images;

[0026] tracking the detection object based on the track chain of the detection object;

[0027] determining that there is a garbage littering behavior based on a tracking result of the detection object.

[0028] Optionally, the determining that there is a garbage littering behavior based on the tracking result of the detection object comprises:

[0029] determining whether the detection object meets an object filtering condition;

[0030] when the detection object meets the object filtering condition, canceling tracking of the detection object.

[0031] Optionally, the determining that the garbage is randomly thrown includes:

[0032] The method further includes determining, based on the tracking result of the detection object, whether the tracking result of the detection object meets a pre-warning condition.

[0033] The method further includes determining that the garbage is randomly thrown when the tracking result of the detection object meets the pre-warning condition.

[0034] The application provides an apparatus for automatically detecting a garbage randomly throwing behavior.

[0035] The apparatus includes an acquisition module configured to acquire a current frame image and a previous frame image of the current frame image in real time.

[0036] The apparatus further includes an identification module configured to identify a detection object in the current frame image.

[0037] The apparatus further includes a detection module configured to determine whether the detection object exists in the previous frame image.

[0038] The apparatus further includes a marking module configured to mark the current frame image as a first frame image of the detection object when the detection object does not exist in the previous frame image.

[0039] The apparatus further includes a judgment module configured to determine that the garbage is randomly thrown based on the detection object in the first frame image.

[0040] Optionally, the apparatus further includes a change identification module.

[0041] The change identification module includes:

[0042] A preliminary judgment submodule configured to determine whether the current frame image is the same as the previous frame image.

[0043] An evaluation submodule configured to perform quality evaluation on the current frame image to obtain a quality evaluation result when the current frame image is not the same as the previous frame image.

[0044] An adjustment submodule configured to adjust the current frame image based on the quality evaluation result.

[0045] Optionally, the first adjustment submodule includes:

[0046] The quality evaluation result includes no difference, abnormal illumination, and abnormal interference.

[0047] An adjustment unit configured to not adjust the current frame image when the quality evaluation result is no difference.

[0048] An illumination adjustment unit configured to adjust the current frame image through an illumination adjustment model when the quality evaluation result is abnormal illumination.

[0049] An interference adjustment unit is used to adjust the current frame image through an interference adjustment model when the quality assessment result indicates an interference anomaly.

[0050] Optionally, the identification module includes:

[0051] The indiscriminate recognition submodule is used to identify the detection object of the current frame image through the first detection model when the quality assessment result is indiscriminate.

[0052] The illumination recognition submodule is used to identify the detection object in the adjusted current frame image through a first detection model when the quality assessment result is an illumination anomaly.

[0053] The interference identification submodule is used to identify the detection object in the adjusted current frame image through a second detection model when the quality assessment result is an interference anomaly.

[0054] Optionally, the detection module includes:

[0055] The detection submodule is used to generate a trajectory chain of the detected object based on the position and frame number of the detected object in the first frame image and its subsequent consecutive frame images;

[0056] The tracking submodule is used to track the detected object based on the trajectory chain of the detected object;

[0057] The judgment submodule is used to determine whether there is littering behavior based on the tracking results of the detected object.

[0058] Optionally, before the determination submodule, the following is included:

[0059] A filtering judgment unit is used to determine whether the detected object meets the object filtering conditions;

[0060] A filtering unit is used to cancel tracking of the detected object when the detected object meets the object filtering conditions.

[0061] Optionally, the determination submodule includes:

[0062] The early warning judgment unit is used to determine, based on the tracking results, whether the tracking results of the detected object meet the early warning conditions;

[0063] The early warning unit is used to determine that there is littering behavior when the tracking result of the detected object meets the early warning conditions.

[0064] This specification also provides an electronic device, wherein the electronic device includes:

[0065] Processor; and,

[0066] a memory storing computer-executable instructions that, when executed, cause the processor to perform any of the above methods.

[0067] The specification also provides a computer-readable storage medium storing one or more programs, which when executed by a processor implement any of the above methods.

[0068] In the present application, by acquiring a current frame image and a previous frame image of the current frame image in real time; identifying a detection object in the current frame image; judging whether the detection object exists in the previous frame image; when the detection object does not exist in the previous frame image, marking the current frame image as the first frame image of the detection object; based on the detection object in the first frame image, determining that there is a garbage random throwing behavior, the present application can automatically detect the randomly thrown garbage and prompt the management personnel, urging the management personnel to timely dispose the randomly thrown garbage, greatly improving the efficiency of processing the randomly thrown garbage and reducing the management cost. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A principle schematic diagram of a method for automatically detecting garbage random throwing behavior provided by an embodiment of the present specification;

[0070] Figure 2 A principle schematic diagram of a method for automatically detecting garbage random throwing behavior provided by another embodiment of the present specification;

[0071] Figure 3 A principle schematic diagram of step S280 of a method for automatically detecting garbage random throwing behavior provided by another embodiment of the present specification;

[0072] Figure 4 A flow schematic diagram of a method for automatically detecting garbage random throwing behavior provided by an embodiment of the present specification;

[0073] Figure 5 A structure schematic diagram of a device for automatically detecting garbage random throwing behavior provided by an embodiment of the present specification;

[0074] Figure 6 A structure schematic diagram of an electronic device provided by an embodiment of the present specification;

[0075] Figure 7 A principle schematic diagram of a computer readable medium provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0076] The following description is provided so as to enable any person skilled in the art to practice the application. The preferred embodiments described herein are only examples of the application and the application is not limited to these embodiments. Various modifications to these embodiments can be made by those skilled in the art without departing from the spirit and scope of the application. Those skilled in the art will further appreciate that the application can be used with any assay or apparatus that is configured to detect a target analyte.

[0077] Example embodiments of the present application will now be described more fully with reference to the accompanying drawings. Example embodiments of the present application, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art. Like reference numerals refer to like elements throughout the specification.

[0078] Features, structures, characteristics or other details described in relation to a certain embodiment are not excluded from being combinable with one or more other embodiments in a suitable manner in the light of the technical idea of the present application.

[0079] In the description of the specific embodiments, features, structures, characteristics or other details described in relation to the application are intended to enable a person skilled in the art to fully understand the embodiments. However, it is not excluded that a person skilled in the art can practice the technical solution of the application without one or more of the specific features, structures, characteristics or other details.

[0080] The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0081] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0082] The term "and / or" or "and / or" includes all combinations of one or more of the associated listed items.

[0083] Figure 1 A schematic diagram of the principle of a method for automatically detecting a garbage littering behavior provided for an embodiment of the present specification, the method comprising:

[0084] S110 acquiring a current frame image and a previous frame image of the current frame image in real time;

[0085] S150 identifying a detection object in the current frame image;

[0086] S160 determining whether the detection object exists in the previous frame image;

[0087] S170 marking the current frame image as a first frame image of the detection object when the detection object does not exist in the previous frame image;

[0088] S180 determining that there is a garbage littering behavior based on the detection object in the first frame image.

[0089] In the method,

[0090] S110 acquiring a current frame image and a previous frame image of the current frame image in real time;

[0091] In real-time monitoring of a region by a video, a current frame image and a previous frame image of the current frame image are acquired in real time from the monitoring video, and the current frame image is a 1280x720 or 1920x1080 resolution RGB format image.

[0092] S150 identifying a detection object in the current frame image;

[0093] The detection object of the current frame image is identified by a detection model. As a preferred, it can be determined whether the current frame image and the previous frame image are the same first. If the current frame image and the previous frame image are the same, it means that the detection object in the current frame image is consistent with the detection object in the previous frame image, that is, the detection object exists in the previous frame image.

[0094] S160 determining whether the detection object exists in the previous frame image;

[0095] The detection object is garbage. When automatically detecting a garbage littering behavior, if the detection object does not exist in the previous frame image, it means that the garbage is the latest garbage that is placed. If the detection object exists in the previous frame image, it means that the garbage has been placed in a certain position before, that is, the current frame image is not the first frame image of the detection object.

[0096] S170 marking the current frame image as a first frame image of the detection object when the detection object does not exist in the previous frame image;

[0097] According to the foregoing steps, when the current frame image is different from the previous frame image, a difference point between the two is found, and based on the difference point, it is identified whether the difference object corresponding to the difference point is the detection object;

[0098] Specifically, if the difference object exists in the previous frame image but does not exist in the current frame image, it indicates that the first frame image of the difference object is not the current frame image, i.e., the difference point does not need to be identified.

[0099] If the difference object exists in the current frame image but does not exist in the previous frame image, the difference point can be further identified. If the difference point is the detection object, it indicates that the detection object does not exist in the current frame image, and it is further determined that the current frame image is the first frame image of the detection object, which can improve the frequency of identification and judgment.

[0100] S180 determines that there is a garbage littering behavior based on the detection object in the first frame image.

[0101] Based on the detection object in the first frame image, the detection object is not placed in a designated area, and it is determined that there is a garbage littering behavior. Specifically, when a certain area is monitored in real time, if the area is a greenery cluster, a road, or the like, no garbage should be placed in all positions in the area. If the area is a garbage disposal area, no garbage should be placed in areas other than garbage cans, garbage bins, and areas for collecting garbage. In other words, garbage cans, garbage bins, and areas for collecting garbage are designated areas for placing garbage, and placing garbage in areas outside the designated areas is a garbage littering behavior.

[0102] Figure 2 A schematic diagram of the principle of an automatic garbage littering behavior detection method according to another embodiment of the present specification is provided. In another embodiment, due to real-time shooting of a video, the number of current frame images is large, in order to save computing power, and because the camera for recording a real-time video is usually arranged outdoors, frame images are easily affected by weather changes and lighting conditions. In order to improve the detection accuracy of the detection object in the current frame image, the following content is added:

[0103] S220 determines whether the current frame image is the same as the previous frame image;

[0104] S230 performs quality assessment on the current frame image when the current frame image is different from the previous frame image, to obtain a quality assessment result;

[0105] S240 adjusts the current frame image by adjusting a model;

[0106] wherein,

[0107] S220 judges whether the current frame image and the previous frame image are the same.

[0108] Since the current frame image and the previous frame image of the current frame image are the same in many cases, in order to save computing power, the same picture can be quickly processed. After obtaining the current frame image and the previous frame image of the current frame image, it is judged whether the current frame image and the previous frame image are the same; the block inter-frame difference accumulation method can be used for judgment.

[0109] The inter-frame difference method is a method for obtaining the contour of a moving target by performing difference operation on two consecutive frames of video image sequence. When an abnormal target moves in the monitoring scene, there will be a more obvious difference between the adjacent two frames of images. The absolute value of the pixel value difference of the corresponding position of the two frames is obtained by subtraction, and it is judged whether it is greater than a preset threshold, and then the object motion characteristics of the video or image sequence are analyzed. The mathematical formula is described as follows:

[0110]

[0111] Where D(x, y) is the difference image between the previous frame image of the current frame image and the current frame image, and the shooting time of the current frame image is t, and the shooting time of the previous frame image is t-1, and I(t) and I(t-1) are the images at t and t-1 respectively, that is, the current frame image at t is denoted as I(t), and the previous frame image of the current frame image at t-1 is denoted as I(t-1), T is the threshold value selected when the difference image is binarized, D(x, y) = 1 represents foreground, and D(x, y) = 0 represents background.

[0112] When the current frame image and the previous frame image are the same, the current frame image inherits all the labels of the previous frame image, that is, the labels of the detection object in the previous frame image are copied to the current frame image, and then it is judged whether the detection object meets the early warning condition, and the intermediate steps are directly skipped to save a lot of computing power. Moreover, the block inter-frame difference accumulation method has fast running speed, strong dynamic environment adaptability, and is not sensitive to scene light changes, effectively reducing the hardware cost.

[0113] S230 judges whether the current frame image and the previous frame image are the same.

[0114] In an embodiment of the present specification, when the current frame image is different from the previous frame image, the quality of the current frame image is evaluated by a quality evaluation model to obtain a quality evaluation result. As preferred, the quality evaluation is performed by a rank-based no-reference image quality assessment method (RankIQA model).

[0115] The RankIQA model generates different levels and different types of ranked distorted images by using image processing transformation on pictures with known quality from data preprocessing. Thus, the image quality of the current frame image is evaluated to facilitate targeted adjustment and identification and detection of the detection object in the current frame image.

[0116] After the quality of the current frame image is evaluated by the quality evaluation model, a quality evaluation result is generated, which includes no difference, abnormal light, and abnormal interference.

[0117] Wherein, no abnormality means that the picture quality of the current frame image is normal, the details and colors of the subject captured are well presented, and the picture content is not disturbed;

[0118] Abnormal light means that there is strong light change such as backlight, sidelight, etc. in the current frame image, and the details and colors of the subject captured in the image cannot be well presented;

[0119] Abnormal interference means that due to rain, snow, dust, fog, etc. in the current frame image, the subject captured in the image is blurred or difficult to identify, etc.

[0120] S240 adjusts the current frame image by an adjustment model;

[0121] In order to facilitate identification and detection of the detection object, based on the quality evaluation result, images with abnormal light or abnormal interference need to be adjusted. The adjustment model includes a light adjustment model and an interference adjustment model.

[0122] In an embodiment of the present specification, when the quality evaluation result is no difference, there is no over-bright, over-dark, blurred, etc. in the current frame image, so the current frame image does not need to be adjusted and can be directly identified;

[0123] When the quality evaluation result is abnormal light, the current frame image is adjusted by a light adjustment model. When the current frame image is adjusted by the light adjustment model, the current frame image is first histogram equalized to make the brightness distribution uniform; then the noise introduced in the previous step is median filtered to reduce the noise; and then normalized to generate a light adjusted image.

[0124] When the quality evaluation result is an interference abnormality, the current frame image is adjusted by an interference adjustment model. When the current frame image is adjusted by the interference adjustment model, the current frame image is first subjected to histogram equalization processing to make the brightness distribution uniform; then, the image hazy effect caused by rain and fog is sharpened; and then, normalization processing is performed to generate an interference adjustment image.

[0125] Due to different interference terms in images with illumination abnormalities and interference abnormalities, in order to improve the accuracy of identification detection, the detection model includes a first detection model and a second detection model, and different detection models are used for identification detection of different images.

[0126] In an embodiment of the present specification, when the quality evaluation result is no difference, a detection object of the current frame image is identified by the first detection model;

[0127] When the quality evaluation result is an illumination abnormality, a detection object of the illumination adjustment image is identified by the first detection model. The first detection model is mainly used for target detection and identification of images with normal image quality or images after adjustment by the illumination adjustment model.

[0128] In an embodiment of the present specification, the first detection model is a trained residual network (ResNet), and of course, the first detection model can also be a trained Yolo V5 network.

[0129] ResNet or Yolo V5 is trained by a first training set, so that it can be used for target detection and identification of images with normal image quality or images after adjustment by the illumination adjustment model. The first training samples in the first training set are pictures including regular garbage, special garbage, garbage piles, and interference objects, and part of the pictures without garbage.

[0130] If other new interference objects are found later, they can be added to the training samples to enrich the first training set and perfect the first detection model.

[0131] In the detection process of the first detection model, regular garbage, special garbage, garbage piles, and interference objects in the illumination adjustment image can be detected. When the first detection model identifies one or more of regular garbage, special garbage, and garbage piles in the illumination adjustment image, it will be framed.

[0132] Among them, the regular garbage refers to the commonly seen bagged garbage; the special garbage refers to the uncommon special garbage, which is usually a small piece of waste, such as clothes, shoes, etc. Samples need to be added or deleted according to the reference scene, otherwise false positives and false negatives will be caused; the garbage pile mainly refers to a pile of garbage whose composition degree far exceeds that of ordinary garbage, such as construction waste and discarded furniture, etc.

[0133] When the first detection model identifies that there is an interference object in the light adjustment image, the interference object is directly excluded and will not be framed.

[0134] When the quality evaluation result is an interference exception, the detection object of the interference adjustment image is identified by the second detection model.

[0135] In an embodiment of the present specification, since the second detection model is mainly used for target detection and identification of the interference adjustment image adjusted by the interference adjustment model, in the detection process of the second detection model, in addition to the conventional garbage, special garbage, garbage pile and interference object, raindrops, snowflakes and other relatively independent interference targets can also be detected.

[0136] When the second detection model identifies that there is one or more of the conventional garbage, special garbage, and garbage pile in the interference adjustment image, it will be framed.

[0137] When the second detection model identifies that there is an interference object and / or interference target in the interference adjustment image, the interference object and / or interference target will be directly excluded and will not be framed. However, if it is a very severe weather condition, such as a severe interference condition like a storm, a large number of false positives will inevitably occur.

[0138] In an embodiment of the present specification, the second detection model can be a trained residual network (ResNet). Of course, the second detection model can also be a trained Yolo V5 network.

[0139] ResNet or Yolo V5 is trained by a second training set to detect and identify the detection object of the interference adjustment image adjusted by the interference adjustment model. The second training samples in the second training set are pictures including conventional garbage, special garbage, garbage pile, interference object, part of the picture without garbage, and pictures including raindrops, snowflakes and other interference targets. The picture including the interference target can be a picture taken on a rainy day, a foggy day, or a snowy day.

[0140] If other new interference objects are found later, they can be added to the training samples to enrich the second training set and improve the second detection model.

[0141] Since the training samples of the first detection model and the second detection model are different, if the second detection model is used to identify the detection light adjustment image, the detection speed may be slowed down; if the first detection model is used to identify the detection interference adjustment image, the detection result may be inaccurate. Therefore, the first detection model and the second detection model cannot be used to replace each other.

[0142] The present application has strong robustness, can adapt to various complex and changeable scenes in reality, and through comprehensive use of multiple deep learning-based models and auxiliary multiple image processing technologies, excellent effects are achieved under normal conditions, and the picture processing and recognition effects under strong light change and meteorological interference conditions are improved.

[0143] Figure 3 A principle schematic diagram of step S280 of a method for automatically detecting garbage random throwing behavior according to another embodiment of the present application is provided, based on the detection object in the first frame image, the detection object is not placed in the designated area, that is, belongs to the garbage random throwing behavior, in order to further confirm the garbage random throwing behavior, the tracking model is used to track the detection object and remind the management user to process, and specifically includes:

[0144] S281 generates a track chain of the detection object based on the position and frame number of the detection object in the first frame image and subsequent continuous frame images, and tracks the detection object based on the track chain of the detection object.

[0145] S282 filters the detection object.

[0146] S283 judges whether the detection object meets the early warning condition.

[0147] S284 determines that there is garbage random throwing behavior when the tracking result of the detection object meets the early warning condition.

[0148] Among them,

[0149] S281 generates a track chain of the detection object based on the position and frame number of the detection object in the first frame image and subsequent continuous frame images, and tracks the detection object based on the track chain of the detection object. The position can be presented in the form of coordinates.

[0150] In an embodiment of the present application, the tracking model corresponds the same detection object in continuous frames, such as the same bag of garbage, uses the same ID mark to determine the correspondence of the detection object in the previous and subsequent frames, and maintains the uniqueness of the detection object.

[0151] In one embodiment of the present disclosure, the tracking model is an Intersection over Union method that uses the intersection between bounding boxes detected across frames to track objects, which has high performance but unsatisfactory accuracy in complex situations.

[0152] In another embodiment of the present disclosure, in places or areas where high accuracy is required, a multi-object tracking method (MOT method) is used for feature extraction and matching, and the position of the detected object in the current frame image is found based on the position of the detected object in the previous frame image. If the detected object with similar texture is found, the information of the detected object in the current frame image is added to the track chain of the detected object.

[0153] If the detected object is not in the previous frame image, the current frame image is the first frame image of the detected object, and a new track chain is established for the detected object.

[0154] Based on the shooting position or scene of the monitoring camera, the tracking model to be used is selected according to actual needs, thereby balancing accuracy and performance.

[0155] S282 filters the detected object;

[0156] Since there are errors in the target detection and recognition of the detection model, the detected object can be subjected to target filtering, the object filtering condition is set through the scene, and invalid detected objects are filtered, so that the result of recognition and detection is more accurate. The invalid detected object refers to a detected object that does not conform to prior knowledge, for example, an object that is too high or has a volume that is too large has a very small probability of being garbage.

[0157] In one embodiment of the present disclosure, the object filtering condition includes a size filtering condition and an aspect ratio filtering condition. The size filtering condition refers to the size of the detected object being outside a preset size range, and the aspect ratio filtering condition refers to the aspect ratio of the detected object being outside a preset aspect ratio range.

[0158] When the detected object does not meet the size filtering condition and the aspect ratio filtering condition, it is determined that the detected object does not meet the object filtering condition, and the detected object is identified as a valid detected object.

[0159] When the detected object meets the size filtering condition or the aspect ratio filtering condition, it is determined that the detected object meets the object filtering condition, and the detected object is identified as an invalid detected object, and the current tracking of the detected object is cancelled.

[0160] When the next frame image of the current image is acquired, the invalid detection object can still be detected, recognized and tracked by the first detection model or the second detection model. When target filtering is performed, it is determined that the detection object meets the object filtering condition, and the tracking of the detection object in this time is cancelled.

[0161] It is worth noting that due to the timely processing of the garbage by the management personnel, there is a case that the detection object does not exist in the current frame image. At this time, the result obtained according to the tracking model is empty.

[0162] S283 determines whether the detection object meets the early warning condition.

[0163] After filtering the invalid detection object, whether the garbage is randomly thrown can be determined by judging whether the valid detection object meets the early warning condition.

[0164] In an embodiment of the present specification, when it is determined whether the detection object meets the early warning condition, only the detection object that is not cancelled tracking is determined to meet the early warning condition. The detection object that is cancelled tracking will not be determined to meet the early warning condition this time.

[0165] The early warning condition can be set as needed, for example, the early warning can be set based on the size of the detection object, the aspect ratio range of the detection object, the position of the detection object, the duration of the existence of the detection object, etc.

[0166] In an embodiment of the present specification, the early warning condition includes a position early warning condition and a duration early warning condition.

[0167] The position early warning condition is that the detection object in the first frame image is located in the preset position. Specifically, the coordinate range of the preset position can be set by manual setting, and the coordinate of the detection object in the first frame image can be obtained by the tracking model. If the coordinate is located in the coordinate range of the preset position, it is determined that the detection object meets the position early warning condition.

[0168] The duration early warning condition is that the duration of the existence of the detection object is not less than M. Wherein, the preset duration is M. If the interval time between the two consecutive frames is a, and a total of b frame images (including the first frame image) are taken from the first frame image, that is, the current frame image is the bth frame image of the detection image, when a(b-1)≥M, it is determined that the detection object meets the duration early warning condition; when a(b-1)<M, it is determined that the detection object does not meet the duration early warning condition.

[0169] When the detection object meets the position warning condition and the time length warning condition, it is determined that the detection object meets the warning condition.

[0170] In the foregoing step, after determining whether the current frame image and the previous frame image are the same, when the current frame image and the previous frame image are the same, the step of determining whether the detection object meets the warning condition is directly jumped to, so as to reduce the load. Before jumping to the step, the current frame image inherits all the marks of the previous frame image. Here, the previous frame image is the previous frame image that has been subjected to target tracking and is output from the target tracking model, rather than the previous frame image that has been subjected to target filtering.

[0171] S284When the tracking result of the detection object meets the warning condition, it is determined that there is a garbage littering behavior.

[0172] After the violation is identified, pictures and videos related to the detection object are stored, information is sent to a management platform, and management personnel are notified to handle, clean up, etc. The management personnel can be property personnel, garbage cleaning personnel, etc.

[0173] In an embodiment of the present specification, the detection object will only be warned once. That is, after the detection object is warned, if the subsequent tracking result of the detection object still meets the warning condition, no further warning will be given. The warning condition can also be used to determine whether the video image has a frame sticking situation. That is, if the frame is stuck for a long time, the warning condition can also be used to remind.

[0174] In an embodiment of the present specification, in order to reduce the algorithm requirement of the system, a pyramid segmentation method can be used to generate multiple copies of the current frame image with different resolutions; under the premise of ensuring the effect, the model of the present specification tries to select a low-resolution image when obtaining the current frame image, thereby reducing the hardware cost.

[0175] The image pyramid is a structure for explaining the image in multiple resolutions. N images with different resolutions are generated by performing multi-scale pixel sampling on the original image. The image with the highest level of resolution is placed at the bottom and arranged in a pyramid shape. Above it is a series of images with gradually reduced pixels (size), up to the top of the pyramid, which only contains an image with a single pixel. This constitutes the image pyramid.

[0176] In an embodiment of the present specification, by using the Gaussian pyramid segmentation method, the image is down-sampled and reduced: the implementation of the Gaussian pyramid segmentation method includes two steps: Gaussian low-pass filtering and down-sampling. First, the image is convolved with a Gaussian kernel; then down-sampling is performed to obtain the target image at different scales.

[0177] To obtain the Gaussian pyramid image of level i+1 layer: 1. Gaussian kernel convolution is performed on the image i; 2. All even rows and columns are removed. The resulting image is the image of the i+1 layer, and the resulting image is only one quarter of the original image. By repeatedly iterating the above steps on the input image i layer (original image), the entire pyramid can be obtained, and the down-sampling gradually loses the information of the image. Through the down-sampling operation of the image, the image is reduced.

[0178] In another embodiment of the present specification, the image is enlarged by up-sampling through the Laplacian pyramid segmentation method. The Laplacian pyramid segmentation method is based on the Gaussian pyramid segmentation method, that is, the difference between the different layers of the Gaussian pyramid. The specific steps are: 1. The image is enlarged to twice the original size in each direction, and the newly added rows and columns are filled with 0; 2. The same kernel as before (multiplied by 4) is used to convolve the enlarged image to obtain the approximate value of the "newly added pixels". The resulting image is the enlarged image, but it will be found to be blurred compared to the original image, because some information has been lost during the scaling process.

[0179] The model in the present specification is pruned to make it run on a low-power device, and the pruned model includes: NNI automatic machine learning toolkit, Autogluon framework.

[0180] By using the NNI automatic machine learning toolkit, the time consumption of manual parameter tuning can be saved, the process of hyperparameter search can be accelerated and simplified, and the framework can be highly customized; by using the Autogluon framework, the parameters can be quickly tuned. After optimization, the hardware cost is effectively reduced, so that it can run on a low-power device. It has cost advantage and is easy to popularize and promote.

[0181] In combination Figure 4 In an embodiment of the present specification, frame images are obtained in real time and are respectively named A, B, C, …, and the interval time between two consecutive frames is T;

[0182] A image includes regular garbage L1 (first appearance), interference object L2, and invalid detection object L3.

[0183] B image includes regular garbage L1, interference object L2, invalid detection object L3, and special garbage L4.

[0184] C image is identical to B image.

[0185] Real-time acquisition of the current frame image B, judge whether B and A are the same; since B and A are not the same, the quality of B is evaluated through the quality evaluation model, if the quality evaluation result of B is no exception, B is input to the first detection model to detect and identify the detection object; if the quality evaluation result of B is light abnormality, after the B image is adjusted by the light adjustment model, the adjusted image is input to the first detection model to detect and identify the detection object; if the quality evaluation result of B is interference abnormality, after the B image is adjusted by the interference adjustment model, the adjusted image is input to the second detection model to detect and identify the detection object;

[0186] Through the first detection model or the second detection model, it is detected that B has regular garbage L1, interference object L2, L3, and special garbage L4, and the interference object L2 is excluded, at this time, B is marked with regular garbage L1, L3, and special garbage L4, and is recorded as B' image;

[0187] Then, L1, L3 and L4 are tracked through the tracking model, wherein, since the first frame image of L1 is A, based on A and B, a track chain based on L1 is generated; based on the first frame image of L3, from the first frame image of L3 to B (including the first frame image of L3 and B), a track chain based on L3 is generated; B is the first frame image of L4, and a new track chain of L4 is created.

[0188] Judge whether L1, L3 and L4 in B meet the object filtering condition, at this time, the invalid detection object L3 is excluded;

[0189] Determine whether L1 and L4 in B meet the early warning condition, if the early warning condition is met, save and send the early warning information, and notify the management personnel to handle in time.

[0190] Real-time acquisition of the current frame image C, judge whether C and B are the same;

[0191] C is the same as B, then C inherits the mark in B'(marked with regular garbage L1, L3, and special garbage L4), and the mark of L1, L3 and L4 in C is completely consistent with the mark in the output B after detection;

[0192] Determine whether L1, L3 and L4 in C meet the early warning condition, if the early warning condition is met, save and send the early warning information, and notify the management personnel to handle in time.

[0193] Figure 5 The structure schematic diagram of the device for automatically detecting garbage disorderly throwing behavior provided by the embodiment of the present application can comprise:

[0194] The acquisition module 510 is configured to acquire a current frame image and a previous frame image of the current frame image in real time.

[0195] an identification module 520, configured to identify a detection object in the current frame image;

[0196] a detection module 530, configured to determine whether the detection object exists in the previous frame image;

[0197] a marking module 540, configured to mark the current frame image as a first frame image of the detection object when the detection object does not exist in the previous frame image;

[0198] a determination module 550, configured to determine that there is a garbage littering behavior based on the detection object in the first frame image.

[0199] Optionally, the method further comprises a change identification module 520;

[0200] the change identification module 520 comprises:

[0201] a preliminary determination submodule, configured to determine whether the current frame image is the same as the previous frame image;

[0202] an evaluation submodule, configured to perform quality evaluation on the current frame image to obtain a quality evaluation result when the current frame image is not the same as the previous frame image;

[0203] an adjustment submodule, configured to adjust the current frame image based on the quality evaluation result.

[0204] Optionally, the first adjustment submodule comprises:

[0205] the quality evaluation result comprises no difference, abnormal illumination, and abnormal interference;

[0206] an adjustment unit, configured to not adjust the current frame image when the quality evaluation result is no difference;

[0207] an illumination adjustment unit, configured to adjust the current frame image through an illumination adjustment model when the quality evaluation result is abnormal illumination;

[0208] an interference adjustment unit, configured to adjust the current frame image through an interference adjustment model when the quality evaluation result is abnormal interference.

[0209] Optionally, the identification module 520 comprises:

[0210] a no-difference identification submodule, configured to identify a detection object of the current frame image through a first detection model when the quality evaluation result is no difference;

[0211] An illumination recognition sub-module is configured to, when the quality evaluation result is illumination abnormality, identify the detection object of the adjusted current frame image by using a first detection model.

[0212] An interference recognition sub-module is configured to, when the quality evaluation result is interference abnormality, identify the detection object of the adjusted current frame image by using a second detection model.

[0213] Optionally, the detection module 530 comprises:

[0214] A detection sub-module is configured to generate a track chain of the detection object based on the position of the detection object and the frame number of the first frame image and the subsequent continuous frame images.

[0215] A tracking sub-module is configured to track the detection object based on the track chain of the detection object.

[0216] A judgment sub-module is configured to determine that there is a garbage random throwing behavior based on the tracking result of the detection object.

[0217] Optionally, before the judgment sub-module, the method comprises:

[0218] A filtering judgment unit is configured to judge whether the detection object meets an object filtering condition.

[0219] A filtering unit is configured to cancel the tracking of the detection object when the detection object meets the object filtering condition.

[0220] Optionally, the judgment sub-module comprises:

[0221] A pre-warning judgment unit is configured to judge whether the tracking result of the detection object meets a pre-warning condition based on the tracking result.

[0222] A pre-warning unit is configured to determine that there is a garbage random throwing behavior when the tracking result of the detection object meets the pre-warning condition.

[0223] The functions of the device of the embodiment have been described in the above method embodiment, and thus the description of the embodiment will not be described in detail, and the relevant description in the foregoing embodiment can be referred to.

[0224] Based on the same inventive concept, the embodiments of the present specification also provide an electronic device.

[0225] The electronic device embodiment of the present application is described below, which can be regarded as a specific physical implementation of the above-mentioned method and device embodiments of the present application. For the details described in the electronic device embodiment of the present application, it should be regarded as a supplement to the above-mentioned method or device embodiments; for the details not disclosed in the electronic device embodiment of the present application, it can be implemented with reference to the above-mentioned method or device embodiments.

[0226] Figure 6 A structural schematic diagram of an electronic device is provided for the embodiment of the present specification. The electronic device 600 according to the embodiment of the present application is described below with reference to Figure 6 Figure 6 The displayed electronic device 600 is only an example and should not bring any limitation to the function and use range of the embodiment of the present application.

[0227] As shown in Figure 6 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components including the storage unit 620 and the processing unit 610, a display unit 640, etc.

[0228] The storage unit stores program codes which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the processing method part of the present specification. For example, the processing unit 610 can perform the steps as shown in Figure 1

[0229] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0230] The storage unit 620 can also include program / utilities 6204 with a set of (at least one) program modules 6205, such as but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof can include the implementation of a network environment.

[0231] The bus 630 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0232] ​​The electronic device 600 can also communicate with one or more external devices 670 such as a keyboard or pointing device, a Bluetooth device, or a Figure 6 device that enables a user to interact with the electronic device 600 and / or any devices (e.g., a router, a modem, a switch, a bridge etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet, through a network adapter 660. The network adapter 660 can be any of a plurality of different types of adapters, such as a

[0233] It should be appreciated that the exemplary embodiments described herein can be implemented in software and / or in a manner that is too hardware dependent to be realized in software alone. Furthermore, although the exemplary embodiments described herein can make reference to software and / or hardware, it will be appreciated that a reference to software can be interpreted as a reference to simply a tangible medium that stores the software. Software in this description should be interpreted as instructions that cause a processor to perform certain steps or functions. Such software can be stored in a memory, such as a ROM, a RAM, a hard disk, a floppy disk, a magnetic tape, an optical data storage device, and the like. Such software can also be stored in an electrical, optical, acoustical or other form of carrier wave or propagated signal, and the like. Figure 1

[0234] Figure 7 A schematic diagram of a computer readable medium according to an embodiment of the present disclosure.

[0235] Implementation Figure 1 ​The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0236] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0237] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0238] In summary, the present application can be implemented in hardware, or implemented in software modules running on one or more processors, or implemented in a combination of both. Those skilled in the art should understand that some or all of the functions of some or all of the components according to the embodiments of the present application can be implemented in practice using a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP). The present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0239] The above-described specific embodiments further illustrate the purposes, technical solutions and beneficial effects of the present application. It should be understood that the present application is not inherently related to any specific computer, virtual device or electronic device, and various general-purpose devices can implement the present application. The above-described specific embodiments are merely examples of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0240] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be mutually referred to. Each of the embodiments mainly describes the differences from other embodiments.

[0241] The above-described specific embodiments are merely examples of the present application and are not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of automatically detecting littering behavior, characterized by, include: Real-time acquisition of the current frame image and the previous frame image of the current frame image; Identify the detection object in the current frame image; wherein, when the current frame image is different from the previous frame image, perform a quality assessment on the current frame image to obtain a quality assessment result; the quality assessment result includes: no difference, abnormal lighting, and abnormal interference; wherein, no abnormality means that the image quality of the current frame image is normal; abnormal lighting means that there is a strong change in lighting in the current frame image; abnormal interference means that the subject captured in the current frame image is blurred or difficult to identify due to interference from rain, snow, dust, or fog; When the quality assessment result is no difference, identification is performed directly; when the quality assessment result is abnormal lighting, histogram equalization is performed on the current frame image while adjusting it through the lighting adjustment model to make the brightness distribution uniform; then filtering is performed to apply median filtering to the noise introduced in the previous step, followed by normalization to generate a lighting adjustment image; the detection object in the lighting adjustment image is identified through the first detection model, and when the first detection model identifies an interfering object in the lighting adjustment image, the interfering object is directly excluded; When the quality assessment result is an interference anomaly, when adjusting the current frame image through the interference adjustment model, the current frame image is subjected to histogram equalization, the image haze caused by rain and fog is sharpened, and then normalization is performed to generate interference adjustment objects; the detection objects of the interference adjustment objects are identified by the second detection model, and interference objects and / or interference targets are excluded. The interference objects are objects that are highly similar to garbage. Determine whether the detected object exists in the previous frame image; When the detected object is not present in the previous frame image, the current frame image is marked as the first frame image of the detected object; Based on the detected objects in the first frame image, it is determined that there is littering behavior; wherein, when the detected object does not meet the size filtering condition and aspect ratio filtering condition, it is determined that the detected object does not meet the object filtering condition and the detected object is identified as a valid detected object; when the detected object meets the size filtering condition or aspect ratio filtering condition, it is determined that the detected object meets the object filtering condition and the detected object is identified as an invalid detected object.

2. The method of claim 1, wherein, After acquiring the current frame image and the previous frame image in real time, the method further includes: Determine whether the current frame image is the same as the previous frame image; When the current frame image is different from the previous frame image, a quality assessment is performed on the current frame image to obtain a quality assessment result; Based on the quality assessment results, adjust the current frame image.

3. The method of claim 2, wherein, Adjusting the current frame image based on the quality assessment result includes: The quality assessment results include: no difference, abnormal lighting, and abnormal interference. When the quality assessment result is no difference, the current frame image is not adjusted; When the quality assessment result indicates abnormal lighting, the current frame image is adjusted using a lighting adjustment model. When the quality assessment result indicates an interference anomaly, the current frame image is adjusted using an interference adjustment model.

4. The method of claim 3, wherein, The identifying the detected object in the current frame image comprises: When the quality evaluation result is no difference, the first detection model is used to identify the detected object of the current frame image; When the quality evaluation result is abnormal illumination, the first detection model is used to identify the detected object of the adjusted current frame image; When the quality evaluation result is abnormal interference, the second detection model is used to identify the detected object of the adjusted current frame image.

5. The method of claim 1, wherein, The determining of the garbage throwing disorderly behavior based on the detected object in the first frame image comprises: Based on the position and frame number of the detected object in the first frame image and the subsequent continuous frame images, a track chain of the detected object is generated; Based on the track chain of the detected object, the detected object is tracked; Based on the tracking result of the detected object, it is determined that the garbage throwing disorderly behavior exists.

6. The method of claim 5, wherein, Before the determining of the garbage throwing disorderly behavior based on the tracking result of the detected object, it comprises: It is judged whether the detected object meets the object filtering condition; When the detected object meets the object filtering condition, the tracking of the detected object is cancelled.

7. The method of claim 5, wherein, The determining of the garbage throwing disorderly behavior based on the tracking result of the detected object comprises: Based on the tracking result, it is judged whether the tracking result of the detected object meets the early warning condition; When the tracking result of the detected object meets the early warning condition, it is determined that the garbage throwing disorderly behavior exists.

8. An apparatus for automatically detecting illegal dumping behavior, characterized in that, It comprises: An acquisition module is configured to acquire a current frame image and a previous frame image of the current frame image in real time; An identification module is configured to identify a detected object in the current frame image; when the current frame image is different from the previous frame image, the current frame image is subjected to quality evaluation to obtain a quality evaluation result; the quality evaluation result comprises no difference, abnormal illumination, and abnormal interference; no difference means that the picture quality of the current frame image is normal; abnormal illumination means that there is strong illumination change in the current frame image; abnormal interference means that due to the interference of rain, snow, dust, and fog, the subject in the image is blurred or difficult to recognize; When the quality evaluation result is no difference, direct identification is performed; when the quality evaluation result is abnormal illumination, the current frame image is subjected to histogram equalization processing to make the brightness distribution uniform when the current frame image is adjusted by an illumination adjustment model; then, filtering processing is performed to subject the noise introduced in the previous step to median filtering, normalization processing is performed, and an illumination adjustment image is generated; the detected object of the illumination adjustment image is identified by the first detection model, and when the first detection model identifies that there is an interference object in the illumination adjustment image, the interference object is directly excluded; When the quality evaluation result is abnormal interference, the current frame image is subjected to histogram equalization processing to sharpen the hazy effect of the picture caused by rain and fog when the current frame image is adjusted by an interference adjustment model, and then normalization processing is performed to generate an interference adjustment object; the detected object of the interference adjustment object is identified by the second detection model, and the interference object and / or interference target are excluded, and the interference object is an object similar to garbage in height; A detection module is configured to determine whether the detection object exists in the previous frame of image. A marking module is configured to mark the current frame of image as a first frame of image of the detection object when the detection object does not exist in the previous frame of image. A judging module is configured to determine that the garbage is randomly thrown based on the detection object in the first frame of image.

9. An electronic device, comprising: The electronic device includes: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-7.

10. A computer readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any of claims 1-7.

Images