Garbage discarding behavior detection method and device, equipment and storage medium

Abstract

The invention provides a garbage discarding behavior detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining a monitoring video stream, and extracting a corresponding video frame image according to a frame; performing target identification on the video frame image, and determining a human body identification result and a target object identification result; determining a space-time interaction relationship between the human body and the target object according to the human body recognition result and the target object recognition result in the first time period; analyzing the object state according to the target object recognition result of the second time period to obtain an object state result; analyzing the human body state according to the human body recognition result of the third time period to obtain a human body state result; and determining a corresponding garbage throwing behavior detection result based on the space-time interaction relationship, the object state result and the human body state result. The time-space interaction relation between the human body and the target object, the object state and the human body state are sequentially analyzed according to the time sequence, the violation behavior is comprehensively judged, and the misjudgment and missed judgment risks are greatly reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting littering behavior. Background Technology

[0002] With the acceleration of urbanization and the continuous increase in residential density in communities, littering in public areas such as inside building entrances and unit entrances has become a prominent pain point in community environmental sanitation management. Some residents, for convenience, directly discard their household waste in these areas instead of designated garbage stations, negatively impacting the community living experience.

[0003] Currently, the main approach to addressing these issues is manual inspection, relying on property management staff to regularly patrol the rooftops or manually reviewing surveillance footage after receiving resident complaints. This method is relatively costly and inefficient. While some related technologies attempt to use computer vision to detect littering, their algorithms typically only detect a single target. For example, they might only analyze whether discarding actions exist within a target area to confirm littering, easily misclassifying normal daily activities as littering. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for detecting littering behavior, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0005] A first aspect of this disclosure provides a method for detecting littering behavior, the method comprising: Acquire the surveillance video stream and extract the corresponding video frame images frame by frame; Target recognition is performed on the video frame images to determine the human body recognition result and the target object recognition result; The spatiotemporal interaction relationship between the human body and the target object is determined based on the human body recognition results and the target object recognition results in the first time period. The object state is analyzed based on the target object recognition results of the second time period to obtain the object state result. The start time of the second time period corresponds to the end time of the first time period. The human body status is analyzed based on the human body recognition results of the third time period to obtain the human body status result. The start time of the third time period corresponds to the end time of the second time period. Based on the spatiotemporal interaction relationship, the object state result, and the human body state result, the corresponding littering behavior detection result is determined.

[0006] In one possible implementation, the step of performing target recognition on the video frame image to determine the human body recognition result and the target object recognition result includes: Feature extraction is performed on each video frame image to obtain corresponding image features, which include texture features, color features, shape features and spatial context features; Based on the image features, target classification and localization are performed to determine the corresponding human body recognition results and target object recognition results.

[0007] In one possible implementation, determining the spatiotemporal interaction relationship between the human body and the object based on the human body recognition result and the target object recognition result in the first time period includes: Perform region recognition on the video frame image to determine the region coordinate information of the target region; Determine the human coordinate information in the human body recognition result and the object coordinate information in the target object recognition result, and determine whether the human body coordinate information and the object coordinate information match the area coordinate information within the first time period; In response to the fact that both the human body coordinate information and the object coordinate information are within the coordinate range of the region coordinate information, the coordinate spacing between the human body coordinate information and the object coordinate information is determined; The spatial interaction relationship between the human body and the object within the target area is determined based on the relationship between the coordinate spacing and the target spacing threshold, and the temporal interaction relationship between the human body and the object within the target area is determined based on the duration when the coordinate spacing is less than the target spacing threshold.

[0008] In one possible implementation, the step of analyzing the object state based on the target object recognition result in the second time period to obtain the object state result includes: The object movement characteristics are determined based on the object coordinate information corresponding to the first target time interval in the second time period. The object movement characteristics include the object movement direction and the object movement speed. The object dwelling characteristics are determined based on the object coordinate information corresponding to the second target time interval in the second time period. The object dwelling characteristics represent the dwelling state of the object in the target area, and the time start point of the second target time interval corresponds to the time end point of the first target time interval. In response to the object's movement feature satisfying the target movement condition and the object's dwelling feature satisfying the target dwelling condition, the corresponding object state result is determined to be a discarded state.

[0009] In one possible implementation, the step of analyzing the human body state based on the human body recognition result in the third time period to obtain the human body state result includes: The human movement trajectory and the state of the object being held by the human are determined based on the human recognition results during the third time period. In response to the human body's movement trajectory satisfying the condition of leaving the target area, and the human body holding an object satisfying the empty-handed state, the corresponding human body state result is determined to be the empty-handed leaving state.

[0010] In one possible implementation, determining the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, the object state result, and the human body state result includes: Based on the spatiotemporal interaction relationship, it is determined whether there is a person-related event. The person-related event indicates that both the human body and the object are within the target area, and there is a relationship between the human body and the object. In response to the existence of the associated event of the person, the corresponding object state result is determined; In response to the object's state being discarded, the corresponding human body state is determined. In response to the human body leaving empty-handed, it is determined that littering has occurred.

[0011] In one possible implementation, the method further includes: Based on the littering behavior detection results, the corresponding video frame images are traced back, and target video frame images containing human faces are selected. Facial features are extracted from the target video frame image, and the corresponding facial features are compared with pre-stored template facial features to determine the corresponding identity information, so as to send the corresponding alarm prompt based on the identity information.

[0012] A second aspect of this disclosure provides a littering behavior detection device for implementing the littering behavior detection method described in any of the preceding claims, the device comprising: The image acquisition module is used to acquire the monitoring video stream and extract the corresponding video frame images frame by frame; The target recognition module is used to perform target recognition on the video frame images and determine the human body recognition result and the target object recognition result; The spatiotemporal relationship determination module is used to determine the spatiotemporal interaction relationship between the human body and the object based on the human body recognition result and the target object recognition result in the first time period. The object state determination module is used to analyze the object state based on the target object recognition result in the second time period to obtain the object state result. The start time of the second time period corresponds to the end time of the first time period. The human body state determination module is used to analyze the human body state based on the human body recognition results of the third time period to obtain the human body state result. The start time of the third time period corresponds to the end time of the second time period. The result determination module is used to determine the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, the object state result, and the human body state result.

[0013] A third aspect of this disclosure provides an electronic device comprising: At least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the littering behavior detection method of this disclosure.

[0014] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the littering behavior detection method described in this disclosure.

[0015] This disclosure discloses a method for detecting littering behavior, including acquiring a surveillance video stream and extracting corresponding video frame images frame by frame; performing target recognition on the video frame images to determine human body recognition results and target object recognition results; determining the spatiotemporal interaction relationship between the human body and the target object based on the human body recognition results and target object recognition results in the first time period; analyzing the object state based on the target object recognition results in the second time period to obtain the object state result, with the start time of the second time period corresponding to the end time of the first time period; analyzing the human body state based on the human body recognition results in the third time period to obtain the human body state result, with the start time of the third time period corresponding to the end time of the second time period; and determining the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, object state result, and human body state result. This disclosure analyzes the spatiotemporal interaction relationship between the human body and the target object, the object state, and the human body state sequentially according to time sequence, comprehensively judging the violation behavior. Through the seamless detection of the three time periods, it achieves full-process automation of target recognition, dynamic association, and behavior judgment. Based on temporal analysis, it significantly reduces the risk of false positives and false negatives, and compared with manual inspection, it has higher detection efficiency and effectively reduces the false positive rate compared with simple target detection.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0018] Figure 1 This illustration shows the implementation flow of a littering behavior detection method according to an embodiment of the present disclosure. Figure 1 ; Figure 2 This illustration shows the implementation flow of a littering behavior detection method according to an embodiment of the present disclosure. Figure 2 ; Figure 3 This illustration shows the implementation flow of a littering behavior detection method according to an embodiment of the present disclosure. Figure 3 ; Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown; Figure 5 A schematic diagram of a littering behavior detection device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0020] This disclosure provides a method for detecting littering behavior, such as... Figure 1 As shown, the method includes: S101. Obtain the monitoring video stream and extract the corresponding video frame images frame by frame; In this step, surveillance cameras deployed inside the main entrances of each building in the community are used to collect continuous video streams in real time. These cameras cover the no-littering zones near the entrances. The acquired video streams are extracted frame by frame to obtain continuous video frame images, which serve as the basis for subsequent detection of littering. When extracting the corresponding video frame images, frame-by-frame sampling is used; for example, one frame is extracted every five frames, and invalid video frames are filtered out to reduce computational resource consumption and improve detection speed.

[0021] S102. Perform target recognition on video frame images to determine the human body recognition result and the target object recognition result; In this step, an object detection model is used to perform object recognition on consecutive video frames to identify the corresponding person and object from the video frames. Objects include garbage bags, delivery packages, and toys. It should be noted that the target object in this step is a garbage bag. During the object recognition and detection process, the object detection model analyzes features such as the object's color, texture, shape, and context to determine whether an object in the image is the target object, i.e., a garbage bag. This avoids misidentifying non-garbage items such as delivery packages or toys as garbage bags, thereby reducing false positives for littering.

[0022] S103. Determine the spatiotemporal interaction relationship between the human body and the object based on the human body recognition results and object recognition results of the first time period. In this step, after target recognition is performed on consecutive video frames, each frame yields human body recognition results and target object recognition results, including bounding box coordinates, category labels, and confidence scores. Based on the human body recognition results and target object recognition results corresponding to multiple consecutive video frames within the first time period, the spatiotemporal interaction relationship between the human body and the target object is analyzed. Specifically, this determines whether the human body and the target object are simultaneously located in a no-littering zone, and if so, whether there is a correlation between the person and the target object. In other words, it determines whether a person is carrying a trash bag and is present in a no-littering zone.

[0023] S104. Analyze the object state based on the target object recognition results of the second time period to obtain the object state results. The start time of the second time period corresponds to the end time of the first time period. In this step, when it is determined that the person and the target object are within a no-littering zone, it is further determined whether the target object was thrown and remained in the no-littering zone. Accordingly, the target object identification results corresponding to multiple consecutive video frames within the second time period are analyzed and determined. It should be noted that the start time of the second time period is connected to the end time of the first time period, representing the subsequent time period after determining the spatiotemporal interaction relationship. For example, after continuously analyzing X frames to determine the spatiotemporal interaction relationship between the person and the target object, further analysis of the subsequent Y frames is performed to determine the corresponding target object state.

[0024] S105. Analyze the human body status based on the human body recognition results of the third time period to obtain the human body status results. The start time of the third time period corresponds to the end time of the second time period. In this step, after determining the state of the target object, the state of the person in subsequent time periods is further assessed to determine whether the person left the no-littering area empty-handed. Accordingly, the human state is analyzed based on the human recognition results corresponding to multiple consecutive video frames within the third time period. For example, after continuously analyzing Y frames to determine that the target object was thrown and remained in the no-littering area, the human recognition results corresponding to the subsequent Z frames are further analyzed to detect whether the person is holding an object. Simultaneously, the person's movement trajectory is combined to determine whether they left empty-handed, which serves as the basis for subsequent judgments on littering behavior.

[0025] S106. Determine the corresponding littering behavior detection results based on the spatiotemporal interaction relationship, object state results, and human state results.

[0026] In this step, the spatiotemporal interaction between the human body and objects, the state of the objects, and the state of the human body are considered to determine whether littering has actually occurred. When the sequence of actions—the person carrying a trash bag into a no-littering zone, the person throwing the trash, the trash remaining in the no-littering zone, and the person leaving empty-handed—is fully detected, littering is confirmed. Compared to related technologies that target only a single object, such as detecting only static trash or analyzing throwing actions, this step recognizes targets and actions based on continuous video frame images, comprehensively analyzing the relationships between people, objects, space, and time, resulting in higher detection accuracy.

[0027] This disclosure analyzes the spatiotemporal interaction between the human body and the target object, the state of the object, and the state of the human body sequentially according to time sequence, and comprehensively judges violations. Through the seamless detection of three time periods, it achieves full automation of the target recognition, dynamic association, and behavior judgment process. Targeted temporal analysis using spatiotemporal context information not only significantly reduces the risk of false positives and false negatives, but also has higher detection efficiency compared to manual inspection, and effectively reduces the false positive rate compared to simple target detection.

[0028] In one possible implementation, target recognition is performed on video frame images to determine human body recognition results and object recognition results, including: Feature extraction is performed on each video frame image to obtain the corresponding image features, which include texture features, color features, shape features and spatial context features; Targets are classified and located based on image features to determine the corresponding human body recognition results and object recognition results.

[0029] In this embodiment, a multimodal large model is preferred for target localization and classification. This model extracts image features from each frame of the image, performs target detection based on the extracted image features, and outputs the corresponding human bounding box coordinates, category labels and their confidence scores, as well as the bounding box coordinates, category labels and their confidence scores of the target object. The image features include texture features, color features, shape features and spatial context features. When identifying objects in the image, the texture features of the garbage bag are non-uniform thin film wrinkled textures, lacking features such as smooth paper textures and hard, regular textures. Based on this, the object in the image can be determined to be a garbage bag, rather than a delivery box or toy shell, avoiding misjudgment of non-garbage objects and providing a reliable data foundation for subsequent spatiotemporal interaction analysis.

[0030] In one possible implementation, determining the spatiotemporal interaction relationship between the human body and the object based on the human body recognition results and object recognition results of the first time period includes: Perform region recognition on video frame images to determine the region coordinate information of the target region; Determine the human coordinate information in the human body recognition result and the object coordinate information in the object recognition result, and determine whether the human body coordinate information and object coordinate information match the regional coordinate information within the first time period; In response to the fact that both human body coordinate information and object coordinate information are within the coordinate range of the region coordinate information, the coordinate distance between human body coordinate information and object coordinate information is determined. The spatial interaction between the human body and objects within the target area is determined based on the relationship between the coordinate spacing and the target spacing threshold, and the temporal interaction between the human body and objects within the target area is determined based on the duration when the coordinate spacing is less than the target spacing threshold.

[0031] In this embodiment, before determining the spatiotemporal interaction between the human body and the object, it is necessary to first determine the no-littering zone, i.e., the coordinate information of the target area in the video frame image. For example, the target area is a rectangular area inside the building entrance, with coordinates ranging from (100, 100, 300, 500). The bounding box coordinates of the human body and the target object in the corresponding video frame image are determined frame by frame to ascertain whether both the human body and the target object fall within the coordinate range of the target area. For example, in a video frame image during the time period 8:00:00-8:00:02, the bounding box coordinates of the human body are (110, 145, 215, 395), and the bounding box coordinates of the target object are (175, 375, 205, 415), both within the coordinate range of the target area, thus indicating the presence of both a human and a target object within the no-littering zone.

[0032] Furthermore, the coordinate distance between the human body and the object is determined based on their coordinate information. The center point of the human body's bounding box and the center point of the target object's (garbage bag's) bounding box are calculated for each video frame, and the corresponding Euclidean distance is determined. This distance is then compared with a preset target distance threshold to determine the spatial interaction relationship between the human body and the object within the target area. For example, if the target distance threshold is set to 50 pixels, and the coordinate distance between the center point of the human body's bounding box and the center point of the target object's bounding box is less than 50 pixels, it is determined that the human body and the object are spatially related.

[0033] Furthermore, provided the coordinate spacing is less than the target spacing threshold, the duration of this state is determined. For example, if the coordinate spacing of 10 consecutive frames is less than 50 pixels, the corresponding temporal interaction relationship is determined to be carrying the target object for 10 consecutive frames. Combining spatial and temporal interaction relationships, the final spatiotemporal interaction relationship can be determined to be a person carrying a trash bag continuously within a no-littering zone. In other words, this embodiment determines the association state of a person carrying an object by combining region limitation, coordinate matching, spacing threshold determination, and duration determination, effectively improving the accuracy of the determination results. Moreover, a multi-target tracking algorithm is used to establish the positional association between the person and the object, ensuring the consistency of the same person and object in consecutive frames. The target tracking algorithm is preferably the DeepSORT algorithm.

[0034] In one possible implementation, the object state is analyzed based on the object recognition results of the second time period to obtain the object state result, including: The object movement characteristics are determined based on the object coordinate information corresponding to the first target time interval in the second time period. The object movement characteristics include the object movement direction and the object movement speed. The object dwelling characteristics are determined based on the object coordinate information corresponding to the second target time interval in the second time period. The object dwelling characteristics represent the dwelling state of the object in the target area, and the time start of the second target time interval corresponds to the time end of the first target time interval. In response to the object's movement features satisfying the target movement condition and the object's dwelling features satisfying the target dwelling condition, the corresponding object state is determined to be a discarded state.

[0035] In this embodiment, the determination of the object's state in the second time period specifically includes two stages. First, the object's movement characteristics are determined based on the object coordinate information corresponding to the first target time interval in the second time period, i.e., whether the target object has been thrown. This is determined by analyzing the coordinate changes and movement speed of the garbage bag in the vertical and horizontal directions. For example, object coordinate information is extracted from three consecutive frames of images from 8:00:00 to 8:00:03: frame 1 (175, 375, 205, 415), frame 2 (178, 390, 208, 430), and frame 3 (180, 410, 210, 450). Based on this, the object's movement characteristics are determined to be vertically downward movement with a movement speed of approximately 11.7 pixels per frame. The object's movement trajectory satisfies the throwing characteristics, i.e., it meets the target movement condition.

[0036] Furthermore, the coordinates of the object corresponding to the second target time interval in the second time period are determined to ascertain whether the thrown object is in a lingering state, i.e., a discarded state. For example, if the coordinates of the garbage bag corresponding to 10 consecutive frames in the time period from 8:00:03 to 8:00:06 are fixed at (180, 410, 210, 450), with no significant positional change, then the target lingering condition is met, and the corresponding object state is determined to be a discarded state. This embodiment further distinguishes between discarded garbage and temporarily placed items by adding a step to determine the object's lingering state, effectively preventing false alarms. That is, by jointly judging the movement and lingering features, the practicality of detection is improved.

[0037] In one possible implementation, the human body state is analyzed based on the human body recognition results of the third time period to obtain the human body state results, including: The human movement trajectory and the state of the object being held by the human are determined based on the human recognition results during the third time period. In response to the human body's movement trajectory satisfying the condition of leaving the target area, and the human body holding an object satisfying the empty-handed state, the corresponding human body state result is determined to be the empty-handed leaving state.

[0038] In this embodiment, the human body status is analyzed based on the human body recognition results determined from multiple consecutive video frames within the third time period, specifically to determine whether a person has left the no-littering area empty-handed. Accordingly, the center point of the human body bounding box in each video frame is extracted to determine whether the human body's movement trajectory meets the condition of leaving the target area. For example, it is determined whether the coordinates of the center point of the human body bounding box in 10 consecutive frames from 8:00:06 to 8:00:08 exceed the coordinate range of the target area. If the coordinates of the center point of the human body bounding box in multiple consecutive frames all exceed the target area, then the human body's movement trajectory is determined to meet the condition of leaving the target area. Simultaneously, based on the coordinates of the human hand area located using target detection and pose estimation, it is detected whether the person is carrying a target object, thereby determining the person's carrying status. When the person's carrying status is empty-handed, the result is determined to be an empty-handed departure. In other words, this embodiment, through dual verification of movement trajectory and carrying status, can not only determine the occurrence of littering but also provide crucial evidence for subsequent identity tracing.

[0039] In one possible implementation, the corresponding littering behavior detection result is determined based on spatiotemporal interaction relationships, object state results, and human body state results, including: Determine whether there are any character-related events based on the spatiotemporal interaction relationship. Character-related events indicate that both the human body and the object are within the target area, and there is a relationship between the human body and the object. In response to an event associated with a person, determine the corresponding object state. In response to the object's state being discarded, determine the corresponding human body state. The system detects littering when the user leaves empty-handed.

[0040] In this embodiment, a person-object association event is determined based on the spatiotemporal interaction between the person and the object, meaning that both the person and the garbage bag are within the no-littering zone, and there is an association between them. Further, based on the verified object state result, it is determined whether the corresponding target object is in a discarded state, i.e., it is determined that the garbage bag was thrown and remained in the no-littering zone. In addition, this embodiment further determines whether the person leaves empty-handed after discarding the garbage bag. When both a person-object association event and the object state are discarded, and the person leaves empty-handed, a detection result of littering by the person is finally determined within the corresponding time period. Compared to related technologies that rely on a single condition, such as the garbage coordinates remaining fixed within a certain time period, this embodiment effectively reduces the occurrence of false positives, eliminating situations where a person temporarily placing garbage is detected as littering. Only when the person and the target object meet the complete sequence conditions is it determined to be a violation, significantly reducing the risk of false positives.

[0041] In one possible implementation, the method further includes: S107. Based on the detection results of littering behavior, backtrack the corresponding video frame images and filter out the target video frame images containing human faces. S108. Extract facial features from the target video frame image, compare the corresponding facial features with the pre-stored template facial features to determine the corresponding identity information, and send the corresponding alarm prompt based on the identity information.

[0042] In this embodiment, identity tracing and alarms are performed for littering behavior occurring during a identified target time period. Specifically, when determining the identity of the perpetrator, multiple video frames corresponding to the littering behavior are first detected by backtracking. Then, target video frames containing faces are selected from these violation video frames. It should be noted that the selection criteria for target video frames in this embodiment include a face size greater than a size threshold, a face angle meeting an angle threshold (frontal or side view), and a face occlusion area less than an area threshold. For example, for a video stream from 8:00:00 to 8:00:08 indicating littering behavior in a building, 10 frames from that time period are backtracked. Based on the face selection criteria, the image corresponding to 8:00:01 is selected, where the face meets the selection criteria: a face pixel size of 120×120, a side view angle of 25°, and an occlusion area less than 5% of the overall area, thus obtaining the target video frame image.

[0043] Furthermore, the facial recognition module is invoked to extract facial feature vectors from the target video frame image. These vectors are compared with pre-stored, owner-authorized template facial features to calculate cosine similarity. If the cosine similarity meets a set similarity threshold, the corresponding owner information is matched to determine the perpetrator's identity, and an alarm message is sent to the corresponding perpetrator. In this embodiment, the alarm message is a structured data result, including violation time information, violation location information, the corresponding violation video frame image, and identity information. It should be noted that if the cosine similarity does not meet the similarity threshold, or if the target video frame image cannot be filtered out, an anonymous alarm is executed. The anonymous alarm includes violation time, violation location information, and the corresponding violation video frame image, thereby ensuring comprehensive management. In addition, this embodiment automatically pushes the alarm message to property management personnel, enabling them to manage accordingly, forming a closed loop of real-time discovery, intelligent judgment, identity tracking, and precise management.

[0044] The following provides a complete embodiment to illustrate the technical solution of this application in detail.

[0045] Front-end equipment deployment: Install surveillance cameras (IPCs) inside the main entrances of each building in the community. The preferred resolution is 1080P or higher, with wide dynamic range (WDR) to adapt to backlighting environments, and low-light capability to ensure nighttime monitoring effectiveness. Network equipment deployment: Communication connection between the camera and the server; the server is used to receive the video stream captured by the camera. Backend server deployment: One or more GPU servers, depending on the size of the community and the number of concurrent paths; Model Deployment: This includes a multimodal large-scale model and a face recognition model. The multimodal large-scale model is used to identify littering behavior, while the face recognition model determines facial information based on the recognition results of the multimodal large-scale model. When training the multimodal large-scale model, a visual language model is preferred as the foundation, such as BLIP-2 or LLaVA models. Fine-tuning is performed using a labeled dataset of littering behavior. The training data includes multimodal littering actions, video clips under different lighting conditions and shooting angles, and annotations including target bounding boxes, behavior labels (e.g., carrying, throwing, loitering), and sequence logical labels (e.g., violation or non-violation), to train the model to learn littering behavior detection. The preferred face recognition models are ArcFace and FaceNet.

[0046] Business system deployment includes a video stream processing service module, an analysis engine, a management system backend, and a mobile app. The video stream processing service module receives video streams from cameras and extracts images frame by frame, sending them to the analysis engine. The analysis engine encapsulates multimodal large models and face recognition models, providing API interfaces. The management system backend is used by property management personnel to receive alarms, view historical records, and manage the resident face database. The mobile app is used by property management personnel for mobile office work and to receive push notifications in real time.

[0047] Face database deployment: Under the premise of legality and compliance, collect and establish an initial community resident face feature database to provide template face features.

[0048] When identifying littering behavior, such as Figure 2 As shown, the analysis engine receives real-time video streams collected by the front-end device and extracts them into multiple continuous video frames. Based on these frames, it executes the littering detection method described above to determine the violation and identify the target facial features. These features are then compared with template facial features in the facial recognition database to determine the corresponding identity information. An alarm is then generated and pushed to the corresponding individual. Simultaneously, the alarm information is sent to the back-end management system for property management personnel. For example, Figure 3 As shown, if the identity information cannot be identified based on the target's facial features, an anonymous alarm message is generated and synchronized to the property management system.

[0049] This disclosure continuously samples and analyzes video streams in real time, sequentially analyzing the spatiotemporal interaction between the human body and target objects, the state of the object, and the state of the human body to comprehensively determine violations. Through seamless detection across three time periods, it automates the entire process of target recognition, dynamic correlation, and behavior determination. Subsequent facial recognition processes are only triggered when a violation is confirmed, avoiding situations where any object in a no-littering zone triggers an alarm. Furthermore, for cases not entered into the facial database, such as when the perpetrator is a visitor, anonymous alarms are set to ensure comprehensive management. Structured information is pushed to the backend for management purposes. Compared to related technologies that only detect single actions, this method can identify genuine littering behavior in complex scenarios, resulting in more accurate detection results.

[0050] To implement the above methods, one example of this application also provides a littering behavior detection device, used to implement any of the above-mentioned littering behavior detection methods, such as... Figure 5 As shown, the device 500 includes: The image acquisition module 501 is used to acquire the monitoring video stream and extract the corresponding video frame images frame by frame; The target recognition module 502 is used to perform target recognition on video frame images and determine the human body recognition result and the object recognition result; The spatiotemporal relationship determination module 503 is used to determine the spatiotemporal interaction relationship between the human body and the object based on the human body recognition results and object recognition results of the first time period. The object state determination module 504 is used to analyze the object state based on the object recognition results of the second time period and obtain the object state result. The start time of the second time period corresponds to the end time of the first time period. The human body state determination module 505 is used to analyze the human body state based on the human body recognition results of the third time period and obtain the human body state result. The start time of the third time period corresponds to the end time of the second time period. The result determination module 506 is used to determine the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, object state result, and human state result.

[0051] In one embodiment, the target recognition module 502 is further configured to extract features from each video frame image to obtain corresponding image features, including texture features, color features, shape features and spatial context features; Based on image features, target classification and localization are performed to determine the corresponding human body recognition results and target object recognition results.

[0052] In one embodiment, the spatiotemporal relationship determination module 503 is further used to perform region recognition on the video frame image and determine the region coordinate information of the target region; Determine the human coordinate information in the human body recognition result and the object coordinate information in the target object recognition result, and determine whether the human body coordinate information and object coordinate information match the regional coordinate information within the first time period; In response to the fact that both human body coordinate information and object coordinate information are within the coordinate range of the region coordinate information, the coordinate distance between human body coordinate information and object coordinate information is determined. The spatial interaction between the human body and objects within the target area is determined based on the relationship between the coordinate spacing and the target spacing threshold, and the temporal interaction between the human body and objects within the target area is determined based on the duration when the coordinate spacing is less than the target spacing threshold.

[0053] In one embodiment, the object state determination module 504 is further configured to determine the object movement characteristics based on the object coordinate information corresponding to the first target time interval in the second time period. The object movement characteristics include the object movement direction and the object movement speed. The object dwelling characteristics are determined based on the object coordinate information corresponding to the second target time interval in the second time period. The object dwelling characteristics represent the dwelling state of the object in the target area, and the time start of the second target time interval corresponds to the time end of the first target time interval. In response to the object's movement features satisfying the target movement condition and the object's dwelling features satisfying the target dwelling condition, the corresponding object state is determined to be a discarded state.

[0054] In one embodiment, the human body state determination module 505 is further configured to determine the human body movement trajectory and the human body holding object state based on the human body recognition results in the third time period. In response to the human body's movement trajectory satisfying the condition of leaving the target area, and the human body holding an object satisfying the empty-handed state, the corresponding human body state result is determined to be the empty-handed leaving state.

[0055] In one possible implementation, the result determination module 506 is further configured to determine whether there is a person-related event based on the spatiotemporal interaction relationship. The person-related event indicates that both the human body and the object are within the target area, and there is a relationship between the human body and the object. In response to an event associated with a person, determine the corresponding object state. In response to the object's state being discarded, determine the corresponding human body state. The system detects littering when the user leaves empty-handed.

[0056] In one embodiment, it also includes a face recognition module 507, which is used to backtrack the corresponding video frame image based on the littering behavior detection result and filter out the target video frame image containing a face; Facial features are extracted from the target video frame image, and the corresponding facial features are compared with the pre-stored template facial features to determine the corresponding identity information, so as to send the corresponding alarm prompt based on the identity information.

[0057] By way of example, this application also provides an electronic device, including: Memory is used to store processor-executable instructions; The processor is used to read executable instructions from memory and execute the instructions to implement the littering behavior detection method described above.

[0058] For example, this application also provides a computer-readable storage medium storing a computer program for performing the above-described littering behavior detection method.

[0059] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0060] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0061] Multiple components in device 1000 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0062] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as a littering behavior detection method. For example, in some embodiments, a littering behavior detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of a littering behavior detection method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform a littering behavior detection method by any other suitable means (e.g., by means of firmware).

[0063] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0064] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0065] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0066] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0067] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0068] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0069] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0071] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for detecting littering behavior, characterized in that, The method includes: Acquire the surveillance video stream and extract the corresponding video frame images frame by frame; Target recognition is performed on the video frame images to determine the human body recognition result and the target object recognition result; The spatiotemporal interaction relationship between the human body and the target object is determined based on the human body recognition results and the target object recognition results in the first time period. The object state is analyzed based on the target object recognition results of the second time period to obtain the object state result. The start time of the second time period corresponds to the end time of the first time period. The human body status is analyzed based on the human body recognition results of the third time period to obtain the human body status result. The start time of the third time period corresponds to the end time of the second time period. Based on the spatiotemporal interaction relationship, the object state result, and the human body state result, the corresponding littering behavior detection result is determined.

2. The method for detecting littering behavior according to claim 1, characterized in that, The step of performing target recognition on the video frame image to determine the human body recognition result and the target object recognition result includes: Feature extraction is performed on each video frame image to obtain corresponding image features, which include texture features, color features, shape features and spatial context features; Based on the image features, target classification and localization are performed to determine the corresponding human body recognition results and target object recognition results.

3. The method for detecting littering behavior according to claim 1, characterized in that, Determining the spatiotemporal interaction relationship between the human body and the object based on the human body recognition result and the target object recognition result in the first time period includes: Perform region recognition on the video frame image to determine the region coordinate information of the target region; Determine the human coordinate information in the human body recognition result and the object coordinate information in the target object recognition result, and determine whether the human body coordinate information and the object coordinate information match the area coordinate information within the first time period; In response to the fact that both the human body coordinate information and the object coordinate information are within the coordinate range of the region coordinate information, the coordinate spacing between the human body coordinate information and the object coordinate information is determined; The spatial interaction relationship between the human body and the object within the target area is determined based on the relationship between the coordinate spacing and the target spacing threshold, and the temporal interaction relationship between the human body and the object within the target area is determined based on the duration when the coordinate spacing is less than the target spacing threshold.

4. The method for detecting littering behavior according to claim 3, characterized in that, The step of analyzing the object state based on the target object recognition result in the second time period to obtain the object state result includes: The object movement characteristics are determined based on the object coordinate information corresponding to the first target time interval in the second time period. The object movement characteristics include the object movement direction and the object movement speed. The object dwelling characteristics are determined based on the object coordinate information corresponding to the second target time interval in the second time period. The object dwelling characteristics represent the dwelling state of the object in the target area, and the time start point of the second target time interval corresponds to the time end point of the first target time interval. In response to the object's movement feature satisfying the target movement condition and the object's dwelling feature satisfying the target dwelling condition, the corresponding object state result is determined to be a discarded state.

5. The method for detecting littering behavior according to claim 3, characterized in that, The step of analyzing the human body state based on the human body recognition results in the third time period to obtain the human body state results includes: The human movement trajectory and the state of the object being held by the human are determined based on the human recognition results during the third time period. In response to the human body's movement trajectory satisfying the condition of leaving the target area, and the human body holding an object satisfying the empty-handed state, the corresponding human body state result is determined to be the empty-handed leaving state.

6. The method for detecting littering behavior according to claim 3, characterized in that, The determination of the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, the object state result, and the human body state result includes: Based on the spatiotemporal interaction relationship, it is determined whether there is a person-related event. The person-related event indicates that both the human body and the object are within the target area, and there is a relationship between the human body and the object. In response to the existence of the associated event of the person, the corresponding object state result is determined; In response to the object's state being discarded, the corresponding human body state is determined. In response to the human body leaving empty-handed, it is determined that littering has occurred.

7. The method for detecting littering behavior according to any one of claims 1-6, characterized in that, The method further includes: Based on the littering behavior detection results, the corresponding video frame images are traced back, and target video frame images containing human faces are selected. Facial features are extracted from the target video frame image, and the corresponding facial features are compared with pre-stored template facial features to determine the corresponding identity information, so as to send the corresponding alarm prompt based on the identity information.

8. A littering behavior detection device, used to implement the littering behavior detection method according to any one of claims 1-7, characterized in that, The device includes: The image acquisition module is used to acquire the monitoring video stream and extract the corresponding video frame images frame by frame; The target recognition module is used to perform target recognition on the video frame images and determine the human body recognition result and the target object recognition result; The spatiotemporal relationship determination module is used to determine the spatiotemporal interaction relationship between the human body and the target object based on the human body recognition result and the target object recognition result in the first time period. The object state determination module is used to analyze the object state based on the target object recognition result in the second time period to obtain the object state result. The start time of the second time period corresponds to the end time of the first time period. The human body state determination module is used to analyze the human body state based on the human body recognition results of the third time period to obtain the human body state result. The start time of the third time period corresponds to the end time of the second time period. The result determination module is used to determine the corresponding littering behavior detection result based on the spatiotemporal interaction relationship, the object state result, and the human body state result.

9. An electronic device, characterized in that, include: At least one processor; The at least one processor is connected to a memory; wherein the memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the littering behavior detection method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the littering behavior detection method according to any one of claims 1-7.

Images