A video-based violent sorting behavior recognition method
By detecting and correlating the trajectory of flying packages in sorting monitoring videos, the system identifies personnel who engage in violent sorting, solving the problem of insufficient accuracy in complex scenarios and achieving efficient identification of violent sorting behavior.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN SHENGSHI TECH CO LTD
- Filing Date
- 2022-12-12
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies cannot accurately identify violent sorting behavior in complex scenarios, resulting in package damage. Existing algorithms have insufficient accuracy in scenarios where multiple people are sorting simultaneously, flight trajectories are obscured, and flight angles are random.
By acquiring package sorting monitoring videos, packages in flight are detected and matched with previous flight trajectories to fit a complete flight trajectory, determining the start and end frames. Based on the start frame, personnel who engage in violent sorting are detected. Target detection models and Kalman filtering algorithms are used to improve recognition accuracy.
It can accurately identify personnel who engage in violent sorting in complex scenarios, improve the accuracy of identification, reduce over-detection, under-detection and false detection, and adapt to multi-angle monitoring scenarios.
Smart Images
Figure CN116189031B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image recognition technology, and more specifically, relates to a video-based method for recognizing violent sorting behavior. Background Technology
[0002] With the surge in logistics volume, employees often resort to rough handling when sorting massive amounts of packages, resulting in damage to parcels and deliveries. For example, the most common form of rough handling is throwing, which causes the most damage to packages and deliveries.
[0003] In existing technologies, image recognition algorithms are used to assist in identifying violent sorting behavior. However, since most express sorting sites are very complex, existing violent sorting behavior identification algorithms can only adapt to simple scenarios and cannot adapt to complex real-world scenarios such as multiple people sorting at the same time, obstructed flight trajectories, and random flight angles relative to the monitoring perspective. In complex real-world scenarios, the tracking of flying packages will be interrupted or disordered, and it will be impossible to accurately identify the violent sorting personnel. Summary of the Invention
[0004] The purpose of this application is to provide a video-based method for recognizing violent sorting behavior, so as to solve the technical problem of insufficient accuracy in the existing technology for recognizing violent sorting behavior.
[0005] To achieve the above objectives, the technical solution adopted in this application is: to provide a video-based method for recognizing violent sorting behavior, comprising:
[0006] Obtain package sorting surveillance video;
[0007] Detect packages that are in flight in surveillance video;
[0008] Each package in flight is associated with and matched against its prior flight trajectory;
[0009] If the association matching is successful, the existing flight trajectory is added to fit a complete flight trajectory. If there are N consecutive frames where the association matching is unsuccessful, a new flight trajectory is created.
[0010] Determine the start and end frames of the flight trajectory;
[0011] Violent sorting personnel detected based on the starting frame of the flight trajectory.
[0012] Preferably, the method for associating and matching each package in flight with a prior flight trajectory includes the following steps:
[0013] Predict the future trajectories of all prior flight paths;
[0014] Obtain the target detection bounding box and the corresponding confidence score of the package that is in flight in the current frame;
[0015] Boxes with confidence scores exceeding the threshold τ are assigned to Dhigh, and boxes with confidence scores below the threshold τ are assigned to Dlow.
[0016] First, associate the target detection bounding box in Dhigh with all prior flight trajectories;
[0017] Then, the target detection bounding box in Dlow is associated with the remaining prior flight trajectory a second time.
[0018] Preferably, the method for associating and matching each package in flight with a prior flight trajectory includes the following steps:
[0019] Predict the future trajectories of all prior flight paths;
[0020] Obtain the target detection bounding box and the corresponding confidence score of the package that is in flight in the current frame;
[0021] The target detection boxes are sorted according to their confidence scores.
[0022] The target detection boxes are associated with the sorting and all prior flight trajectories.
[0023] Preferably, after associating the target detection boxes according to the sorting and all prior flight trajectories, the method further includes the following steps:
[0024] If a match is successfully found, the IoU threshold for the match is lowered.
[0025] If the association matching fails, the target detection box will be determined as the background wrapping the target.
[0026] Preferably, if the association matching is successful, the method of adding the prior flight trajectory and fitting a complete flight trajectory includes the following steps:
[0027] Find video frames P between the last frame and the current frame of any previously matched flight path;
[0028] Retrieve the background of the target within video frame P;
[0029] The flight trajectory T is formed based on the background package target;
[0030] The flight trajectory T is added to the previously matched flight trajectories to fit a complete flight trajectory.
[0031] Preferably, the method for detecting violent sorting personnel based on the starting frame of the flight trajectory includes:
[0032] Detect all human targets in the starting frame of the flight trajectory;
[0033] Detect targets that are making throwing motions from all human targets.
[0034] Preferably, when detecting a target making a throwing motion from all human targets, if no target making a throwing motion is detected, then all human targets in the neighborhood of the starting frame of the flight trajectory are detected, and the target making a throwing motion is detected from all human targets.
[0035] Preferably, after detecting a target making a throwing motion from all human targets, the process includes the following steps:
[0036] Calculate the distance between the center point of the target making the throwing motion and the distance to the package, and select the target that makes the throwing motion and meets the distance threshold as the violent sorting personnel; if there are multiple targets that make the throwing motion and meet the distance threshold, select the target with the shortest distance as the violent sorting personnel.
[0037] Preferably, after detecting the violent sorting personnel based on the starting frame of the flight trajectory, the process further includes the following steps:
[0038] Detect the flight speed, flight time, and whether the package flips in each flight path;
[0039] Output the violation level of the violent sorting action.
[0040] Preferably, the method for detecting whether a package in each flight path has been overturned includes the following steps:
[0041] Obtain the target detection bounding boxes of the corresponding packages in the starting frame, midpoint frame, and ending frame of the flight trajectory, respectively;
[0042] Perform edge detection on the bounding boxes of the target detection boxes corresponding to the start frame, midpoint frame, and end frame respectively, and output the edge trajectory;
[0043] Determine the structural similarity between the edge trajectories corresponding to the starting frame, midpoint frame, and ending frame.
[0044] The video-based violent sorting behavior recognition method provided in this application, compared with the prior art, associates and matches each package in flight with the prior flight trajectory based on the video stream, fits the multi-segment flight trajectory into a complete flight trajectory, determines the start frame and end frame of the flight trajectory, and accurately identifies the violent sorting personnel based on the start frame of the flight trajectory. It can adapt to the recognition of monitoring behavior from multiple angles and solves the technical problem of insufficient accuracy in complex scenarios. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 A flowchart illustrating the video-based violent sorting behavior recognition method provided in this application embodiment;
[0047] Figure 2 This is a schematic diagram of a complex parcel sorting monitoring video scenario provided in an embodiment of this application;
[0048] Figure 3 for Figure 2 The image shows a scene illustrating the flight process of a package located in the starting frame, midpoint frame, and ending frame. Detailed Implementation
[0049] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0050] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.
[0051] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0052] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0053] Please refer to the following: Figures 1 to 3 The video-based method for recognizing violent sorting behavior provided in this application will now be described. The video-based method for recognizing violent sorting behavior includes:
[0054] Step S1: Obtain the package sorting monitoring video;
[0055] Step S2: Detect packages that are in flight in the surveillance video;
[0056] Step S3: Associate and match each package in flight with its previous flight trajectory;
[0057] Step S4: If the association matching is successful, add the existing flight trajectory and fit it into a complete flight trajectory. If there are N consecutive frames where the association matching is unsuccessful, create a new flight trajectory.
[0058] Step S5: Determine the start frame and end frame of the flight trajectory;
[0059] Step S6: Detect the violent sorting personnel based on the starting frame of the flight trajectory.
[0060] It is understandable that in step S1, based on the package sorting monitoring video, it can be shown that this application is based on video stream recognition, which is different from the single-frame image recognition and detection of the prior art.
[0061] In step S2, since there are usually many package targets in the monitoring scene and interference has not been eliminated, packages in flight state in the monitoring video can be detected first. This can distinguish between packages in a stationary state and packages in flight state in the monitoring scene, thus reducing the complexity.
[0062] One method for detecting packages in flight in surveillance videos can be based on a pre-trained object detection model, using YOLOX to detect express delivery packages in flight. Training this object detection model requires a training dataset that labels various types of packages in flight and various types of packages in stationary states.
[0063] In step S3, each package in flight is associated with and matched against a prior flight trajectory, which can be understood as a flight trajectory created some time prior to the current video frame. Thus, even if a package in flight flips, deforms, or is obstructed by foreign objects, causing trajectory tracking to be interrupted, the prior flight trajectory can still be retrieved through association matching to fit a complete flight trajectory. Furthermore, it can adapt to different monitoring perspectives, including situations where the monitoring direction is parallel to the plane containing the flight trajectory.
[0064] In step S4, if the association matching is successful, it means that the package in flight in the current frame is the same target as the packages in flight in the previous frames. Then, the previous flight trajectory is added to fit a complete flight trajectory. If the association matching is unsuccessful, it means that the package in flight in the current frame is not the same target as the packages in flight in the previous frames. Generally, it will be temporarily treated as background. If there are N consecutive frames where the association matching is unsuccessful, that is, the first frame in the N consecutive frames where the package in flight is the starting frame of the flight, a new flight trajectory is created. At the same time, the starting point of the first frame in the N consecutive frames where the package in flight is the starting point of the flight trajectory.
[0065] Thus, by repeating the above steps, a flight trajectory can be formed for each package in flight. Preferably, if a previous flight trajectory fails to be matched for x consecutive frames, the previous flight trajectory is considered terminated.
[0066] In step S5, the video frame where the starting point of the previous flight trajectory is located is taken as the starting frame of the flight trajectory, and the video frame where the ending point of the previous flight trajectory is located is taken as the ending frame of the flight trajectory.
[0067] In step S6, the violent sorting personnel detected based on the starting frame of the flight trajectory are narrowed down, thereby accurately identifying the violent sorting personnel and solving the problems of multiple detections, missed detections, and false detections caused by the complexity of the scene.
[0068] The video-based violent sorting behavior recognition method provided in this application, compared with the prior art, associates and matches each package in flight with the prior flight trajectory based on the video stream, fits the multi-segment flight trajectory into a complete flight trajectory, determines the start frame and end frame of the flight trajectory, and accurately identifies the violent sorting personnel based on the start frame of the flight trajectory. It can adapt to the recognition of monitoring behavior from multiple angles and solves the technical problem of insufficient accuracy in complex scenarios.
[0069] In another embodiment of this application, step S3, the method of associating and matching each package in flight with a prior flight trajectory, includes the following steps:
[0070] Predict the future trajectories of all prior flight paths;
[0071] Obtain the target detection bounding box and the corresponding confidence score of the package that is in flight in the current frame;
[0072] Boxes with confidence scores exceeding the threshold τ are assigned to Dhigh, and boxes with confidence scores below the threshold τ are assigned to Dlow.
[0073] First, associate the target detection bounding box in Dhigh with all prior flight trajectories;
[0074] Then, the target detection bounding box in Dlow is associated with the remaining prior flight trajectory a second time.
[0075] Understandably, in complex scenarios with multiple prior flight trajectories and multiple target detection boxes, the target boxes are categorized into Dhigh and Dlow based on confidence scores and a threshold τ, and then associated twice. Since the confidence score in Dhigh is higher than the threshold τ, it still has a high association accuracy even when associated with all prior flight trajectories. After the first association, the number of prior flight trajectories to be associated is reduced. Therefore, associating the target detection boxes in Dlow with the remaining prior flight trajectories a second time can improve the success rate of the association matching of target detection boxes in Dlow.
[0076] It is worth noting that the prediction of future trajectories can be based on existing Kalman filtering algorithms. Kalman filtering algorithms can be used to predict the future multi-frame trajectories of the previous flight trajectory. The IoU between the predicted bounding box and the actual target detection box is used as the similarity between the two matches, for example, by using the Hungarian algorithm to complete the matching.
[0077] In another embodiment of this application, step S3, the method of associating and matching each package in flight with a prior flight trajectory, includes the following steps:
[0078] Predict the future trajectories of all prior flight paths;
[0079] Obtain the target detection bounding box and the corresponding confidence score of the package that is in flight in the current frame;
[0080] The target detection boxes are sorted according to their confidence scores.
[0081] The target detection boxes are associated with the sorting and all prior flight trajectories.
[0082] It is understandable that sorting the target detection boxes according to their confidence scores allows for priority association of target detection boxes with higher confidence scores. After association, the previously associated flight paths are then removed to improve the accuracy of target detection boxes with lower confidence scores.
[0083] Furthermore, after associating the target detection boxes with all prior flight trajectories based on the sorting, the process also includes the following steps:
[0084] If a match is successfully found, the IoU threshold for the match is lowered.
[0085] If the association matching fails, the target detection box will be determined as the background wrapping the target.
[0086] Understandably, based on the method of this embodiment, on the one hand, the similarity IoU threshold can be preset to a relatively high value. This is because target detection boxes with higher confidence scores are more likely to reach the similarity IoU threshold during association matching. In this way, background packages can be filtered out as packages in flight, improving the robustness of recognition and detection. On the other hand, since the accuracy of target detection boxes with lower confidence scores can be ensured after priority association of target detection boxes with higher confidence scores, by lowering the similarity IoU threshold for association matching, the success rate of association matching for target detection boxes with lower confidence scores can be further improved. In this way, the probability of incorrectly identifying packages in flight as background packages can be reduced.
[0087] Furthermore, in step S4, if the association matching is successful, the method of adding the prior flight trajectory and fitting a complete flight trajectory includes the following steps:
[0088] Find video frames P between the last frame and the current frame of any previously matched flight path;
[0089] Retrieve the background of the target within video frame P;
[0090] The flight trajectory T is formed based on the background package target;
[0091] The flight trajectory T is added to the previously matched flight trajectories to fit a complete flight trajectory.
[0092] Understandably, in the case of discontinuous flight trajectories, if the current frame is successfully matched, then the video frame P between the last frame of the previously matched flight trajectory and the current frame is searched. Video frame P is the video frame where the flight trajectory is discontinuous. Video frame P may be 1 frame or multiple frames. Of course, if video frame P is not found, it means that there is no discontinuous or missing trajectory between the last frame of the previously matched flight trajectory and the current frame.
[0093] The background package target in video frame P is retrieved; that is, the target was identified as a background package target after the association matching failed. The successful matching in the current frame confirms that there are discontinuities in the flight trajectory. This means that identifying a package in flight as a background package during this discontinuity is erroneous. Misjudgments are usually caused by factors such as the angle between the flight trajectory and the surveillance camera, partial occlusion of the target, and background elements, leading to low confidence levels. Therefore, a flight trajectory T is formed based on the background package target. Flight trajectory T is then added back to the previously matched flight trajectory to fit a complete flight trajectory, further improving the robustness of the recognition and detection.
[0094] Thus, the complete flight trajectory obtained by fitting includes not only the common parabolic trajectory, but also straight trajectory, M-shaped trajectory, etc. Among them, the M-shaped trajectory is generally the case where a collision occurs and the trajectory changes drastically.
[0095] In another embodiment of this application, the method for detecting violent sorting personnel based on the starting frame of the flight trajectory in step S6 includes:
[0096] Detect all human targets in the starting frame of the flight trajectory;
[0097] Detect targets that are making throwing motions from all human targets.
[0098] Understandably, in complex environments, there may be multiple targets in a single frame. If a target making a throwing motion is detected from all human targets and the time of its throwing motion matches the starting time of the flight trajectory, it can be determined that the target making the throwing motion is the violent sorting personnel.
[0099] Furthermore, when detecting a target making a throwing motion from all human targets, if no target making a throwing motion is detected, then all human targets in the neighborhood of the starting frame of the flight trajectory are detected, and the target making a throwing motion is detected from all human targets.
[0100] It is understood that the time period of the starting frame neighborhood of the flight trajectory can be understood as the time period between several frames before the starting frame of the flight trajectory and several frames after the starting frame of the flight trajectory. That is, the time period is expanded to detect the target that makes the throwing action, and ultimately the violent sorting personnel corresponding to the flight trajectory are identified.
[0101] Furthermore, after detecting targets making throwing motions from all human targets, the process includes the following steps:
[0102] Calculate the distance between the center point of the target making the throwing motion and the distance to the package, and select the target that makes the throwing motion and meets the distance threshold as the violent sorting personnel; if there are multiple targets that make the throwing motion and meet the distance threshold, select the target with the shortest distance as the violent sorting personnel.
[0103] Understandably, filtering out people who are farther away by judging the distance between the target's center point and the package helps reduce false identification. Furthermore, if multiple targets that meet the distance threshold and are making throwing motions appear in the same frame, selecting the target with the shortest distance as the person violently sorting the package further eliminates interfering targets.
[0104] In another embodiment of this application, after detecting the violent sorting personnel based on the starting frame of the flight trajectory, the method further includes the following steps:
[0105] Detect the flight speed, flight time, and whether the package flips in each flight path;
[0106] Output the violation level of the violent sorting action.
[0107] Understandably, the faster the package flies, the longer the flight time, and the more likely it is to be damaged in the air. Therefore, by detecting the package's flight speed, flight time, and whether it is flipped, a rating can be implemented for violent sorting actions.
[0108] Furthermore, the method for detecting whether packages in each flight path have been overturned includes the following steps:
[0109] Obtain the target detection bounding boxes of the corresponding packages in the starting frame, midpoint frame, and ending frame of the flight trajectory, respectively;
[0110] Perform edge detection on the bounding boxes of the target detection boxes corresponding to the start frame, midpoint frame, and end frame respectively, and output the edge trajectory;
[0111] Determine the structural similarity between the edge trajectories corresponding to the starting frame, midpoint frame, and ending frame.
[0112] It is understandable that structural similarity can be calculated using the NCC algorithm in OpenCV, which includes a scaling process to avoid interference from inconsistent target sizes due to distance; combined with edge detection to obtain edge trajectories, it can further avoid interference from lighting, sharpness, background, etc.
[0113] The midpoint frame can be understood as the midpoint video frame located in the video frame sequence containing the start frame and the end frame, that is, the video frame located at the midpoint of the time corresponding to the start frame and the end frame.
[0114] If the structural similarity between the edge trajectories corresponding to the starting frame, midpoint frame, and ending frame is high, it indicates that the package did not flip during the entire flight. If the structural similarity between the edge trajectories corresponding to the starting frame, midpoint frame, and ending frame is low, it indicates that the package flipped during the entire flight.
[0115] Finally, after detecting the personnel involved in the violent sorting and rating the violent sorting actions, the continuous frames of the package's trajectory, together with the previously cached 10 frames of video, are combined to form a continuous video record. This record is then reported to the management platform as a suspected violent sorting incident so that the responsible parties can be held accountable for the damage to the package caused by the violent sorting behavior.
[0116] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A video-based method for recognizing violent sorting behavior, characterized in that, include: Obtain package sorting surveillance video; Detect packages that are in flight in surveillance video; Associating and matching each package in flight with its prior flight trajectory, including the following steps: Predict the future trajectories of all prior flight paths; obtain the target detection boxes and corresponding confidence scores of packages in flight in the current frame; sort the target detection boxes according to their confidence scores. The target detection boxes are associated with the sorting and all prior flight trajectories; If the association match is successful, the similarity IoU threshold of the association match is lowered; if the association match is unsuccessful, the target detection box is determined to be a background-wrapped target. If the association matching is successful, the previous flight trajectory is added to fit a complete flight trajectory. The steps include: finding video frame P between the last frame and the current frame of the previous successfully matched flight trajectory; retrieving the background of video frame P that wraps around the target; forming a flight trajectory T based on the background wrapping around the target; and adding the flight trajectory T to the previous successfully matched flight trajectory to fit a complete flight trajectory. If N consecutive frames fail to match, a new flight path is created. Determine the start and end frames of the flight trajectory; Violent sorting personnel detected based on the starting frame of the flight trajectory.
2. The video-based violent sorting behavior recognition method as described in claim 1, characterized in that, Methods for detecting violent sorting personnel based on the starting frame of the flight trajectory include: Detect all human targets in the starting frame of the flight trajectory; Detect targets that are making throwing motions from all human targets.
3. The video-based violent sorting behavior recognition method as described in claim 2, characterized in that, When detecting a target making a throwing motion from all human targets, if no target making a throwing motion is detected, then all human targets in the neighborhood of the starting frame of the flight trajectory are detected, and the target making a throwing motion is detected from all human targets.
4. The video-based violent sorting behavior recognition method as described in claim 1, characterized in that, After detecting targets making throwing motions from all human targets, the steps include: Calculate the distance between the center point of the target making the throwing motion and the distance to the package, and select the target that makes the throwing motion and meets the distance threshold as the violent sorting personnel; if there are multiple targets that make the throwing motion and meet the distance threshold, select the target with the shortest distance as the violent sorting personnel.
5. The video-based violent sorting behavior recognition method as described in any one of claims 1 to 4, characterized in that, After detecting the violent sorting personnel based on the starting frame of the flight trajectory, the following steps are also included: Detect the flight speed, flight time, and whether the package flips in each flight path; Output the violation level of the violent sorting action.
6. The video-based violent sorting behavior recognition method as described in claim 5, characterized in that, The method for detecting whether packages in each flight path have been overturned includes the following steps: Obtain the target detection bounding boxes of the corresponding packages in the starting frame, midpoint frame, and ending frame of the flight trajectory, respectively; Perform edge detection on the bounding boxes of the target detection boxes corresponding to the start frame, midpoint frame, and end frame respectively, and output the edge trajectory; Determine the structural similarity between the edge trajectories corresponding to the starting frame, midpoint frame, and ending frame.