Identifying robust vision-based systems and methods for handling rough practices in distribution centers

Abstract

A method and system for detecting rough handling. The system has a camera and a computing device. The computing device is configured to: receive a video stream captured by the camera; detect people and packages from the video frames of the video stream; construct a person trajectory and a package trajectory based on the detected people and packages; identify an action between a person trajectory and a package trajectory, wherein the action includes the person picking up, holding, and putting down the package; and, in a predetermined frame after putting down the package, determine the existence of rough handling when the movement distance of the package is greater than a threshold distance and the movement speed of the package is greater than a threshold speed.

Description

[0001] Cross-references

[0002] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citation and / or discussion of such references are provided solely to clarify the description of this disclosure and do not imply that any such reference is “prior art” as disclosed herein. All references cited and discussed in the specification are incorporated herein by reference in their entirety, to the same extent that each reference is individually incorporated by reference. Technical Field

[0003] This disclosure relates generally to image processing, and more specifically to systems and methods for identifying crude processing in distribution centers using robust vision schemes. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. Within the scope described in this background section, the work of the person making the disclosure, and various aspects of the specification that may not conform to prior art at the time of application, are neither explicitly nor implicitly acknowledged as prior art in this disclosure.

[0005] A distribution center is a warehouse used to receive, store, and redistribute large volumes of goods to other destinations. These centers typically employ workers to handle incoming and outgoing goods, often requiring manual unloading and parcel handling, especially near palletized or conveyor belt locations. Rough handling is a problem that arises when distribution center workers handle goods roughly. Workers may throw parcels onto conveyor belts, slam them to the ground, or push them into areas unlikely to impact other goods. This rough handling can damage goods and cause losses to the distribution center or the parcel owner. In response, some distribution centers hire staff to monitor the center's handling workers. However, it is difficult for staff to monitor multiple workers simultaneously, and this incurs significant labor costs for large centers.

[0006] Therefore, there is a need in this field to address the aforementioned defects and shortcomings. Summary of the Invention

[0007] In some aspects, this disclosure relates to a system for detecting rough handling, including an imaging device and a computing device, the computing device including a processor and a storage device storing computer-executable code. The computer-executable code, when executed at the processor, is configured to:

[0008] Receive a video stream comprising multiple video frames captured by the imaging device;

[0009] Multiple character bounding boxes and multiple wrapping bounding boxes are obtained from the multiple video frames;

[0010] Based on the obtained character bounding box and the package bounding box, construct multiple character trajectories and multiple package trajectories;

[0011] Identify an action between one of the multiple person trajectories and one of the multiple package trajectories, wherein the action includes a corresponding person in one of the multiple person trajectories picking up, holding, and putting down a corresponding package in one of the multiple package trajectories; and

[0012] In a predetermined frame after the corresponding package is placed, if the moving distance of the corresponding package is greater than a threshold distance and the moving speed of the corresponding package is greater than a threshold speed, the existence of brute force processing is determined.

[0013] In some embodiments, the computer-executable code is configured to obtain the plurality of person bounding boxes and the plurality of wrapping bounding boxes for each of the plurality of video frames by:

[0014] Detect people from each of the video frames to obtain the plurality of person bounding boxes;

[0015] Detect packages from each video frame to obtain the bounding boxes of the detected packages; and

[0016] Delete the stationary package bounding boxes from the detected package bounding boxes to obtain the plurality of package bounding boxes.

[0017] In some embodiments, the computer-executable code is further configured to:

[0018] Calculate the frame difference between each video frame and the previous frame of each video frame;

[0019] The region with a high frame difference in each video frame is defined as the foreground region; and

[0020] The detected bounding box that does not overlap with the foreground region is defined as the static bounding box.

[0021] In some embodiments, the computer-executable code is further configured to add an additional wrapping bounding box to the obtained wrapping bounding box, wherein the additional wrapping bounding box is located in a foreground region of the foreground region when there is no overlap between a foreground region of the foreground region and the detected person bounding box and the detected wrapping bounding box.

[0022] In some embodiments, the computer-executable code is configured to insert a character bounding box in one of the character trajectories when there is a missing position between two positions of the character trajectory, and to insert a package bounding box in one of the package trajectories when there is a missing position between two positions of the package trajectory.

[0023] In some embodiments, the computer-executable code is further configured to detect and obtain key points of a person corresponding to each of the plurality of person bounding boxes.

[0024] In some embodiments, the key points of the person include a wrist key point. When the closest distance between the wrist key point and the corresponding package boundary frame in the package boundary frame is less than 10 pixels, the corresponding person picks up the corresponding package. When the closest distance after picking up the package is less than 10 pixels, the corresponding person holds the corresponding package. When the closest distance after holding the package is equal to or greater than 10 pixels, the corresponding person puts down the corresponding package.

[0025] In some embodiments, the threshold distance is the pixel size d of the corresponding person's torso. torso The relative distance between them.

[0026] In some embodiments, the corresponding person picks up the corresponding package in frame (tu), holds the corresponding package from frame (t-u+1) to frame (t-1), and puts down the corresponding package in frame t; the movement distance and movement speed are calculated for each frame starting from frame (t+x) until the movement distance is greater than the threshold distance and the movement speed is greater than the threshold speed; for frame (t+y), the movement distance is determined by d qq / d torso The calculation shows that the moving speed is determined by... Calculate d qq d is the pixel distance between the center of the corresponding bounding box of a wrapper and the center of the bounding box between the (t-1)th frame and the (t+y)th frame. (t-1)to(t+y) The distance between the center of the bounding box of the corresponding package and the center of the bounding box in frame (t-1) and frame (t+y) is the pixel distance; and the movement distance threshold is 1, the threshold speed is 0.1, t, u, x, and y are positive integers, x is in the range of 1 to 4, and x ≤ y. In some embodiments, x = 2, and the computer-executable code is configured to delete the action when y equals 19.

[0027] In some aspects, this disclosure relates to a method for detecting rough handling. In some embodiments, the method includes:

[0028] A video stream consisting of multiple video frames is captured by an imaging device;

[0029] The computing device obtains multiple character bounding boxes and multiple wrapping bounding boxes from the multiple video frames;

[0030] The computing device constructs multiple character trajectories and multiple package trajectories based on the obtained character bounding boxes and package bounding boxes;

[0031] The computing device identifies an action between one of the multiple person trajectories and one of the multiple package trajectories, wherein the action includes a corresponding person in one of the multiple person trajectories picking up, holding, and putting down a corresponding package in one of the multiple package trajectories; and

[0032] The computing device determines the existence of brute-force processing when, within a predetermined frame after the corresponding package is placed, the moving distance of the corresponding package is greater than a threshold distance and the moving speed of the corresponding package is greater than a threshold speed.

[0033] In some embodiments, obtaining the plurality of person bounding boxes and the plurality of wrapping bounding boxes includes:

[0034] Detect people in each video frame to obtain the bounding boxes of the multiple people;

[0035] Detect packages from each video frame to obtain the bounding boxes of the detected packages; and

[0036] Delete the stationary package bounding boxes from the detected package bounding boxes to obtain the plurality of package bounding boxes.

[0037] In some embodiments, the method further includes:

[0038] Calculate the frame difference between each video frame and the previous frame of each video frame;

[0039] The region with a high frame difference in each video frame is defined as the foreground region; and

[0040] The detected bounding box that does not overlap with the foreground region is defined as the static bounding box.

[0041] In some embodiments, the method further includes:

[0042] Add an additional package bounding box to the obtained package bounding box.

[0043] Wherein, when there is no overlap between a foreground region in the foreground region and the detected person bounding box and the detected wrapping bounding box, the additional wrapping bounding box is located in a foreground region in the foreground region.

[0044] In some embodiments, the method further includes obtaining key points of the characters corresponding to each of the plurality of character bounding boxes obtained.

[0045] In some embodiments, the key points of the person include a wrist key point. When the closest distance between the wrist key point and the corresponding package boundary frame in the package boundary frame is less than 10 pixels, the corresponding person picks up the corresponding package. When the closest distance after picking up the package is less than 10 pixels, the corresponding person holds the corresponding package. When the closest distance after holding the package is equal to or greater than 10 pixels, the corresponding person puts down the corresponding package.

[0046] In some embodiments, the threshold distance is the pixel size d of the corresponding person's torso. torso The relative distance between them.

[0047] In some embodiments, the corresponding person picks up the corresponding package in frame (tu), holds the corresponding package from frame (t-u+1) to frame (t-1), and puts down the corresponding package in frame t; the movement distance and movement speed are calculated for each frame starting from frame (t+x) until the movement distance is greater than the threshold distance and the movement speed is greater than the threshold speed; for frame (t+y), the movement distance is determined by d qq / d torso The calculation shows that the moving speed is determined by... Calculate d qq d is the pixel distance between the center of the corresponding bounding box of a wrapper and the center of the bounding box between the (t-1)th frame and the (t+y)th frame. (t-1 ) to(t+y) The distance between the center of the bounding box of the corresponding package and the center of the bounding box in frame (t-1) and frame (t+y) is the pixel distance; and the movement distance threshold is 1, the threshold speed is 0.1, t, u, x, and y are positive integers, x is in the range of 1 to 4, and x ≤ y. In some embodiments, x = 2, and the computer-executable code is configured to delete the action when y equals 19.

[0048] In some aspects, this disclosure relates to a non-transitory computer-readable medium for storing computer-executable code. When the computer-executable code is executed at a processor of a computing device, it is configured to perform the methods described above.

[0049] These and other aspects of this disclosure will become apparent from the following description of preferred embodiments in conjunction with the accompanying drawings and their headings, although variations and modifications therein may affect the novel conception of this disclosure without departing from it. Attached Figure Description

[0050] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements in the embodiments.

[0051] Figure 1 A crude handling identification system according to certain embodiments of the present disclosure is schematically depicted.

[0052] Figure 2 A conceptual workflow for frame background subtraction according to certain embodiments of the present disclosure is schematically depicted.

[0053] Figure 3A The results of people and packages detection in frames according to certain embodiments of the present disclosure are schematically depicted.

[0054] Figure 3B A schematic depiction of the corresponding Figure 3A The result of background subtraction of the frame.

[0055] Figure 4A A conceptual workflow for updating the trajectory of people and packages according to certain embodiments of this disclosure is schematically depicted.

[0056] Figure 4B An update of the trajectory according to certain embodiments of this disclosure is schematically depicted.

[0057] Figure 4C An update of the trajectory according to certain embodiments of the present disclosure is schematically depicted, wherein an insertion is performed.

[0058] Figure 4D The initialization of a new trajectory according to certain embodiments of this disclosure is schematically depicted.

[0059] Figure 4E An update of a trajectory according to certain embodiments of the present disclosure is schematically depicted, wherein the trajectory remains the same without any change.

[0060] Figure 4F The deletion of trajectories according to certain embodiments of the present disclosure is schematically depicted, in which the trajectories of a large number of frames do not match any person or package.

[0061] Figure 5A The calculation of the distance from a person to a package is illustrated schematically according to certain embodiments of the present disclosure.

[0062] Figure 5B A conceptual workflow for updating actions according to certain embodiments of this disclosure is schematically depicted.

[0063] Figure 5C A conceptual workflow for determining whether an action is a rough handling action, according to certain embodiments of the present disclosure, is illustrated schematically.

[0064] Figure 5D Operation according to certain embodiments of this disclosure is illustrated schematically.

[0065] Figure 5E The diagram schematically depicts the calculation of a human torso according to certain embodiments of the present disclosure.

[0066] Figure 5F The calculation of certain distances between a person and a package in different frames, and between packages in different frames, is illustrated schematically according to certain embodiments of the present disclosure.

[0067] Figure 6 A conceptual workflow of a crude processing identification system according to certain embodiments of the present disclosure is schematically depicted.

[0068] Figure 7 A method for identifying rough handling actions according to certain embodiments of the present disclosure is illustrated schematically. Detailed Implementation

[0069] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the accompanying drawings, throughout the views, the same numerals indicate the same parts. Unless the context clearly specifies otherwise, the terms “a,” “an,” and “the” as used herein and throughout the claims have the meaning of the plural. Furthermore, as used in the description and claims of this disclosure, unless the context clearly specifies otherwise, “in” has the meaning of “in” and “on”. Titles or subtitles may be used in the specification for the reader's convenience, without affecting the scope of the present disclosure. In addition, some terms used in this specification are given more specific definitions below.

[0070] The terms used in this specification generally have their common meaning in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe this disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of this disclosure. It will be understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms may be used for any one or more terms discussed herein, without any particular significance in whether the terms are elaborated or discussed herein. This disclosure provides synonyms for certain terms. The statement of one or more synonyms does not preclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is merely illustrative and in no way limits the scope or meaning of this disclosure or any exemplary terms. Likewise, this disclosure is not limited to the various embodiments given in this specification.

[0071] Unless otherwise defined, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that, unless explicitly defined, terms, such as those defined in commonly used dictionaries, should be interpreted as having the meaning consistent with their meaning in the relevant field and the context of this disclosure, and should not be idealized or given an overly formal meaning.

[0072] As stated in this article, the terms “including,” “contains,” “carries,” “has,” “includes,” “involves,” etc., should be understood as open-ended, meaning including but not limited to.

[0073] As described herein, at least one of the phrases A, B, and C should be interpreted as representing logic (A or B or C) using non-exclusive logical OR. It should be understood that one or more steps within the method may be performed in different orders (or simultaneously) without altering the principles of this disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0074] As described herein, the term "module" can refer to or include application-specific integrated circuits (ASICs), electronic circuits, combinational logic circuits, field-programmable gate arrays (FPGAs), processors (shared, dedicated, or grouped) that execute code, other suitable hardware components that provide the described functionality, or some or all of the above, such as in a system-on-a-chip. The term "module" can include memory (shared, dedicated, or grouped) that stores code executed by a processor.

[0075] The term "code" as used in this document can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared" as used above means that some or all of the code from multiple modules can be executed using a single (shared) processor. Furthermore, some or all of the code from multiple modules can be stored in a single (shared) memory. The term "group" as used above means that some or all of the code from a single module can be executed using a group of processors. Furthermore, a group of memories can be used to store some or all of the code from a single module.

[0076] As described herein, the term "interface" generally refers to a communication tool or device used at the interaction point between components to perform data communication between components. Generally, interfaces can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0077] This disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, shown as solid line blocks, and virtual software components, shown as dashed line blocks. Those skilled in the art will understand that, unless otherwise stated, these computer components may be implemented as software, firmware, or hardware components or combinations thereof, but are not limited to these forms.

[0078] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0079] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0080] In some aspects, this disclosure relates to rough handling systems and methods with the capability to detect rough handling of packages by workers in a distribution center. In some embodiments, this disclosure provides robust vision solutions and algorithms for intelligently identifying rough handling, recognizing rough handling actions in real time, and alerting the distribution center to postpone further rough handling actions. For example, if a worker quickly throws a package into the air or slams it to the ground, the system can recognize the action, locate the worker, immediately pinpoint the area of ​​the action, and then notify distribution center staff to inspect the package and instruct the worker to handle the goods more carefully. It should be noted that systems according to certain embodiments of this disclosure can be used in the cloud, on a local desktop, or even on lightweight and low-cost embedded devices. Furthermore, the system is powerful enough to be suitable for various large distribution centers or small distribution rooms. In summary, certain embodiments of this disclosure aim to build intelligent and robust rough handling recognition systems by detecting, tracking, and understanding worker actions.

[0081] Figure 1 A coarse handling identification system according to certain embodiments of this disclosure is schematically depicted. For example... Figure 1 As shown, system 100 includes imaging device 110, computing device 130, detection model service 170, and display device 190. Imaging device 110, computing device 130, detection model service 170, and display device 190 communicate with each other via network 120.

[0082] In some embodiments, imaging device 110 is a red-green-blue (RGB) camera, a grayscale camera, an RGB depth (RGBD) camera, or other types of image capture device. In some embodiments, imaging device 110 is a standard or wide-angle RGB camera. In some embodiments, there are multiple imaging devices 110, each installed in a monitored scenario, such as a conveyor belt in a parcel sorting center of an e-commerce platform. One or more workers work at the conveyor belt, and the field of view of imaging device 110 covers the workers and the parcels on or beside the conveyor belt. Imaging device 110 is configured to capture video streams in real time and send the captured video streams or video frames to a coarse-processing recognition application 138 of computing device 130. Because coarse-processing recognition application 138 can process a large number of video frames in parallel, the number of imaging devices 110 and the number of workers and parcels they cover can be large. In some embodiments, imaging device 110 is configured to capture video frames at, for example, 30 frames per second (fps), 60 fps, or any other suitable frame rate.

[0083] Network 120 can be wired or wireless, and can take various forms. Examples of networks may include, but are not limited to, a local area network (LAN), a wide area network (WAN) including the Internet, or any other type of network. In some embodiments, network 120 may be an interface, such as a system interface, USB drive, flash drive, hard disk drive, floppy disk, SD card, or optical drive, or any other type of interface for communicating between computing device 130 and imaging device 110, detection model service 170, and display device 190. In some embodiments, network 120 may include multiple different networks, each for communication between two or more of devices or services 110, 130, 170, and 190.

[0084] In some embodiments, computing device 130 may be a cloud computer, server computer, cluster, or local general-purpose computer that manages the rough handling identification system. In some embodiments, computing device 130 is a dedicated computer with limited processing, memory, and storage capacity, which reduces the cost of system 100 while still having sufficient capability to perform the functions of system 100. In some embodiments, computing device 130 is an embedded device. Figure 1 As shown, computing device 130 may include, but is not limited to, processor 132, memory 134, and storage device 136. In some embodiments, computing device 130 may include other hardware and software components (not shown) to perform its respective tasks. Examples of such hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0085] Processor 132 may be a central processing unit (CPU) configured to control the operation of computing device 130. Processor 132 may execute an operating system (OS) or other applications of computing device 130. In some embodiments, computing device 130 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. Memory 134 may be volatile memory, such as random access memory (RAM), for storing data and information during operation of computing device 130. In some embodiments, memory 134 may be an array of volatile memory. In some embodiments, computing device 130 may run on more than one memory 134. Storage device 136 is a non-volatile data storage medium for storing the operating system (not shown) and other applications of computing device 130. Examples of storage device 136 may include non-volatile memory such as flash memory, memory cards, USB drives, hard disk drives, floppy disks, optical drives, solid-state drives (SSDs), or any other type of data storage device. In some embodiments, storage device 136 may be local storage, remote storage, or cloud storage. In some embodiments, computing device 130 may have multiple storage devices 136, which may be the same storage device or storage devices of different types, and applications of computing device 130 may be stored in one or more storage devices 136 of computing device 130. In some embodiments, computing device 130 is a cloud computer, where processor 132, memory 134, and storage devices 136 are shared resources provided on demand via the Internet. In some embodiments, computing device 130 is an embedded system designed for robust processing identification, and may include a microprocessor with integrated memory and peripheral interfaces, or may include a general-purpose microprocessor.

[0086] like Figure 1As shown, storage device 136 includes a rough handling identification application 138. Rough handling identification application 138 is configured to provide rough handling identification and send warning messages to staff managing the distribution center so that staff can warn workers who have handled packages roughly. In some embodiments, warnings are automatically delivered to workers with rough handling issues. Rough handling identification application 138 includes a video capture module 140, a video decoder 142, a background subtraction (BGS) module 144, a detection module 146, a post-processing module 148, a tracking module 150, a motion recognition module 152, an output module 154, and a user interface 156. In some embodiments, rough handling identification application 138 may include other applications or modules necessary for its operation. It should be noted that the various modules are implemented by computer-executable code or instructions, collectively constituting an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code, and computing device 130 may be implemented by a simple circuit board with other accessory components mounted on the circuit board. In some embodiments, some or all modules of the brute-force identification application 138 may be located at a remote computing device or distributed in the cloud.

[0087] The video capture module 140 is configured to instruct the imaging device 110 to capture a video stream of a region of interest (e.g., a package or product distribution center) during the initialization of the coarse processing recognition application 138, and to send the captured video stream to the video decoder 142. In some embodiments, the video frames of the video stream are RGB images. When multiple imaging devices 110 are present, each imaging device 110 has a device identifier (ID) or camera ID, and each video frame has the camera ID of the imaging device 110 that acquired the video frame, a timestamp or frame number of the video frame, and the RGB image itself.

[0088] The video decoder 142 is used to decode the video stream into video frames upon receiving it, and to send each video frame to the BGS module 144 and the detection module 146. In some embodiments, each video frame is stored at a specific location in the storage device 136 and is assigned a pointer, which the module in the brute-force identification application 138 can use to manipulate the video frame. In some embodiments, when the video frame is an analog image, the video decoder 142 is also configured to convert the analog signal of the video frame into a digital signal. In some embodiments, video frames from each camera have the same camera ID and are placed in the same queue. The following discussion will use a queue of video frames as an example; the processing of multiple queued video frames captured by multiple imaging devices 110 is essentially the same. In this example, the queued video frames are captured by one imaging device 110 and have T frames: 1, ..., t-1, t, t+1, ..., T. The t-th frame is the current frame, and T and t are positive integers. Because the brute-force processing recognition application 138 operates in parallel, the video decoder 142 does not need to send video frames sequentially to the BGS module 144 and the detection module 146. For example, the video decoder 142 is configured to send video frames to the BGS module 144 and the detection module 146 whenever a frame is available. Each video frame has its corresponding camera ID (or queue ID) and timestamp (or / and frame number) so that the modules can identify the frames and logically organize the frame information.

[0089] The BGS module 144 is configured to subtract the previous frame from the current frame upon receiving the current frame from the video decoder 142 to obtain the foreground region of the current frame, and then send the foreground region to the post-processing module 148. The information in the video frames captured by the imaging device 110 may be complex, and brute-force processing only occurs on moving or undulating frames. Therefore, filtering out stationary regions in the frame and focusing only on moving regions can help speed up the runtime of downstream processes and improve the accuracy of the results. In some embodiments, for simplicity and efficiency, the BGS module 144 is configured to use frame difference for foreground calculation. In some embodiments, moving regions will have large pixel differences across two consecutive frames, and the goal of the BGS module 144 is to find these regions in the t-th frame. Specifically, the BGS module 144 is configured to calculate the pixel difference between the t-th frame and its preceding (t-1)-th frame, denoted as I, binarize I using a motion threshold, and generate a binary mask Im. In some embodiments, each pixel in frame t and frame (t-1) is converted to a grayscale value in the range of 0 to 255 before subtraction, and the motion threshold is set in the range of 1 to 100. In some embodiments, the motion threshold is set in the range of 10 to 30. In some embodiments, the motion threshold is set to 15. In some embodiments, pixels with a difference equal to or greater than the motion threshold are binarized to 1, while pixels with a difference less than the motion threshold are binarized to 0. That is, if the absolute value of the difference between pixels in frame t and frame (t-1) is greater than 15, it is assigned the value 1. Therefore, the binary mask I... m It has a value of 0 or 1 corresponding to each pixel in the t-th frame. In some embodiments, when generating the binary mask I... m Subsequently, the BGS module 144 was also configured to support the binary mask I. m Dilation and erosion algorithms are applied to remove noise. After dilation and erosion, a binary mask I is used. m This includes connected regions with a value of 1. In some embodiments, the BGS module 144 is also configured to filter connected regions using a predetermined size threshold so that only connected regions larger than the predetermined size threshold are retained. In one example, the predetermined size threshold is set to 1000 pixels in a 1280×720 or 1920×1080 video frame. The size of the video frame can vary, and in some embodiments, the video frame can also be an image cropped from a captured full-size frame. The filtered connected regions are foreground regions. For example, if frame t has f foreground regions compared to frame (t-1), these f foreground regions can be represented as... Where t represents the current time or current frame, and f represents the number of foreground regions. In some embodiments, to detect regions with movement (foreground), the BGS module 144 may also use the current frame t and a predetermined number of frames prior to frame t to calculate a reference image, and subtract the reference image from frame t to obtain the foreground region. In some embodiments, the BGS module 144 may use a background subtraction algorithm different from the algorithm described above. In some embodiments, pixel differences may be calculated directly from the RGB values ​​of pixels, instead of converting RGB values ​​to grayscale values, and the difference threshold may vary accordingly. In some embodiments, blocks are used instead of pixels to calculate frame differences, and each block may include several adjacent pixels, such as four or nine pixels. It should be noted that when the current frame is the first frame of the video stream, there is no previous frame, so there is no need to calculate the foreground region. In other words, the BGS module 144 begins its function from the second frame of the video stream. Figure 2 A flowchart 200 schematically depicts the function of the BGS module 144. (For example...) Figure 2 As shown, frame (t-1) 202 and RGB frame t 204 are provided. Pixel values ​​206 and 208 of the pixels in the two frames are calculated, which can be grayscale values ​​converted from RGB color values. The pixel values ​​of frame (t-1) are subtracted pixel by pixel from the pixel values ​​of frame t to obtain a pixel value difference 210. The absolute value of the pixel value difference 210 is compared with a motion threshold. Pixels with a difference equal to or greater than the motion threshold are assigned a value of 1, while pixels with a difference less than the motion threshold are assigned a value of 0, thereby generating a binary mask 212. The binary mask 212 is optimized through dilation and erosion 214 to obtain a filtered binary mask. The filtered binary mask contains connected regions, where pixels in the connected regions have a binary value of 1. The connected regions are then filtered by a size threshold at 216 to obtain the foreground region of frame t.

[0090] Return to reference Figure 1The detection module 146 is configured to, upon receiving the current frame from the video decoder 142, communicate with the detection model service 170 to detect bounding boxes and keypoints for people and bounding boxes for packages in the current frame, and send the detected bounding boxes for people, keypoints, and packages to the post-processing module 148. In some embodiments, the detection model service 170 is a standalone service that receives the current t-th frame from the detection module 146, detects people and packages in the t-th frame, and returns the detected people and packages to the detection module 146. The detection results from the detection module 146 include people and stationary / moving packages. The detection model service 170 can be implemented by, for example, a cloud computing device or a server computer. To maximize the use of hardware resources, the detection model service 170 is shared as a standalone service across all camera video streams. However, the detection model service 170 can also be a local service included as part of the detection module 146. In some embodiments, the detection model service 170 can use multiple detection models. In some embodiments, the detection model is a deep learning-based detection model. In some embodiments, the detection model is CenterNet, which provides good performance and high efficiency for the scenarios described in this disclosure. As described above, the detection model service 170 can process multiple frames in parallel after receiving copies of multiple frames from the detection module 146, with each processing instance named an inference model instance. When the detection results are sent back to the detection module 146 from the detection model service 170, this information includes the camera ID and the timestamp (frame number) of the corresponding frame, allowing the detection results to be well organized based on the camera ID and timestamp. In some embodiments, the detection includes person categories and package categories. The person detection results include the bounding box and multiple keypoints for each person detected from frame t. Keypoints may include, for example, the center of the head (multiple keypoints), the center of the neck (1 keypoint), the shoulder (2 keypoints), the elbow (2 keypoints), the wrist (2 keypoints), the hip (2 keypoints), the knee (2 keypoints), and the ankle (2 keypoints). The person detection results are used... The expression represents the current time or current frame, where t represents the current moment or current frame, and k is a positive integer representing the number of people detected from the current frame. It shows the detection results for the first detected person, including the bounding box and key points of the first detected person. This is the detection result for the k-th detected person, including its bounding box and keypoints. The bounding box can be defined by any suitable format, such as the center point position (pixel coordinates), the width of the bounding box (in pixels), and the height of the bounding box (in pixels). The package detection result only includes the bounding box of each package detected from frame t. The package detection result is expressed using... The expression represents the current time or current frame, where t represents the current moment or current frame, and l is a positive integer representing the number of packets detected from the current frame. This is the detection result of the first detection packet, including the bounding box of the first detection packet. This is the detection result of the l-th detection packet, including the bounding box of the l-th detection packet. The detection model service 170 is configured to provide two types of detection—people and packages—providing bounding boxes and keypoints for the people category and only bounding boxes for the package category. The advantage of this configuration is that it has different detection features for different detection categories. In some embodiments, the detection model service 170 is trained using both moving and stationary people, as well as moving and stationary packages, enabling it to detect people and packages in video frames more completely. However, some packages may be blurred in certain video frames if they move too fast and may not be detected by the detection model service 170; this will be handled by the post-processing process. In some embodiments, based on the model used by the detection model service 170, the result may also be a segmentation mask instead of bounding boxes and keypoints. The detection module 146 is also used to, after receiving the detection result from the detection model service 170, organize the result according to the camera ID and timestamp (frame ID), and send the organized result to the post-processing module 148. For example, detection module 146 can organize detection results into multiple queues, each queue including frames for one of the camera IDs. Within each queue, detection module 146 can organize the detection results sequentially according to the frame's timestamp or frame number. In some embodiments, detection model service 170 may also be part of detection module 146.

[0091] The post-processing module 148 is configured to, upon receiving the foreground region from the BGS module 144 and the detected person bounding box and key points from the detection module 146, and the detected package bounding box, remove the static package bounding box from the detected bounding boxes based on the foreground region, add an additional package bounding box based on the foreground region to obtain a moving package bounding box, and send the detected person bounding box and key points, as well as the moving package bounding box, to the tracking module 150. It should be noted that the detected person bounding box is usually within the foreground region, and no further processing of the detected person bounding box and key points is required. For the detected package bounding box, the post-processing module 148 is configured to remove static packages from the scene background. Specifically, the post-processing module 148 is configured to calculate the proportion of overlap between the package bounding box and any foreground region. If this proportion is equal to or greater than a predetermined overlap threshold, the detected package is considered a moving package and retained, where the overlap threshold can be 0.4. Otherwise, the detected package is deleted. In some embodiments, the post-processing module 148 is also configured to filter the person bounding box and package bounding box based on their size. For example, if a connected region corresponds to a person bounding box, and the connected region is smaller than 40 pixels × 60 pixels, the corresponding person bounding box is deleted. If a connected region corresponds to a package bounding box, and the connected region is smaller than 25 pixels × 25 pixels, the corresponding package bounding box is deleted. The size thresholds for person and package bounding boxes can vary depending on the distance between the imaging device 110 and the person / package, the image resolution, and / or the actual package size. The post-processing module 148 is also configured to provide additional detection results for fast-moving packages that were not captured by the detection model service 170. Because fast-moving packages are very blurry in the frame, the detection model service 170 has difficulty detecting them. However, fast-moving packages are clear in the background subtraction result. Therefore, the post-processing module 148 is configured to add high-speed signal markers to these package detections, bringing these fast-moving packages from the background subtraction into the final detection result to facilitate package tracking. Bounding box deletion and addition are as follows: Figure 3A and 3B As shown. Figure 3A The diagram schematically depicts the detection results of the detection model service 170 (or detection module 146) for the current frame. For example... Figure 3A As shown, the detection results include the detected person p1, and five detected packages q1, q2, q3, q4, and q5. The dashed box q0 is for illustration only; it is a package present in the current frame but not detected by the detection model service 170 because it is blurred due to its rapid movement. Figure 3B The foreground region generated by the BGS module 144 is schematically depicted. For example... Figure 3B As shown, the foreground region includes connected regions d1, d2, d3, and d4. The post-processing module 148 will... Figure 3AEach package bounding box in the ... Figure 3B The connected regions are compared. When one of the package bounding boxes overlaps with one of the foreground connected regions, the package bounding box is retained. Otherwise, the package bounding box is deleted. The overlap is evaluated by an overlap threshold, which is the percentage of the package bounding box within the corresponding connected region. In some embodiments, the overlap threshold is set to 0.2–0.6. In some embodiments, the threshold is set to 0.3–0.5. In one embodiment, the threshold is set to 0.4. For example, if package bounding boxes q1 and q2 overlap with foreground connected regions d3 and d4, and the overlap exceeds 0.4 of the area of ​​the detected package bounding boxes, package bounding boxes q1 and q2 are identified as moving packages and retained. In contrast, package bounding boxes q3, q4, and q5 do not overlap with any foreground connected regions (or the overlap is less than the overlap threshold) and are considered stationary packages and deleted. After that, there are foreground connected regions d1 and d2. Connected region d1 corresponds to the person bounding box p1 and is shelved. For a foreground connected region d2 without a corresponding bounding box, the post-processing module 148 further defines a bounding box surrounding the connected region d2 and names the newly defined bounding box a new wrapping bounding box q0. The wrapping bounding box q0 corresponds to the blurred wrapping in the frame. In some embodiments, the newly added bounding box q0 is marked with a high-speed signal marker because the wrapping moves very fast. After the deletion and addition of the wrapping bounding boxes, the post-processing result includes the character bounding box p1 and character key points, as well as the wrapping bounding boxes q1, q2, and q0. The post-processing module 148 further sends the post-processing result to the tracking module 150. The post-processing result can be represented as k character bounding boxes and key points. and l′ bounding boxes Where l′ is the number l minus the number of deleted static wrapper bounding boxes plus the number of additional wrapper bounding boxes.

[0092] The tracking module 150, upon receiving the bounding boxes and keypoints of the person detected in the current frame, as well as the bounding box of the moving package, tracks the person and the package in parallel and sends the updated trajectory of the person and the package to the action recognition module 152. As described above, there are k detected bounding boxes of the person and l′ of the moving package. The current frame is frame t. This disclosure assumes that there are m person tracking trajectories and n package tracking trajectories, which are available or active when processing frame t. Each of the m person tracking trajectories tracks the position of a specific person over time, and the trajectory includes consecutive bounding boxes and keypoints of the person in multiple consecutive frames before frame t. The bounding box and keypoints of the person corresponding to a frame are also referred to as the position in the trajectory. Each of the n package tracking trajectories tracks the position of a specific moving package over time, and the trajectory includes consecutive bounding boxes of the package in multiple consecutive frames before frame t. The bounding box of the package corresponding to a frame is also referred to as a position in the trajectory. When the current frame is the first frame of the video, there are no trajectories and no foreground calculation. After detecting the person and package from the first frame, a trajectory is initialized for each detected person and each detected package.

[0093] In the current frame t (t > 1), there are m character tracking trajectories and n package tracking trajectories, where m and n are positive integers (or 0 in some cases). The available trajectories are updated as follows. First, for each of the k detected character bounding boxes, the tracking module 150 is configured to compare the detected character bounding box with the character bounding box corresponding to the latest position in each character tracking trajectory. In some embodiments, matching is performed by calculating the Euclidean distance between the center of the detected character bounding box and the center of the bounding box at the latest position of the character tracking trajectory, and a greedy search optimization algorithm is used to find the best match based on the nearest neighbor metric. In some embodiments, the best match can also be confirmed by comparing the keypoints of the detected character with the keypoints of the best-matching character. When the best-matching character bounding box is determined, the detected character (his bounding box and keypoints in the current frame) is added to the character tracking trajectory with the matching character. If the character bounding box matching at the latest position of the corresponding character tracking trajectory is the bounding box detected in frame (t-1), the character tracking trajectory adds the detected character bounding box and keypoints as the last position of the character tracking trajectory.

[0094] When the best-matching character bounding box is identified, but the matching character bounding box was detected in a frame prior to frame (t-1), the tracking module 150 is configured to insert a character bounding box for the frames between the current frame and the frame with the matching bounding box. The center of the inserted character bounding box can be a point evenly distributed between the center of the corresponding character bounding box in the current frame and the center of the matching bounding box. The keypoints of the inserted character can be points evenly distributed between the corresponding keypoints in the current frame and the corresponding keypoints in the matching frame.

[0095] If the detected person's bounding box does not match, the tracking module 150 is configured to initialize a new person tracking trajectory, using the detected person as the starting position of the new person tracking trajectory.

[0096] After all k detected person bounding boxes have been matched, the tracking module 150 is further configured to check the person tracking trajectories of the detected person bounding boxes that have not been matched. If the latest position of the person tracking trajectory is in frame (t-10), the person tracking trajectory is deleted. If the latest position of the person tracking trajectory is between frame (t-9) and frame (t-1), the person tracking trajectory is retained. It should be noted that the person trajectory deletion threshold 10 here is a predetermined positive integer, and the person trajectory deletion threshold can be any number in the range of, for example, 2 to 30. In some embodiments, this disclosure sets the person trajectory deletion threshold based on the frame rate. For example, when the frames per second (fps) is 25, the threshold can be approximately 10, while when the fps is 5, the threshold can be approximately 5.

[0097] In some embodiments, the tracking module 150 is configured to process the moving package bounding box and package tracking trajectory in the same way, except that the package bounding box has no key points.

[0098] Figure 4A A flowchart 400 schematically depicts the function of the tracking module 150. (For example...) Figure 4A As shown, the detected character bounding box and moving package bounding box 402 of the current frame generated by post-processing module 148 are available, and the trajectory 404 established by tracking module 150 in the previous frame is available. The bounding boxes are compared with the latest position of the trajectory (the character bounding box is compared with the character tracking trajectory, and the package bounding box is compared with the package tracking trajectory). At 406, if the bounding box matches the latest position of a trajectory, then if there are no frames between the current frame and the latest position frame, the bounding box is added as the last position of the trajectory at 410. If there are one or more frames between the current frame and the latest position frame, then the bounding box is added as the last position of the trajectory at 412, and interpolated content is added for each frame between the current frame and the frame with the latest position.

[0099] At 408, if the bounding box does not match any of the latest positions of the trajectory, a new trajectory is initialized at 414, with the bounding box serving as the first position of the initialized trajectory. If the trajectory does not match any bounding box and its latest position is less than 10 frames prior to the current frame, the trajectory is preserved at 416. If the trajectory does not match any bounding box and its latest position is 10 frames (or more) prior to the current frame, the trajectory is discarded.

[0100] Figures 4B to 4F Some embodiments of the updated trajectory are schematically depicted. For example... Figure 4B As shown, a trajectory 420 exists. The first position of trajectory 420 is frame (t-11), and the latest position of trajectory 420 is position (boundary box) 422 in frame (t-1). When the post-processed bounding box (detected character bounding box or moving package bounding box) 424 matches the bounding box at the latest position 422, the post-processed bounding box 424 is added as the last position of trajectory 420. Figure 4C As shown, a trajectory 426 exists. The first position of trajectory 426 is frame t-9, and the latest position of trajectory 426 is position (boundary box) 428 in frame t-3. A post-processed bounding box 434 is matched with the bounding box at the latest position 428, and is added as the last position of trajectory 426. Two positions 430 and 432 are inserted between the latest position 428 and the post-processed bounding box 434. In some embodiments, positions 430 and 432 are evenly distributed between positions 428 and 434. For example, if the center of position / boundary box 428 is (x1, y1) and the center of position / boundary box 434 is (x2, y2), then the centers of positions / boundary boxes 430 and 432 are respectively... and The width and height of bounding boxes 430 and 432 can be the average of the width and height of bounding boxes 428 and 434. Furthermore, when trajectory 426 is a character tracking trajectory, keypoints for the character are added at insertion positions 430 and 432. The positions of the keypoints in positions 430 and 432 can be calculated using the same method as calculating the bounding box centers. Figure 4D As shown, the post-processing bounding box 436 does not match the latest position of any trajectory, a new trajectory is initialized, and the post-processing bounding box 436 serves as the first position of the new trajectory. Figure 4EAs shown, a trajectory 438 exists. The first position of trajectory 438 is frame (t-29), and the latest position of trajectory 438 is position (boundary box) 440 in frame (t-5). No post-processed bounding box matches the latest position 440. Since the latest position 440 is within a predetermined number of frames, for example, within 10 frames from the current frame, trajectory 438 is preserved. In some embodiments, the predetermined number can vary from 2 to 30 frames. In some embodiments, the predetermined number is 5 to 15. In this embodiment, the predetermined number is set to 10. Figure 4F As shown, trajectory 442 exists. The first position of trajectory 442 is in frame (t-37), and the latest position of trajectory 442 is position (boundary box) 444 in frame (t-10). No post-processed bounding box matches the latest position 444. Since the latest position 440 is 10 frames away from the current frame, trajectory 442 is discarded. As mentioned above, the discard threshold or predetermined number can be varied.

[0101] It is important to note that Figures 4B to 4F The illustrated trajectory can be a person tracking trajectory or a bounding box tracking trajectory, which can begin from any frame where a person or package is first detected. Matching the post-processed person / package bounding box with the bounding box of the latest position of the person / package trajectory only requires the bounding box. In some embodiments, keypoints can also be used to match the post-processed person bounding box, but this consumes significant computational resources and is not preferred. In some embodiments, the tracking module 150 is configured to perform dual tracking to track the person and package in parallel, comparing only the post-processed person bounding box with the latest position of the person tracking trajectory, and only the post-processed package bounding box with the latest position of the package tracking trajectory. In some embodiments, the tracking module 150 is configured to use the detection tracking framework described above to start and update the trajectories of the person and package separately. In some embodiments, the tracking module 150 may also use other measurements instead of calculating the Euclidean distance between the detected person / package bounding box and the latest position of the person / package trajectory, and use a greedy search optimization algorithm based on the nearest neighbor metric to find the best match. For example, detected bounding boxes, object edges, color appearance similarity or depth features, and related algorithms such as multi-hypothesis tracking, hypergraph-based association, pose estimation, and tracking can be used to complete person tracking and package tracking tasks. After updating the person tracking trajectory and package tracking trajectory, the tracking module 150 is also configured to send the updated trajectory to the action recognition module 152.

[0102] The motion recognition module 152 is configured to update the motion using the received person tracking trajectory and package tracking trajectory when it receives updated person tracking trajectory and package tracking trajectory from the tracking module 150, identify abrupt handling actions from the updated motion, and send the abrupt handling actions to the output module 154. As described above, each person tracking trajectory received from the tracking module 150 includes the bounding box and keypoints of the person at each time point or each frame, and each package tracking trajectory received from the tracking module 150 includes the bounding box of the package at each time point or each frame. A complete motion includes multiple consecutive frames of the person trajectory and the corresponding package trajectory. The motion recognition module 152 starts the motion when the person picks up the package, continues the motion when the person holds and puts down the package, and ends after a predetermined number of frames after putting down the package. The predetermined number is in the range of 5 to 50 frames. In some embodiments, the predetermined number is 10 to 25 frames. In some embodiments, the predetermined number is 20. Assuming the current time is t (or the current frame t), and the previous time was t-1 (or the previous frame t-1), the action recognition module 152 has the action from the previous time t-1 and will update the action using the person and package trajectory updated with frame t. Each action may include the person picking up the package, and optionally may also include the person holding and putting down the package. Let... Let be the action at time t-1, where s t-1 This represents the total number of actions at time t-1. Each action 'a' contains information about the character who triggered the action and the package that character picked up from the start of the action up to frame (t-1). The action updates at time t are as follows.

[0103] At time t, someone p i and his tracking trajectory f i t And package q j and its tracking trajectory Person tracking trajectory f i t Each location includes person p i The bounding box and key points, the package tracking trajectory Each location includes package q j The bounding box. The action recognition module 152 first determines the bounding box based on the person tracking trajectory f. i t Key information in the data, and the tracking trajectory The bounding box information of the package is used to determine the person p. j At time t, whether the package q is picked up, held, or put down. j . Figure 5A The determination of the distance between a human wrist keypoint and the package bounding box is shown in one example according to this disclosure. Figure 5A As shown, in the current frame, person p iThere are two key wrist points, w1 and w2, wrapped around q. j There is a top-left corner point q0. The shortest distance d1 from the wrist keypoint w1 to the bounding box is the distance from keypoint w1 to q0, and the shortest distance d2 from the wrist keypoint w2 to the bounding box is the distance from keypoint w2 to qx. The line w2-q... x Perpendicular to the left edge of the wrapping bounding box. Since distance d1 is less than distance d2, distance d1 is defined as the distance from the person's wrist to the wrapping bounding box. In some embodiments, when person p... i Any wrist key points and wrap q j When the bounding box of a person is within a predetermined distance at time t, the action recognition module 152 determines that the person is p. i At time t, pick up the package q j In some embodiments, the predetermined distance is in the range of 2 to 50 pixels. In some embodiments, the range is 5 to 20 pixels. In some embodiments, the predetermined distance is set to 10 pixels. Based on this information, the action recognition module 152 is further configured to recognize the action set A. t-1 Updated to A t That is, if person p i Pick up the package q j Then the module begins recording potential actions. If person p i Hold or drop off the package q j Then this module will perform the relevant actions. Updated to It also includes the location information of the person and the package; if person p i Lost package q j For a period of time, this module updates relevant actions using package location information. arrive And calculate package q j The distance and speed at which it moves are determined by its tracking information.

[0104] Figure 5B A flowchart 500 for an update action according to certain embodiments of the present disclosure is illustrated schematically, but the present disclosure is not limited thereto. References Figure 5BIn step 502, the motion recognition module 152 calculates the distance between each person and each package detected at the current time t (or in the current frame t). For each trajectory, there may be insertion positions added for people or packages in the trajectory. In step 504, the motion recognition module 152 also calculates the distance between the inserted person and the package or inserted package at that time for each inserted person (in the frame with the inserted person's position); and for each inserted package, calculates the distance between the inserted package and the person or inserted person at that time (in the frame with the inserted package's position). In some embodiments, each calculated distance is marked so that the distance between the same person (or inserted person) and the same package (or inserted package) in the same frame is not calculated repeatedly.

[0105] In step 506, the action recognition module 152 determines whether the person and package used to calculate the distance constitute an action, that is, whether an action has already occurred between the person and the package, and compares the distance with a predetermined distance. In some embodiments, the threshold distance is 10 pixels.

[0106] If the action recognition module 152 determines in step 506 that the person and package used to calculate the distance do not belong to an action, and the calculated distance between them is equal to or less than a predetermined distance of 10 pixels, then in step 508, the action recognition module 152 starts the action and defines the action state at the current time as "picking up". If the person and package do not belong to an action and the distance is greater than 10 pixels, no further analysis is required because there is no interaction between the person and the package. In some embodiments, when there is an insertion of a person or package, the action recognition module 152 can start the action from the earliest inserted frame with a distance of less than 10 pixels, and the action state in frames after the action starts is updated to "holding" or "putting down" based on the distance in each inserted frame. After step 508, processing continues for other person-package pairs, or if all person-package pairs have been analyzed, the process continues to the next frame.

[0107] If the action recognition module 152 determines in step 506 that the person and the package belong to an action, and the distance between them is equal to or less than a predetermined distance of 10 pixels, then in step 510, the state of the action at the current time is defined as "held," because a "pick-up" action already occurred in a previous frame (the first frame of the action). In some embodiments, when an insertion of a person or package occurs, the action recognition module 152 can update the state of the inserted frame in the action based on the distance in each inserted frame. After step 510, processing continues for other person-package pairs, or if all person-package pairs have been analyzed, the process continues to the next frame.

[0108] If the action recognition module 152 determines that the person and the package belong to the same action, and the distance between them is greater than 10 pixels, then in step 512, the action recognition module 152 determines whether the action is abrupt, as described below. The state of the action in the previous frame could be picking up, holding, or putting down.

[0109] Figure 5C A flowchart 520 for determining whether an action is a rough handling procedure, according to certain embodiments of the present disclosure, is schematically depicted, but the present disclosure is not limited thereto. References Figure 5C In step 522, the action recognition module 152 calculates the distance and speed between the package and the person after it is put down; in step 524, the action recognition module 152 determines whether there is a rough handling action based on the calculated distance and speed; in step 526, when the action is determined to be a rough handling action, the action recognition module 152 provides a score for the rough handling action based on a predetermined standard; in step 528, the action recognition module 152 can use a neural network to confirm the rough handling action, where this step is optional; then in step 530, the action recognition module 152 sends the rough handling action information to the output module 154 and sends the score to the output module 154.

[0110] Calculations are displayed Figures 5D to 5F middle. Figure 5D It shows a person p i Operation package q j Action A. The action starts from frame (tx), and the person p... i Pick up the package q j Then person p i Holding package q j For a period of time, until the (t-1)th frame; at the tth frame, person p i Put down the package q j The action ends after a predetermined number of frames in a dropped state (e.g., 20 frames from frame t to frame (t+19)), or when coarse processing is observed in any frame between frame t and frame (t+19). In some embodiments, the predetermined number is 10 to 30 frames. Figure 5D In the illustrated embodiment, the predetermined quantity is 20. In some embodiments, the distance is calculated with reference to the size of a human body relay or the human torso. Figure 5E As shown, person p i The frame contains head keyframes and two hip keyframes. The pixel distance d can be determined within the frame. head-hip1 The pixel distance d can be determined in the frame. head-hip2 , person p i The size of the human torso is the distance d head-hip1 and distance d head-hip2 The average value is defined as d. torsoIn some embodiments, the size of the human torso in the (t-1)th frame is used to calculate d. torso , through d torso Distance and velocity are calculated by comparison. In some embodiments, the calculation of distance and velocity for each frame may also use the human torso in that frame as a reference. For example, if the human torso is 10 pixels and the distance is 20 pixels, then the reference distance is the distance divided by the torso, i.e., 20 pixels / 10 pixels = 2.

[0111] In some embodiments, in order to have reliable calculations, distance and velocity calculations begin several frames after the drop, for example from frame (t+2), which is three frames after the last holding frame (t-1). Figure 5F The diagram schematically depicts the distance from frame (t-1) to frame (t+2) and the calculation of the velocity at frame (t+2). In a standard-sized frame, (t-1)-q j It is package q j The bounding box in frame (t-1) has its center point at o; (t+2)-q j It is package q j In the bounding box of frame (t+2), there is a center point o′ and a top-left corner point q0; It is a person p i One of the wrist keypoints in frame (t-1), it is closer to the bounding box (t-1)-q than the other wrist keypoints. j Taking the calculation of frame (t+2) as an example, the distance and velocity can be calculated as follows.

[0112] (1) Horizontal distance d of the package h That is, the horizontal pixel distance from center o to center o′, with the reference horizontal distance being d. h / d torso .

[0113] (2) Vertical distance d of the package v That is, the vertical pixel distance from center o to center o′, with the reference wrapping vertical distance being d. v / d torso .

[0114] (3) Package distance d qq That is, the pixel distance from center o to center o′, with the reference wrapping distance being d. qq / d torso .

[0115] (4) Person-package pixel distance d pq The key point of the wrist Distance from point q0. Note the pixel distance d between the character and the package. pq It is the key point of the wrist. With the bounding box (t+2)-q j The closest distance between edges. For example... Figure 5F As shown, the distance d pq It is the key point of the wrist. The distance between the point and point q0, but in other cases it can also be... To the bounding box (t+2)-q j The distance to another point on the edge is only d pq It is the closest distance between the wrist keypoint and the bounding box.

[0116] (5) Person-package distance, i.e., d pq / d torso .

[0117] (6) Instantaneous velocity v (t+2) The distance to a reference package is calculated based on a predetermined number of previous frames. In some embodiments, the predetermined number of frames is set to two or three. When the predetermined number of frames is set to three, the instantaneous velocity v at frame (t+2) is... (t+2) The calculation is as follows: Where d (t-1)to(t+2) It is the pixel distance between the centers of the bounding boxes of frames (t-1) and (t+2). Similarly, the instantaneous velocity v at frame t+3... (t+3) The calculation is as follows: Where d t to(t+3) It is the distance between the centers of the bounding boxes of frame t and frame (t+3). The maximum velocity v is also defined. max It is the maximum instantaneous velocity calculated from frame (t+2) to the current frame. Therefore, v max(t+2) The instantaneous velocity v is the velocity v when the current frame is the (t+2)th frame. (t+2) , because v (t+2) It is the only instantaneous velocity it possesses at this moment. When the current frame is frame (t+3), v max(t+3) It will be the instantaneous velocity v (t+2) and v (t+3) The larger one, when the current frame is frame (t+4), v max(t+4) It will be the instantaneous velocity v (t+2) v (t+3) and v (t+4) The largest of them. In some embodiments, this disclosure stores only one v. max Wherein, whenever the new instantaneous velocity at the current frame is greater than the stored v max Update its value when the value is updated.

[0118] (7) Calculate the acceleration a based on the average velocity of the current frame and the average velocity of the previous frame. Therefore, the acceleration a at frame (t+3) is... (t+3) Calculated as: a(t+3) =v (t+3) -v (t+2) .

[0119] (8) The average velocity of the current frame is calculated by dividing the distance the package has moved from frame (t-1) to the current frame by the frame number and the torso. Therefore, the average velocity of frame (t+2) is... Same as instantaneous velocity. The average velocity of the (t+3)th frame is... The average velocity of frame (t+4) is

[0120] In some embodiments, after calculating the distance and velocity, the action recognition module 152 is configured to determine whether the action is abrupt or invasive based on certain criteria from frames that are a predetermined number of frames from the last held frame. For example, the predetermined number of frames could be 2 to 12 frames. In some embodiments, the predetermined number is set to 6, returning to the reference... Figure 5D The last frame held is frame (t-1). The judgment of whether the action is rough processing starts from frame (t+5), 6 frames from frame (t-1). The criteria for rough processing in frame (t+5) include: (1)d h / d torso and d v / d torso (1) at least one of them is greater than 1; (2) 1 ≤ d h / d torso ≤5;(3)d pq Greater than 30 pixels; (4)d pq / d torso Greater than 1; (5)v max(t+5) Greater than 0.1 (instantaneous velocity v) (t+2) v (t+3) v (t+4) and v (t+5) (the largest one in the middle), or the current frame's bounding box has a high-speed marker (generated by the high-speed blurred region); (6) 0.05≤a (t+5) ≤1; (7) The average velocity v of the current frame average(t+5)Greater than 0.1. If all the above conditions (1) to (7) are met in frame (t+5), the action is determined to be a rough handling action, and no judgment is needed for frames after frame (t+5). If the above criteria are not met in frame (t+5), distance and speed calculations and rough handling judgments are performed in frame (t+6). The calculation and determination will end when one of the frames meets all the criteria. As described in step 516 below, if no frame meets all the criteria after a predetermined number of frames, the action is not a rough handling action, and the action recognition module 152 will delete the action. The predetermined number of frames can be, for example, 10 to 50 frames. In some embodiments, the predetermined number is 20, and the calculation and determination will stop at frame t+19, which is 20 frames away from frame (t-1). It should be noted that the above criteria are for illustrative purposes only, and the criteria may vary depending on practical conditions.

[0121] In some embodiments, after determining that an action is a rough handling action, the action recognition module 152 may be further configured to provide a rough handling score for the action. In some embodiments, rough handling actions are classified as highly rough handling actions or mildly rough handling actions, wherein highly rough handling actions are assigned 3 points and mildly rough handling actions are assigned 1 point. The requirements for highly rough handling include: cumulative distance >= 3.0, or average speed >= 0.2, or maximum speed >= 0.3. Correspondingly, the requirements for mildly rough handling include: cumulative distance < 3.0, average speed < 0.2, and maximum speed < 0.3. Taking frame (t+5) as an example, the cumulative distance in frame (t+5) is d. pq / d torso The average speed is Maximum speed v max(t+5) It is the instantaneous velocity v (t+2) v (t+3) v (t+4) and v (t+5) The maximum value in, where the (t-1)th frame is the last held frame and the tth frame is the dropped frame.

[0122] In some embodiments, rough handling actions are categorized as high, medium, or low severity. High severity actions are scored 3 points, medium severity actions 2 points, and low severity actions 1 point. The requirements for high severity include: cumulative distance >= 3.0, average speed >= 0.2, or maximum speed >= 0.3. The requirements for medium severity are: 3.0 > cumulative distance >= 1.5, 0.2 > average speed >= 0.15, and 0.3 > maximum speed >= 0.25. Any rough handling action that does not meet the high or medium severity requirements is considered a low severity action.

[0123] In some embodiments, when an insert frame is available, the insert frame can also be analyzed as the current frame as described above to determine whether bruising occurs.

[0124] In some embodiments, the action recognition module 152 is further configured to determine whether the package detection in the current frame is lost for the package in the action. If lost, the tracking of the package is deleted, and the action is deleted.

[0125] When the action recognition module 152 determines in step 512 that the action is not abrupt and the number of drop frames from the first drop frame to the current frame is less than 20, in step 514, the state of the current frame is updated to drop and other character-package pairs are processed, or if all character-package pairs have been analyzed, the process continues to the next frame.

[0126] When the action recognition module 152 determines in step 512 that the action is not abrupt and the number of frames from the first drop frame to the current frame is equal to or greater than 20, in step 516, the action recognition module 152 deletes the action and continues processing other person-package pairs, or if all person-package pairs have been analyzed, it continues to the next frame. The action is deleted because the system determines that the action is not abrupt after the package has been dropped for a sufficient period of time to make this judgment.

[0127] When the action recognition module 152 determines in step 512 that the action is abruptly processed, in step 518, the action recognition module 152 terminates the action and sends the abruptly processed action to the output module 154. Then, it continues processing other person-package pairs, or, if all person-package pairs have been analyzed, continues to the next frame. The action terminates because the system determines the action is abruptly processed based on the currently available next frame, requiring no further analysis of the action. (The last sentence appears to be incomplete and possibly refers to a separate process involving a person / package.) i With package q j The actions in between are considered abrupt actions. The abrupt action information includes: the video segment of the abrupt action (from the pick-up frame to 20 frames from the last held frame or 19 frames from the put-down frame), and the human p. i Character bounding box and key points in motion frames, q j The action frame contains the bounding box, calculated distance, calculated speed, and roughness score. The action recognition module 152 is then configured to output the roughness action information to the output module 154.

[0128] In some embodiments, the action recognition module 152 is optionally configured to apply video clips of rough-processed actions to a neural network to determine whether the action is rough-processed before outputting the rough-processed action information to the output module 154. However, the above judgment already has high accuracy, and neural network confirmation may not be necessary. In some embodiments, the neural network is implemented as a filter model to remove certain negative cases of rough-processed actions. The input to this module is the video clips of the identified rough-processed actions, along with associated person trajectories and wrap trajectories. In some embodiments, the model is I3D, a deep learning neural network. I3D is configured to classify the input video clips into two categories: rough actions or normal actions. The I3D model is a 3D convolutional neural network trained on video clips from both action classes. It is worth noting that other measurements (e.g., body keypoint trajectories and wrap trajectories) and association algorithms (e.g., pose estimation methods and other action recognition methods, etc.) can also be used to perform this task. The output of this module is the positive action items filtered by the network.

[0129] Based on the above description, it can be understood that the action can end in the following situations: (1) when it is confirmed that this is a rough action before the 20th frame, or (2) if the action is still not a rough action at the 20th frame, then the action is deleted (as the speed of putting down the package decreases, it will not be a rough action after the 20th frame), or (3) the action is not a rough action before the 20th frame, and the package is picked up again. If so, then the cycle of pick-up-hold-put down is repeated.

[0130] Return to reference Figure 1 The output module 154 is configured to prepare a rough handling report and send it to the user interface 156 after receiving rough handling action information (with or without I3D model filtering). Preparation may include creating a video clip of the rough handling action, recording the calculated distance and velocity of relevant frames in the video clip, recording the score of the rough handling action, optionally retrieving the complete trajectory of the person who caused the rough handling action and the complete trajectory of the package that was rough handled, and optionally preparing a warning recording or warning message for the rough handling.

[0131] User interface 156 is used to send the prepared report to display device 190 for display. In some embodiments, the responsible administrator monitors reports from display device 190. When a rough handling report is received, the administrator will view the report on the display device, especially the video clip of the report, to confirm the rough handling, notify the worker (person) who caused the rough handling, and inspect the package that was handled roughly by the worker.

[0132] In some embodiments, no administrator involvement is required in the process. The display device 190 is installed in a place where the worker can see it and has a speaker or messaging function, so that the display device 190 can broadcast warning recordings to the worker or send warning messages to the worker, who can then react accordingly, such as taking note of their handling habits and possibly inspecting packages that they have handled roughly.

[0133] In some embodiments, the user interface 156 is also configured for system administrators to adjust parameters of the imaging device 110, computing device 130, detection model service 170, display device 190, and coarse processing identification application 138.

[0134] As described above, the detection model service 170 is preferably a standalone service configured to receive frames from the detection module 146, detect people and packages in the frames, and send the detected people and packages to the detection module 146. Frames in the detection model service 170 can be processed in parallel. The detection results may include IDs (e.g., the camera ID of the frame), the timestamp or frame number of the frame, people detected in the frame (including the bounding boxes and keypoints of the people), and packages detected in the frame as package bounding boxes.

[0135] Display device 190 is configured to display the rough handling to an administrator upon receiving output from user interface 156. In some embodiments, display device 190 is configured to display video footage of the rough handling action, including a bounding box of a worker performing the rough handling and a bounding box of a package being rough handled. In some embodiments, calculated distances and speeds, as well as states of picking up, holding, and putting down, may be hidden from the display or may be displayed if the administrator chooses to do so. Display device 190 may be a computer monitor, a television, or even an audio device configured to broadcast warning voices or warning messages. In some embodiments, display device 190 may be controlled by computing device 130, or in some embodiments, by a computing device external to and communicating with computing device 130.

[0136] It should be noted that system 100 may further include other components as needed, and is not... Figure 1 All components shown are necessary for all applications. For example, when the computational load is not heavy, the detection model service 170 may not be necessary, and its functionality can be directly housed in the tracking module 150. In some embodiments, system 100 monitors rough handling in real time. In other embodiments, system 100 may also use recorded video to determine rough handling.

[0137] Figure 6 A conceptual workflow 600 of a coarse-handling identification system according to certain embodiments of this disclosure is illustrated schematically. For example... Figure 6 As shown, from the moment a worker picks up a package to the moment they put it down, the rough handling detection system collects information about the worker's handling process and provides real-time feedback to back-end staff. When the system detects rough handling, it sends video footage of the action to the back-end and alerts the dispatch staff that the worker handled the package roughly. The staff can then instruct the worker to handle the package more carefully, reducing potential damage. Therefore, one employee can simultaneously serve multiple processing areas and monitor the number of workers. This helps reduce labor costs, lower package damage rates, and reduce potential complaints from package owners.

[0138] In some respects, this disclosure relates to a method for identifying or monitoring rough handling. Figure 7 It schematically depicts what can be made by Figure 1 The system 100 shown implements a crude processing identification method. In some embodiments, Figure 7 The method 700 shown corresponds to the functionality of the coarse-processing identification application 138. It should be specifically noted that, unless otherwise stated in this disclosure, the steps of the method may be arranged in a different order, and therefore are not limited to... Figure 7 The order shown.

[0139] like Figure 7 As shown, in step 702, the video capture module 140 instructs the imaging device 110 to acquire a video stream of the region of interest in real time and sends the video stream to the video decoder 142. The imaging device 110 can be a wide-angle surveillance camera, and the region of interest can be the work area of ​​a parcel sorting center of an e-commerce company. Multiple imaging devices 110 can be installed in multiple different parcel sorting centers, each covering a work area, which may include one or more conveyor belts and one or more workers adjacent to the conveyor belts and handling parcels within the area. Each video stream includes real-time video frames, each video frame having a camera ID identifying the imaging device 110 that captured the video frame, a timestamp or frame number of the video frame, and the video frame itself, such as an analog or digital RGB image. The frame rate of the video stream can vary depending on the scene and hardware, for example, it can be 30fps.

[0140] In step 704, after receiving video frames from imaging device 110, video decoder 142 decodes each video stream into video frames and sends each video frame to BGS module 144 and detection module 146. In some embodiments, video decoder 142 stores video frames in each video stream as a queue, with the queue ID being the same as or equivalent to the device ID or camera ID. In some embodiments, the video stream includes analog video frames, and video decoder 142 converts the analog signals into digital signals. The modules of the brute-force processing identification application 138 process the video streams in parallel in the same manner, using one of the video streams to explain the following process. An example video stream has its specific camera ID and has T frames: 1, ..., t-1, t, t+1, ..., T. The t-th frame is the current frame, and T and t are positive integers. The t-th frame has its camera ID, timestamp, and digital image.

[0141] In step 706, when frame t is received from video decoder 142, BGS module 144 subtracts frame (t-1) from frame t to obtain the foreground region t of frame t, and sends the foreground region t to post-processing module 148. In some embodiments, BGS module 144 performs the subtraction as follows: converts the RGB values ​​of pixels in frame (t-1) and frame t to grayscale values ​​of 0 to 255, calculates the pixel difference between frame (t-1) and frame t, compares the absolute value of the difference with a predetermined difference threshold, and if the absolute value of the difference is equal to or greater than the difference threshold, binarizes the pixel to 1; if the absolute value of the difference is less than the difference threshold, binarizes the pixel to 0. Dilates and erodes the binary values ​​to obtain an initial binary mask containing connected regions with a value of 1, and filters the connected regions in the initial mask using a connected region threshold to obtain the foreground region t of frame t. In some embodiments, the predetermined difference threshold is 20. In some embodiments, the connected component filtering threshold is 1000 pixels, meaning only connected components with 1000 pixels are retained, while smaller connected components are deleted. Therefore, the t-th foreground region of frame t includes the camera ID, timestamp, and the foreground region of frame t. The t-th foreground region is represented as... The number of foreground regions is f, where f is 0 or a positive integer. When f is 0, there is no foreground region. It is the first of the f foreground regions. It is the f-th foreground region in the t-th frame. In some embodiments, step 706 may further include filtering connected regions using a predetermined size threshold so as to retain only connected regions larger than the predetermined size threshold. For example, the predetermined size threshold may be 1000 pixels in a 1280×720 or 1920×1080 video frame.

[0142] In step 708, upon receiving frame t from video decoder 142, detection module 146 detects the bounding boxes and keypoints of people and the wrapping bounding boxes of people from frame t, and sends the bounding boxes and keypoints of people and the wrapping bounding boxes to post-processing module 148. In some embodiments, detection module 146 uses detection model service 170 for people detection and wrapping detection. In some embodiments, detection model service 170 uses CenterNet to process frames received from detection module 146 in parallel. Detection model service 170 uses two types of detection: people detection and wrapping detection. Detection model service 170 detects bounding boxes and keypoints for people category objects, and only detects bounding boxes for wrapping category objects. In some embodiments, detection module 146 may also perform the detection function itself without using detection model service 170. The detection result of frame t can be represented as... and Where t represents the result of frame t, k is a positive integer (or 0 if no person was detected), representing the number of people detected in frame t, and l is a positive integer (0 indicates no package was detected), representing the number of packages detected in frame t. This is the detection result of the k-th detected person, including the bounding box and key points of the k-th detected person. This is the detection result of the first detection packet, including the bounding box of the first detection packet. The result also includes the camera ID and the timestamp of the t-th frame.

[0143] In process 710, upon receiving the t-th foreground region of the t-th frame from the BGS module 144 and the detected person bounding boxes, key points, and wrapping bounding boxes from the detection module 146, the post-processing module 148 removes the detected static wrapping bounding boxes based on the t-th foreground region and adds additional motion (or moving) wrapping bounding boxes to obtain moving wrapping bounding boxes, and sends the person bounding boxes, key points, and moving wrapping bounding boxes to the tracking module 150. In some embodiments, the post-processing module 148 removes static wrapping bounding boxes and adds additional moving wrapping bounding boxes by: overlapping the wrapping bounding boxes detected in the t-th frame with the t-th foreground region; deleting detected wrapping bounding boxes that do not overlap or only slightly overlap with the t-th foreground region; and adding additional moving wrapping bounding boxes corresponding to these foreground regions when the t-th foreground region is large enough that it does not overlap with any detected person bounding boxes and wrapping bounding boxes. In some embodiments, each additional moving wrapping bounding box is marked with a high-speed signal marker. In some embodiments, the result of the post-processing module 148 is determined by... and This indicates that the detected person is the same as the result of detection module 146, and the bounding box only contains moving bounding boxes. The added additional moving bounding boxes are marked with a high-speed signal marker. Here, l′ is the number l minus the number of deleted stationary bounding boxes plus the number of added bounding boxes. In practice, l′ is usually less than the number l. In some embodiments, step 710 may also include filtering the person bounding box and the bounding box based on their size, for example, deleting person bounding boxes with corresponding connected regions smaller than 40 pixels × 60 pixels, and deleting bounding boxes with corresponding connected regions smaller than 25 pixels × 25 pixels.

[0144] In step 712, after receiving the bounding box and key points of the person and the bounding box of the moving package from the post-processing module 148, the tracking module 150 tracks the person and the moving package to obtain or update the person tracking trajectory and the package tracking trajectory, and sends the updated person tracking trajectory and package tracking trajectory to the motion recognition module 152. When the current frame is the first frame of the video, there are no trajectories yet, and the BGS module 146 and the post-processing module 148 do not need to perform their functions to initialize the person trajectory for each person detected by the detection module 146 and the package trajectory for each package detected by the detection module 146. When the current frame is frame t, before processing frame t, there are m person trajectories and n package trajectories available. The trajectory of each person can be initialized from any previous frame when he was first detected and continuously tracked up to frame (t-1) or one or more frames before frame (t-1), that is, the latest frame where the person is still visible. Each moving package trajectory can be initialized from any previous frame when the moving package is first detected, and continuously tracked up to frame (t-1) or one or more frames before frame (t-1), which is the latest frame where the moving package is still visible. In other words, a person's trajectory or a moving package's trajectory is only deleted if no person or package is seen within a predetermined number of consecutive frames (e.g., 5–20 frames). In one example, the predetermined number of consecutive frames is 10. The position of a person in the latest frame of their trajectory is their last position. The position of a package in the latest frame of its trajectory is its last position. The result of the previous step includes k people from frame t. and l' mobile packages Each person has their own bounding box and keypoints, and each package has its own bounding box. For each of the k people, the person's bounding box is compared with the last position (person bounding box) of each of the m person trajectories to find a match. If a match is found, the person's bounding box at frame t is added to the matched person trajectory as the latest position of the person trajectory. Further, if the last position of the person trajectory before matching is not at frame (t-1), an insertion is made between the last position and frame t, inserting one or more person bounding boxes and keypoints. If no match is found, a new person trajectory is built using the person's bounding box at frame t. For each of the l′ moving packages, the package bounding box is compared with the last position (package bounding box) of each of the n package trajectories to find a match. If a match is found, the package bounding box at frame t is added to the matched package trajectory as the latest position of the package trajectory. Further, if the last position of the package trajectory before matching is not at frame (t-1), an insertion is made between the last position and frame t, inserting one or more package bounding boxes. If no match is found, a new package trajectory is established using the bounding box of the package from frame t. For a person or package trajectory that does not match in frame t, the number of frames from the latest position to the current frame is calculated; if it exceeds a threshold (e.g., 10 frames), the trajectory is deleted. Therefore, at time t, there are multiple person trajectories and multiple package trajectories. For a person p... i His tracking trajectory is now f i t For a package q j His tracking trajectory is now Person tracking trajectory f i t Each location includes the character's bounding box and keypoints at that location or frame, and the package tracking trajectory. Each location includes a bounding box that wraps around that location or frame.

[0145] In step 714, upon receiving updated person tracking trajectory and package tracking trajectory from tracking module 150, action recognition module 152 updates the action using the current (updated) person tracking trajectory and moving package tracking trajectory. Each action includes multiple consecutive person tracking positions (or frames) and corresponding package tracking positions (or frames), starting from when the person picks up the package and ending at a predetermined number of frames after the person puts down the package. Assume the action at time (t-1) or in frame (t-1) is... Among them are s t-1 Each action involves both character and package trajectories. Action updates are performed on the trajectory of each character and each package, following a process similar to... Figure 5B The description is as follows.

[0146] For person p in frame t i and package q j It belongs to the character tracking trajectory f i t Package tracking trajectory The action recognition module 152 first calculates the person p i and package q j The distance between them is recorded in step 502. The character tracking trajectory f before frame t... i t When one or more consecutive characters are inserted, or when the package tracking trajectory is before frame t. When one or more packages are inserted consecutively, in step 504, the distance between the person and the corresponding inserted package, the person inserting the package and the corresponding package, or the person inserting the package and the corresponding inserted package are all calculated and recorded.

[0147] The action recognition module 152 then determines the person p in step 506. i and package q j Does it belong to A? t-1 One of them will be people p i and package q j The distance between them is compared to a predetermined value (e.g., 10 pixels). If person p i and package q j It does not belong to one of the actions, in human p i and package q j When the distance to a human is greater than a threshold (e.g., 10 pixels), the action recognition module 152 sets it aside; when the distance to a human is greater than a threshold (e.g., 10 pixels), the action recognition module 152 sets it aside; i and package q j When the distance is equal to or less than the threshold, a new action is started in step 508, where the state of the new action in frame t is "pick up".

[0148] If person p i and package q j If it is not one of the actions, and there is a person or package inserted, then the action recognition module 152 will start processing from the earliest frame with the insertion, for example, frame (t-3). If person p i and package q j If the distance is greater than the threshold of the (t-3)th frame, no processing is required; if person p... i and package q j If the distance at frame (t-3) is less than the threshold, then for person p... i and package q j A new action is initiated, where the state of the new action in frame t is "pick up". The action recognition module 152 then continues processing for frames (t-2), (t-1), and t.

[0149] If person pi and package q j This belongs to one of the actions, the state of the action in frame (t-1) is "pick up" or "hold", and if the person p in the current frame... i and package q j If the distance between them is equal to or less than the threshold of 10 pixels, then the action recognition module 152 sets the action state of the current frame (the t-th frame) to "hold" in step 510.

[0150] In step 512, if person p i and package q j If an action is classified as "put down" in frame (t-1) and the distance between the person and the package is greater than a threshold (e.g., 10 pixels), then the action recognition module 152 determines whether the action is abrupt and checks whether the number of frames after the last "hold" frame is equal to or greater than the put-down threshold number (in the range of 10 to 30). In some embodiments, the put-down threshold number is 20.

[0151] In very rare cases, when person p i and package q j When an action belongs to one of the actions, the state of the action in frame (t-1) is "put down". If the distance between the person and the package in frame t is equal to or less than the distance threshold, the action recognition module 152 will recognize the action before the end of frame (t-1) and start a new action from frame t. If person p... i and package q j If an action belongs to one of the actions and there is a person or package inserted, the action recognition module 152 will similarly start processing from the earliest frame with the insertion and continue processing subsequent frames until frame t.

[0152] Through the above process, the action recognition module 152 uses the person tracking trajectory f in the current frame. i t Package tracking trajectory Updater p i and package q j Available actions or to begin a new action. For all k individuals. And all l' moving packages Repeat this process. Changes to actions include updating certain actions, initializing new actions, and ending certain actions. Currently available data includes the character's trajectory, the package's trajectory, the distance between the character and the package in the frame, and actions. And the action at the end of frame t. The determination of whether this action is abrupt is described below.

[0153] Return to reference Figure 7 In step 716, the action recognition module 152 determines whether the action at frame t is a rough handling action.i and package q j Action A in Figure 5D As shown in the image. The action begins in frame (tx), and the person p... i Pick up the package q j Then person p i Holding package q j For a period of time, until the (t-1)th frame; at the tth frame, person p i Put down the package q j The action ends a certain number of frames after the last hold or release. For example, Figure 5D The action ends 20 frames after the last holding frame (t-1). Therefore, the action includes frames (tx) to (t+19). In some embodiments, the determination of the rough handling can be performed at any drop frame before the ending frame (t+19). In fact, if rough handling is observed before frame (t+19), the action can end before frame (t+19). In some embodiments, the action recognition module 152 determines the rough handling based on multiple distances and velocities calculated starting from frame (t+2), i.e., three frames with the drop distance. The calculation includes referencing (or comparing) the horizontal distance d of the package. h / d torso Reference package vertical distance d v / d torso Reference direct package distance d qq / d torso Person-package pixel distance d pq Reference Person - Package Distance d pq / d torso Maximum speed, average speed, and acceleration a. Since distance and speed are calculated starting from frame (t+2), acceleration can only be calculated starting from frame (t+3). In some embodiments, the motion recognition module 152 determines the roughness from frame (t+5), calculates the distance, speed, and acceleration of frame (t+5), and the criteria for roughness include: (1) d h / d torso and d v / d torso At least one is greater than 1; (2) 1≤d h / d torso ≤5;(3)d pq Greater than 30 pixels; (4)d pq / d torso Greater than 1; (5)v max(t+5) Greater than 0.1, or the current frame contains a high-speed flag; (6) 0.05≤a (t+5) ≤1; (7) The average velocity v of the current frame average(t+5)The score is greater than 0.1. If the criterion is met, the action recognition module 152 terminates the action and further calculates the brunt processing score of the action; if the criterion is not met, the action continues, and the action recognition module 152 determines the next frame, i.e., frame (t+6). If the action recognition module 152 determines that there is no brunt processing for each frame from (t+5) to (t+19), the action is considered a normal action and will be discarded.

[0154] In step 718, after determining an action to be a rough action, the action recognition module 152 may optionally calculate a roughness score for the action and send the determined rough action and its roughness score to the output module 154. In some embodiments, the action recognition module 152 defines a high roughness score of 3 and a low roughness score of 1. A rough action is assigned a high roughness score at frame (t+5) or in any frame after frame (t+5) of the action: cumulative distance >= 3.0, average speed >= 0.2, or maximum speed >= 0.3. Here, the cumulative distance is the distance between the package in the current frame and the wrist keypoint of the person in the last holding frame, the average speed is the speed of the package in the last three frames (the current frame and the two frames before it), and the maximum speed is the maximum speed in a frame from frame (t+3) to the current frame. In some embodiments, the action recognition module 152 defines a high roughness score of 3, a medium roughness score of 2, and a low roughness score of 1. A high level of rough handling is given when the cumulative distance is greater than or equal to 3.0, the average speed is greater than or equal to 0.2, or the maximum speed is greater than or equal to 0.3. A moderate level of rough handling is given when 3.0 > cumulative distance > or equal to 1.5, 0.2 > average speed > or equal to 0.15, or 0.3 > maximum speed > or equal to 0.25. A low level of rough handling is given when the requirements for high or moderate level rough handling are not met.

[0155] In step 720, after receiving the rough processing action and its rough processing score from the action recognition module 152, the output module 154 prepares the rough processing output and sends the processed output to the display device 190. The rough processing output includes a video clip of the rough processing action, the trajectory of the person and the package (which may be a part of the trajectory corresponding to the action), including the bounding box and key points of the person and the bounding box of the package at each location, and the rough processing score.

[0156] When display device 190 receives rough-processed output from output module 154, it can display certain information from the output. For example, display device 190 can display a rough-processed video clip, showing the bounding box of the person and the bounding box of the surrounding area in each frame, and displaying the rough-processed score at the top of the video clip. Person keypoints, distances, velocities, and accelerations may be part of the output but are not displayed. Personnel managing display device 190 can choose to access that hidden information when needed.

[0157] Variations of the above method are available. For example, the method can change the order of certain steps, use detection module 146 to perform person and package detection instead of detection model service 170, change the threshold, select the human torso as a reference from different frames, select different number of frames for the action, and select different frames for the starting distance and speed calculation. In some embodiments, these parameter changes can be configured using user interface 156.

[0158] In another aspect, this disclosure relates to a non-transitory computer-readable medium for storing computer-executable code. When the code is executed at the processor of a computing device, the methods described above can be performed. In some embodiments, the non-transitory computer-readable medium may include, but is not limited to, any physical or virtual storage medium. In some embodiments, the non-transitory computer-readable medium may be implemented as follows: Figure 1 The storage device 136 of the computing device 130 shown.

[0159] In summary, certain embodiments of this disclosure provide a rough handling identification application with the following advantages: (1) The system automatically identifies rough handling, saving the cost of monitoring rough handling. (2) The system is robust in judging rough handling and can be installed in different environments or scenarios without complex configuration. (3) The system can be implemented by installing a local camera and connecting to one or more cloud computing devices via a network. Therefore, there are almost no limitations on computing power, making it simple, novel, and easy to expand. (4) The system can identify rough handling actions in real time, improving the efficiency of locating and correcting problems. (5) The BGS module, detection module, and detection model service can perform functions in parallel, resulting in fast analysis speed. (6) The system uses frame difference to calculate the background, removes static packages and adds high-speed packages using the background, compensating for the deficiencies of the detection model service. (7) The system uses insertion technology for the trajectory of people and packages to ensure accurate determination of the time of picking up, holding, or putting down the action. (8) The system combines different distances, speeds, and accelerations for rough handling judgment, which removes noise and accurately and robustly judges rough handling.

[0160] The foregoing description of exemplary embodiments of this disclosure is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Many modifications and variations are possible in accordance with the foregoing teachings.

[0161] The embodiments were chosen and described to explain the principles of this disclosure and its practical application, thereby enabling others skilled in the art to utilize this disclosure and various embodiments, as well as various modifications suitable for the particular intended use. Alternative embodiments will become apparent to those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. A system for detecting rough handling, comprising an imaging device and a computing device, the computing device including a processor and a storage device storing computer-executable code, wherein the computer-executable code is configured to: Receive a video stream comprising multiple video frames captured by the imaging device; Multiple character bounding boxes and multiple wrapping bounding boxes are obtained from the multiple video frames; Based on the obtained character bounding box and the package bounding box, construct multiple character trajectories and multiple package trajectories; Identify the action between one of the multiple character trajectories and one of the multiple package trajectories, wherein... The action includes a person in one of the multiple person trajectories picking up, holding, and putting down a package in one of the multiple package trajectories; and In a predetermined frame after the corresponding package is placed, if the moving distance of the corresponding package is greater than a threshold distance and the moving speed of the corresponding package is greater than a threshold speed, the existence of brute force processing is determined. In this context, the corresponding person picks up the corresponding package in frame tu, holds the corresponding package from frame t-u+1 to frame t-1, and puts down the corresponding package in frame t. Wherein, for the t+y-th frame, the moving distance is determined by The calculation shows that the moving speed is determined by... calculate, It is the pixel distance between the center of the bounding box of the corresponding package between frame t-1 and frame t+y. It is the pixel size of a person's torso, where t, u, and y are positive integers.

2. The system according to claim 1, wherein, The computer-executable code is configured to obtain the plurality of person bounding boxes and the plurality of wrapping bounding boxes for each of the plurality of video frames by performing the following operations: Detect people from each of the video frames to obtain the plurality of person bounding boxes; Detect packages from each video frame to obtain the bounding boxes of the detected packages; as well as Delete the stationary package bounding boxes from the detected package bounding boxes to obtain the plurality of package bounding boxes.

3. The system according to claim 2, wherein, The computer-executable code is also configured to: Calculate the frame difference between each video frame and the previous frame of each video frame; The region with a high frame difference in each video frame is defined as the foreground region; and The detected bounding box that does not overlap with the foreground region is defined as the static bounding box.

4. The system according to claim 3, wherein, The computer-executable code is further configured to add an additional wrapping bounding box to the obtained wrapping bounding box, wherein the additional wrapping bounding box is located in a foreground region of the foreground region when there is no overlap between a foreground region of the foreground region and the detected person bounding box and the detected wrapping bounding box.

5. The system according to claim 1, wherein, The computer-executable code is configured to insert a character bounding box into one of the character trajectories when there is a missing position between two positions of the character trajectory, and to insert a package bounding box into one of the package trajectories when there is a missing position between two positions of the package trajectory.

6. The system according to claim 1, wherein, The computer-executable code is also configured to detect and obtain key points of a person corresponding to each of the multiple person bounding boxes.

7. The system according to claim 6, wherein, The key points of the person include the wrist key point. When the closest distance between the wrist key point and the corresponding package boundary frame in the package boundary frame is less than 10 pixels, the corresponding person picks up the corresponding package. When the closest distance after picking up the package is less than 10 pixels, the corresponding person holds the corresponding package. When the closest distance after holding the package is equal to or greater than 10 pixels, the corresponding person puts down the corresponding package.

8. The system according to claim 7, in, Starting from frame t+x, calculate the moving distance and moving speed for each frame until the moving distance is greater than the threshold distance and the moving speed is greater than the threshold speed; Wherein, the threshold distance is 1, the threshold speed is 0.1, x is a positive integer, x is in the range of 1 to 4, and x ≤ y.

9. The system according to claim 8, wherein, x=2, and the computer-executable code is configured to delete the action when y equals 19.

10. A method for detecting rough handling, comprising: Capture a video stream that includes multiple video frames; Multiple character bounding boxes and multiple wrapping bounding boxes are obtained from the multiple video frames; Based on the obtained character bounding box and the package bounding box, construct multiple character trajectories and multiple package trajectories; Identify an action between one of the multiple person trajectories and one of the multiple package trajectories, wherein the action includes a corresponding person in one of the multiple person trajectories picking up, holding, and putting down a corresponding package in one of the multiple package trajectories; and In a predetermined frame after the corresponding package is placed, if the moving distance of the corresponding package is greater than a threshold distance and the moving speed of the corresponding package is greater than a threshold speed, the existence of brute force processing is determined. In this context, the corresponding person picks up the corresponding package in frame tu, holds the corresponding package from frame t-u+1 to frame t-1, and puts down the corresponding package in frame t. Wherein, for the t+y-th frame, the moving distance is determined by The calculation shows that the moving speed is determined by... calculate, It is the pixel distance between the center of the bounding box of the corresponding package between frame t-1 and frame t+y. It is the pixel size of a person's torso, where t, u, and y are positive integers.

11. The method according to claim 10, wherein, Obtaining the multiple character bounding boxes and the multiple wrapper bounding boxes includes: Detect people in each video frame to obtain the bounding boxes of the multiple people; Detect packages from each video frame to obtain the bounding boxes of the detected packages; and Delete the stationary package bounding boxes from the detected package bounding boxes to obtain the plurality of package bounding boxes.

12. The method of claim 11, further comprising: Calculate the frame difference between each video frame and the previous frame of each video frame; The region with a high frame difference in each video frame is defined as the foreground region; as well as The detected bounding box that does not overlap with the foreground region is defined as the static bounding box.

13. The method of claim 12, further comprising: Add an additional package bounding box to the obtained package bounding box. Wherein, when there is no overlap between a foreground region in the foreground region and the detected person bounding box and the detected wrapping bounding box, the additional wrapping bounding box is located in a foreground region in the foreground region.

14. The method of claim 10 further includes obtaining key points of the characters corresponding to each of the obtained character bounding boxes.

15. The method according to claim 14, wherein, The key points of the person include the wrist key point. When the closest distance between the wrist key point and the corresponding package boundary frame in the package boundary frame is less than 10 pixels, the corresponding person picks up the corresponding package. When the closest distance after picking up the package is less than 10 pixels, the corresponding person holds the corresponding package. When the closest distance after holding the package is equal to or greater than 10 pixels, the corresponding person puts down the corresponding package.

16. The method according to claim 15, in, Starting from frame t+x, calculate the moving distance and moving speed for each frame until the moving distance is greater than the threshold distance and the moving speed is greater than the threshold speed; Wherein, the threshold distance is 1, the threshold speed is 0.1, x is a positive integer, x is in the range of 1 to 4, and x ≤ y.

17. The method according to claim 16, wherein, When x=2, delete the action when y=19.

18. A non-transitory computer-readable medium storing computer-executable code, wherein the computer-executable code is configured, when executed at a processor of a computing device, to: Receive a video stream consisting of multiple video frames captured by an imaging device; Multiple character bounding boxes and multiple wrapping bounding boxes are obtained from the multiple video frames; Based on the obtained character bounding box and the package bounding box, construct multiple character trajectories and multiple package trajectories; Identify the action between one of the multiple character trajectories and one of the multiple package trajectories, wherein... The action includes a person in one of the multiple person trajectories picking up, holding, and putting down a corresponding package in one of the multiple package trajectories; and In a predetermined frame after the corresponding package is placed, if the moving distance of the corresponding package is greater than a threshold distance and the moving speed of the corresponding package is greater than a threshold speed, the existence of brute force processing is determined. In this context, the corresponding person picks up the corresponding package in frame tu, holds the corresponding package from frame t-u+1 to frame t-1, and puts down the corresponding package in frame t. Wherein, for the t+y-th frame, the moving distance is determined by The calculation shows that the moving speed is determined by... calculate, It is the pixel distance between the center of the bounding box of the corresponding package between frame t-1 and frame t+y. It is the pixel size of a person's torso, where t, u, and y are positive integers.

Images