A method, apparatus, equipment, and medium for detecting sorting behavior.

Abstract

This invention discloses a method, apparatus, device, and medium for detecting sorting behavior. The method includes: acquiring a current sorting video captured in a sorting scenario; extracting optical flow from every two adjacent sorting video frames in the current sorting video to determine each extracted optical flow map; transforming each optical flow map to determine a spatiotemporal static feature map corresponding to the current sorting video; inputting the spatiotemporal static feature map into a preset detection network model to detect preset sorting behavior; and obtaining the current detection result based on the output of the preset detection network model. Through the technical solution of this invention, automatic detection of sorting behavior can be achieved, timely detection of sorting behavior that damages packages by sorters can be identified without human intervention, reducing labor costs and ensuring detection accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to a method, apparatus, device and medium for detecting sorting behavior. Background Technology

[0002] With the rapid development of the logistics industry, logistics stations receive a large number of packages every day. Sorting staff need to manually sort these packages for proper distribution.

[0003] Currently, video surveillance can be used to manually detect whether sorting staff at logistics stations are damaging packages, thereby reducing the likelihood of package damage caused by sorting activities.

[0004] However, in the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0005] Existing manual inspection methods increase labor costs and are not easy to detect in a timely manner if sorters damage packages, resulting in missed inspections. Summary of the Invention

[0006] This invention provides a method, apparatus, equipment, and medium for detecting sorting behavior, enabling automatic detection of sorting behavior, timely detection of sorting workers damaging packages, eliminating the need for manual intervention, reducing labor costs, and ensuring detection accuracy.

[0007] In a first aspect, embodiments of the present invention provide a method for detecting sorting behavior, comprising:

[0008] Acquire the current sorting video captured in the sorting scenario;

[0009] Optical flow is extracted from every two adjacent sorting video frames in the current sorting video to determine each extracted optical flow map;

[0010] The optical flow maps are transformed to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0011] The spatiotemporal static feature map is input into a preset detection network model to detect preset sorting behaviors, and the current detection result is obtained based on the output of the preset detection network model.

[0012] Secondly, embodiments of the present invention also provide a detection device for sorting behavior, comprising:

[0013] The current sorting video acquisition module is used to acquire the current sorting video captured in the sorting scenario;

[0014] The optical flow map determination module is used to extract optical flow from every two adjacent sorting video frames in the current sorting video and determine each extracted optical flow map.

[0015] The spatiotemporal static feature map determination module is used to transform each of the optical flow maps to determine the spatiotemporal static feature map corresponding to the current sorting video;

[0016] The preset sorting behavior detection module is used to input the spatiotemporal domain static feature map into the preset detection network model to detect the preset sorting behavior, and obtain the current detection result based on the output of the preset detection network model.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0018] One or more processors;

[0019] Memory, used to store one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the sorting behavior detection method provided in any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sorting behavior detection method as provided in any embodiment of the present invention.

[0022] The embodiments of the above invention have the following advantages or beneficial effects:

[0023] By extracting optical flow from every two adjacent sorting video frames captured in the current sorting video, and transforming each extracted optical flow map, a spatiotemporal static feature map containing temporal motion features and spatial static features is determined. This spatiotemporal static feature map is then input into a preset detection network model to detect whether a preset sorting behavior exists. Based on the output of the preset detection network model, the current detection result is obtained, thereby realizing automatic real-time detection of sorting behavior. This allows for timely detection of sorting behavior that damages packages, eliminating the need for manual intervention and reducing labor costs. Furthermore, the detection process considers both the temporal motion features and spatial static features of the packages, ensuring the accuracy of sorting behavior detection. Attached Figure Description

[0024] Figure 1 This is a flowchart of a sorting behavior detection method provided in Embodiment 1 of the present invention;

[0025] Figure 2This is a flowchart of a sorting behavior detection method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a sorting behavior detection device provided in Embodiment 3 of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0029] Example 1

[0030] Figure 1 This is a flowchart illustrating a sorting behavior detection method according to Embodiment 1 of the present invention. This embodiment is applicable to detecting whether there is any damage to packages during the sorting process by sorters. The method can be executed by a sorting behavior detection device, which can be implemented in software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0031] S110. Obtain the current sorting video captured for the sorting scenario.

[0032] Here, "sorting scenario" refers to the application scenario where sorters sort packages. "Current sorting video" refers to the video captured by the camera within the current time period showing sorters sorting packages. The preset video length of the current sorting video can be set based on business needs and the actual scenario. For example, the average time required for a sorter to move a package during sorting can be set as the preset video length of the current sorting video, so that the complete package movement trajectory can be obtained based on the current sorting video, improving detection accuracy.

[0033] Specifically, a camera can be set up in the sorting scenario to capture real-time video of the sorting process, enabling real-time monitoring of the sorting personnel's actions. This embodiment can acquire a preset video length of the currently captured sorting video. For example, it can acquire a currently captured sorting video with a preset video length of 2 seconds. Alternatively, this embodiment can acquire a preset video length of the currently captured sorting video at intervals of that preset length and automatically detect sorting actions based on each acquired video, thereby achieving real-time monitoring of the sorting personnel's behavior.

[0034] It should be noted that the preset video length of the current sorting video can be arbitrarily set based on business needs and actual scenarios. This allows for the modeling of spatiotemporal features of time series of arbitrary lengths based on the current sorting video, without being limited by the time length, thereby improving the applicability of application scenarios.

[0035] S120. Perform optical flow extraction on every two adjacent sorting video frames in the current sorting video to determine the extracted optical flow maps.

[0036] Optical flow graphs are used to characterize the image changes between two adjacent sorting video frames, and can include the movement of packages in the sorting scene. Optical flow is a vector. Each pair of adjacent sorting video frames corresponds to one optical flow graph.

[0037] Specifically, optical flow is extracted from every two adjacent sorting video frames in the current sorting video to determine the motion of all possible moving objects in the sorting scene within the shooting time interval between each two adjacent sorting video frames. This includes the motion of packages when the sorter performs destructive sorting actions, thereby obtaining the optical flow maps corresponding to the current sorting video. For example, based on sorting video frame p at time t... t The sorting video frame p at time t-1 t-1 The optical flow graph at time t can be determined as flow. t If the current sorting video includes n sorting video frames, i.e. p0, p1, ..., p2, ... n-1 Then, n-1 optical flow maps can be obtained, namely flow1, flow2, ..., flow n-1 .

[0038] For example, S120 may include: inputting each two adjacent sorting video frames in the current sorting video into a preset optical flow extraction network model, and obtaining the optical flow map corresponding to each two adjacent sorting video frames according to the output of the preset optical flow extraction network model.

[0039] The preset optical flow extraction network model can be a pre-set network model that extracts optical flow from two image frames. For example, the preset optical flow extraction network model can be, but is not limited to, the PWC-Net optical flow estimation convolutional network model. In this embodiment, the preset optical flow extraction network model is obtained by pre-training based on sample data to ensure the accuracy of optical flow extraction.

[0040] Specifically, for each pair of adjacent sorting video frames, the sorting video frame p at time t can be... t The sorting video frame p at time t-1 t-1The input is fed into a pre-trained optical flow extraction network model for optical flow extraction, and the optical flow graph at time t is obtained based on the output of the pre-trained optical flow extraction network model. t This allows for the rapid acquisition of optical flow maps using a pre-defined optical flow extraction network model, thereby improving detection efficiency.

[0041] S130. Transform each optical flow map to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0042] In this context, a spatiotemporal static feature map can refer to an image that includes both temporal motion features and spatial static features. Temporal motion features can be used to characterize the movement of packages in a sorting scenario within a preset video length time period. Spatial static features can be used to characterize the spatial appearance of moving packages in a sorting scenario.

[0043] Specifically, by transforming and merging all optical flow maps, a spatiotemporal static feature map that simultaneously contains temporal motion features and spatial static features can be obtained.

[0044] S140. Input the spatiotemporal static feature map into the preset detection network model to detect the preset sorting behavior, and obtain the current detection result based on the output of the preset detection network model.

[0045] The preset detection network model can be a pre-set two-dimensional convolutional neural network model used to detect preset sorting behaviors. For example, the preset detection network model can be, but is not limited to, Faster R-CNN, CenterNet, and YOLO. In this embodiment, the preset detection network model is pre-trained based on sample data to ensure detection accuracy. Preset sorting behaviors can refer to sorting actions that damage packages, such as throwing or tossing them. When preset sorting behaviors exist, the packages have specific trajectories, such as parabolic trajectories, which can be used for automatic detection of the preset sorting behaviors.

[0046] Specifically, a spatiotemporal static feature map is input into a preset detection network model. This model detects the presence of preset sorting behavior based on the spatiotemporal static feature map and outputs the current detection result. This enables automatic real-time detection and alarm of preset sorting behavior, reducing the risk of package damage due to such behavior. If the preset detection network model detects preset sorting behavior, it can locate the area where the behavior occurs and output the located area for easier identification. The area where the preset sorting behavior occurs can be marked using a rectangle; by outputting the coordinates of the top-left and bottom-right vertices of the rectangle, the area can be quickly and easily identified.

[0047] It should be noted that, since a spatiotemporal static feature map containing both temporal motion features and spatial static features can be obtained, a lightweight two-dimensional convolutional neural network model can be used to detect preset sorting behaviors without the need for a temporal network model or a three-dimensional convolutional neural network model, which greatly improves detection efficiency and ensures detection accuracy.

[0048] The technical solution of this embodiment extracts optical flow from every two adjacent sorting video frames in the captured current sorting video, and transforms each extracted optical flow map to determine a spatiotemporal static feature map containing temporal motion features and spatial static features. The spatiotemporal static feature map is then input into a preset detection network model to detect whether a preset sorting behavior exists. Based on the output of the preset detection network model, the current detection result is obtained, thereby realizing the automatic real-time detection of preset sorting behavior. This can promptly detect sorting behavior that damages packages by sorters without human intervention, reducing labor costs. Furthermore, the detection process considers both the temporal motion features and spatial static features of the package, thus ensuring the accuracy of sorting behavior detection.

[0049] Based on the above technical solution, S130 may include: performing image transformation on each optical flow map to determine the temporal motion feature map corresponding to each optical flow map; and superimposing the various temporal motion feature maps to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0050] Specifically, for each optical flow map, feature extraction and transformation of motion information within the optical flow map can yield a temporal motion feature map that more clearly characterizes the motion, thereby improving detection accuracy. By overlaying the obtained temporal motion feature maps and using the overlaid image as a spatiotemporal static feature map, the temporal motion features are transformed into spatial static features, resulting in a spatiotemporal static feature map that simultaneously contains both temporal motion and spatial static features.

[0051] For example, performing image transformation on each optical flow map to determine the temporal motion feature map corresponding to each optical flow map may include: for each optical flow map, converting the optical flow map into a corresponding RGB color image, and using the RGB color image as the temporal motion feature map corresponding to the optical flow map.

[0052] In this context, an RGB color image refers to an image where each pixel is composed of R (Red), G (Green), and B (Blue) components. Specifically, by converting each optical flow map into a corresponding RGB color image, different colors can be used to represent different motion conditions. For example, the darker the color at a certain location in an RGB color image, the faster the object at that location is moving. For stationary objects in a sorting scenario, the same color can be used for representation, thereby extracting moving packages, effectively removing interference from complex backgrounds, and further improving detection accuracy.

[0053] Example 2

[0054] Figure 2 This is a flowchart of a sorting behavior detection method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further optimizes the step of "converting the optical flow diagram into a corresponding RGB color image". Explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0055] See Figure 2 The sorting behavior detection method provided in this embodiment specifically includes the following steps:

[0056] S210. Obtain the current sorting video captured for the sorting scenario.

[0057] S220. Extract optical flow from every two adjacent sorting video frames in the current sorting video and determine the extracted optical flow maps.

[0058] S230. For each optical flow map, convert the optical flow map into a corresponding HSV color image.

[0059] HSV color images can refer to images where each pixel is composed of H (Hue), S (Saturation), and V (Value).

[0060] Specifically, the optical flow information corresponding to each pixel in the optical flow map can be transformed to determine the hue (H), saturation (S), and brightness (V) values ​​corresponding to each pixel, thereby obtaining the transformed HSV color image.

[0061] For example, S230 may include: determining the optical flow amplitude and optical flow angle corresponding to each pixel based on the horizontal and vertical movement velocities corresponding to each pixel in the optical flow map; determining the hue H value and saturation S value corresponding to each pixel based on the optical flow amplitude and optical flow angle; and generating an HSV color image based on the hue H value, saturation S value, and preset brightness V value corresponding to each pixel.

[0062] The preset brightness V value can be a brightness V value set in advance based on business needs and the scenario. For example, the preset brightness V value can be set to 255 so that white can be used to represent stationary object areas in the sorting scenario. That is, areas in the sorting scenario that do not move are represented by white, which can effectively remove the interference of complex backgrounds.

[0063] Specifically, if the image size of the sorted video frames is h×w, where h is the image height and w is the image width, then the obtained optical flow graph size is h×w×2, representing the two-dimensional instantaneous velocity field. The optical flow graph at time t... t (i, j, 0) represents the horizontal movement speed corresponding to pixel (i, j); flow t (i, j, 1) represents the vertical motion velocity corresponding to pixel (i, j). This embodiment can determine the corresponding optical flow amplitude and optical flow angle based on the horizontal and vertical motion velocities corresponding to each pixel. For example, for the horizontal motion velocity flow corresponding to each pixel (i, j)... t (i, j, 0) and vertical velocity flow t The square root of (i, j, 1) is used as the optical flow amplitude magt(i, j) for pixel (i, j). The vertical motion velocity flow for each pixel (i, j) is then determined. t (i, j, 1) and the horizontal velocity flow t The ratio between (i, j, 0) and the corresponding arctangent angle are determined, and the product of this arctangent angle and 180 / π is the optical flow angle angt(i, j) corresponding to pixel (i, j). Using the optical flow amplitude and optical flow angle of each pixel, the hue H value and saturation S value of each pixel are determined. Therefore, based on the hue H value, saturation S value and preset brightness V value of each pixel, the converted HSV color image can be generated.

[0064] For example, determining the hue H value and saturation S value corresponding to each pixel based on the optical flow amplitude and optical flow angle may include: determining the optical flow amplitude corresponding to each pixel as the hue H value corresponding to the corresponding pixel, and determining the optical flow angle corresponding to each pixel as the saturation S value corresponding to the corresponding pixel.

[0065] Specifically, it can be done via hsv t [i, j, 0] = mag t (i, j), representing the optical flow amplitude mag for each pixel. t (i, j) is determined as the hue H value hsv corresponding to the corresponding pixel. t [i, j, 0]. Via hsv t[i, j, 1] = ang t (i, j), representing the optical flow angle mag for each pixel. t (i, j) is determined as the hue H value hsv corresponding to the corresponding pixel. t [i, j, 1]. This can be achieved via hsv. k [i, j, 2] = 255, set the brightness V value corresponding to each pixel to the preset brightness V value 255, so as to use white to represent and sort out the areas of objects that do not move in the image.

[0066] S240. Convert the HSV color image into the corresponding RGB color image, and use the RGB color image as the temporal motion feature map corresponding to the optical flow map.

[0067] Specifically, an HSV color image can be converted into an RGB color image (rgb) based on the conversion function f between HSV and RGB color images. t =f(hsv t This allows us to obtain a temporal motion feature map represented by an RGB color image. In the converted temporal motion feature map, i.e., the RGB color image, the darker the color of the object region, the faster the object moves. White represents the region of no moving object. This allows us to filter out a large number of stationary objects in the sorting scene by displaying them as white, showing only the dark-colored moving object regions, effectively removing background interference and further improving detection accuracy.

[0068] S250. Overlay the various temporal motion feature maps to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0069] Specifically, each temporal motion feature map is superimposed, i.e. Among them, w t RGB color image at time t t The weights can be set based on business needs, so that all the moving object regions with dark colors in each temporal motion feature map can be displayed in the same image, that is, the spatiotemporal static feature map F is obtained after superposition. This spatiotemporal static feature map can show the position change of the moving package at different times, that is, the movement trajectory of the package. Thus, the spatiotemporal static feature map integrates the features of the spatiotemporal domain.

[0070] S260. Input the spatiotemporal static feature map into the preset detection network model to detect the preset sorting behavior, and obtain the current detection result based on the output of the preset detection network model.

[0071] Specifically, because the background of the spatiotemporal static feature map F is clean, a lightweight pre-defined detection network model can be used to detect pre-defined sorting behaviors on the spatiotemporal static feature map F, improving detection efficiency while ensuring detection accuracy. By effectively removing interference from complex backgrounds, the model achieves higher generalization accuracy and detection efficiency in scenarios with frequently changing and complex environments such as logistics stations.

[0072] The technical solution of this embodiment first converts each optical flow map into a corresponding HSV color image, and then converts the HSV color image into a corresponding RGB color image. By using the HSV color image, the optical flow map can be accurately converted into an RGB color image, ensuring the accuracy of the temporal motion feature map, and thus ensuring the accuracy of detection.

[0073] The following are embodiments of the sorting behavior detection device provided in this invention. This device and the sorting behavior detection method in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the sorting behavior detection device, please refer to the embodiments of the sorting behavior detection method described above.

[0074] Example 3

[0075] Figure 3 This is a schematic diagram of a sorting behavior detection device provided in Embodiment 3 of the present invention. This embodiment can be applied to detect whether there is any damage to the packages during the sorting process by sorters. Figure 3 As shown, the device specifically includes: a current sorting video acquisition module 310, an optical flow map determination module 320, a spatiotemporal static feature map determination module 330, and a preset sorting behavior detection module 340.

[0076] The system includes: a current sorting video acquisition module 310, used to acquire the current sorting video captured for the sorting scene; an optical flow map determination module 320, used to extract optical flow from every two adjacent sorting video frames in the current sorting video and determine the extracted optical flow maps; a spatiotemporal static feature map determination module 330, used to transform and process each optical flow map to determine the spatiotemporal static feature map corresponding to the current sorting video; and a preset sorting behavior detection module 340, used to input the spatiotemporal static feature map into a preset detection network model to detect preset sorting behaviors and obtain the current detection result based on the output of the preset detection network model.

[0077] The technical solution of this embodiment extracts optical flow from every two adjacent sorting video frames in the captured current sorting video, and transforms each extracted optical flow map to determine a spatiotemporal static feature map containing temporal motion features and spatial static features. The spatiotemporal static feature map is then input into a preset detection network model to detect whether a preset sorting behavior exists. Based on the output of the preset detection network model, the current detection result is obtained, thereby realizing the automatic real-time detection of preset sorting behavior. This can promptly detect sorting behavior that damages packages by sorters without human intervention, reducing labor costs. Furthermore, the detection process considers both the temporal motion features and spatial static features of the package, thus ensuring the accuracy of sorting behavior detection.

[0078] Optionally, the spatiotemporal domain static feature map determination module 330 includes:

[0079] The temporal motion feature map determination unit is used to perform image transformation on each optical flow map and determine the temporal motion feature map corresponding to each optical flow map.

[0080] The spatiotemporal static feature map determination unit is used to overlay various temporal motion feature maps to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0081] Optionally, the temporal motion feature map determination unit is specifically used for:

[0082] For each optical flow map, the optical flow map is converted into a corresponding RGB color image, and the RGB color image is used as the temporal motion feature map corresponding to the optical flow map.

[0083] Optionally, the temporal motion feature map determination unit includes:

[0084] The optical flow diagram conversion subunit is used to convert the optical flow diagram into the corresponding HSV color image;

[0085] The HSV color image conversion subunit is used to convert HSV color images into corresponding RGB color images.

[0086] Optionally, the optical flow graph transformation subunit is specifically used for:

[0087] Based on the horizontal and vertical motion velocities of each pixel in the optical flow map, the optical flow amplitude and angle of each pixel are determined; based on the optical flow amplitude and angle, the hue (H) and saturation (S) values ​​of each pixel are determined; and based on the hue (H), saturation (S), and preset brightness (V) values ​​of each pixel, an HSV color image is generated.

[0088] Optionally, the optical flow graph transformation subunit is also specifically used for:

[0089] The optical flow amplitude corresponding to each pixel is determined as the hue H value corresponding to the corresponding pixel, and the optical flow angle corresponding to each pixel is determined as the saturation S value corresponding to the corresponding pixel.

[0090] Optionally, the optical flow graph determination module 320 is specifically used for:

[0091] Input each pair of adjacent sorting video frames in the current sorting video into the preset optical flow extraction network model, and obtain the optical flow map corresponding to each pair of adjacent sorting video frames based on the output of the preset optical flow extraction network model.

[0092] Optionally, the preset detection network model is a two-dimensional convolutional neural network model.

[0093] The sorting behavior detection device provided in the embodiments of the present invention can execute the sorting behavior detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the sorting behavior detection method.

[0094] It is worth noting that in the embodiments of the sorting behavior detection device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0095] Example 4

[0096] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Figure 4 A block diagram is shown of an exemplary electronic device 12 suitable for implementing embodiments of the present invention. Figure 4 The electronic device 12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0097] like Figure 4 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0098] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0099] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0100] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0101] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0102] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0103] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the steps of a sorting behavior detection method provided in this embodiment, the method including:

[0104] Acquire the current sorting video captured in the sorting scenario;

[0105] Optical flow is extracted from every two adjacent sorting video frames in the current sorting video to determine the extracted optical flow maps.

[0106] The optical flow maps are transformed to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0107] The spatiotemporal static feature map is input into the preset detection network model to detect the preset sorting behavior, and the current detection result is obtained based on the output of the preset detection network model.

[0108] Of course, those skilled in the art will understand that the processor can also implement the technical solution of the sorting behavior detection method provided in any embodiment of the present invention.

[0109] Example 5

[0110] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the sorting behavior detection method steps provided in any embodiment of the present invention. The method includes:

[0111] Acquire the current sorting video captured in the sorting scenario;

[0112] Optical flow is extracted from every two adjacent sorting video frames in the current sorting video to determine the extracted optical flow maps.

[0113] The optical flow maps are transformed to determine the spatiotemporal static feature map corresponding to the current sorting video.

[0114] The spatiotemporal static feature map is input into the preset detection network model to detect the preset sorting behavior, and the current detection result is obtained based on the output of the preset detection network model.

[0115] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0116] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0117] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0118] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0120] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for detecting sorting behavior, characterized in that, include: Acquire the current sorting video captured in the sorting scenario; Optical flow is extracted from every two adjacent sorting video frames in the current sorting video to determine each extracted optical flow map; For each optical flow map, the optical flow map is converted into a corresponding HSV color image; the HSV color image is converted into a corresponding RGB color image, and the RGB color image is used as the temporal motion feature map corresponding to the optical flow map; the temporal motion feature maps are superimposed to determine the spatiotemporal static feature map corresponding to the current sorting video. The spatiotemporal static feature map refers to an image containing temporal motion features and spatial static features. The temporal motion features are used to characterize the movement of packages in the sorting scene within a preset video length time period, and the spatial static features are used to characterize the appearance features of the moving packages in the sorting scene in space. The spatiotemporal static feature map is input into a preset detection network model to detect whether a preset sorting behavior exists, and the current detection result is obtained based on the output of the preset detection network model.

2. The method according to claim 1, characterized in that, The step of converting the optical flow map into a corresponding HSV color image includes: Based on the horizontal and vertical motion velocities corresponding to each pixel in the optical flow map, determine the optical flow amplitude and optical flow angle corresponding to each pixel. Based on the optical flow amplitude and the optical flow angle, determine the hue H value and saturation S value corresponding to each pixel; An HSV color image is generated based on the hue (H) value, saturation (S) value, and preset brightness (V) value corresponding to each pixel.

3. The method according to claim 2, characterized in that, The step of determining the hue (H) value and saturation (S) value corresponding to each pixel based on the optical flow amplitude and the optical flow angle includes: The optical flow amplitude corresponding to each pixel is determined as the hue H value corresponding to the corresponding pixel, and the optical flow angle corresponding to each pixel is determined as the saturation S value corresponding to the corresponding pixel.

4. The method according to claim 1, characterized in that, The step of extracting optical flow from every two adjacent sorting video frames in the current sorting video and determining the extracted optical flow maps includes: Each pair of adjacent sorting video frames in the current sorting video is input into a preset optical flow extraction network model, and an optical flow map corresponding to each pair of adjacent sorting video frames is obtained based on the output of the preset optical flow extraction network model.

5. The method according to any one of claims 1-4, characterized in that, The preset detection network model is a two-dimensional convolutional neural network model.

6. A device for detecting sorting behavior, characterized in that, include: The current sorting video acquisition module is used to acquire the current sorting video captured in the sorting scenario; The optical flow map determination module is used to extract optical flow from every two adjacent sorting video frames in the current sorting video and determine each extracted optical flow map. The spatiotemporal static feature map determination module is used to convert each optical flow map into a corresponding HSV color image; convert the HSV color image into a corresponding RGB color image, and use the RGB color image as the temporal motion feature map corresponding to the optical flow map; and perform superposition processing on each of the temporal motion feature maps to determine the spatiotemporal static feature map corresponding to the current sorting video. The spatiotemporal static feature map refers to an image containing temporal motion features and spatial static features. The temporal motion features are used to characterize the movement of packages in the sorting scene within a preset video length time period, and the spatial static features are used to characterize the appearance features of the moving packages in the sorting scene in space. The preset sorting behavior detection module is used to input the spatiotemporal domain static feature map into the preset detection network model to detect whether a preset sorting behavior exists, and to obtain the current detection result based on the output of the preset detection network model.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the sorting behavior detection method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the sorting behavior detection method as described in any one of claims 1-5.

Images