Method and device for recognizing baggage sorting behavior

Abstract

The present disclosure provides a method and device for recognizing baggage sorting behavior, which performs target detection on image frames in a target video to obtain at least one human body detection frame and at least one baggage detection frame, tracks the movement trajectories of the human body detection frame and the baggage detection frame, respectively, recognizes the dropped time of any baggage based on the tracked current movement trajectory for any baggage in the process of trajectory tracking, obtains movement information from the dropped time of any baggage to the current time, and recognizes the sorting behavior of any baggage based on the movement information.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is filed based on and claims priority from a Chinese patent application with application number 2022101688776 and filing date February 23, 2022. The entire contents of the Chinese patent application are incorporated herein by reference. The present disclosure relates to the field of computer vision technology, and more particularly to a method and apparatus for recognizing baggage sorting behavior. [Background technology]

[0002] With the development of e-commerce, online shopping, and the logistics industry, demand for courier services is also increasing. However, with the rapid development of the courier industry, when sorting packages, sorters are frequently throwing and discarding packages in an attempt to speed up sorting, damaging the items inside the packages and harming the interests of consumers and businesses. Summary of the Invention

[0003] Therefore, one object of the present disclosure is to provide a method for recognizing luggage sorting behavior, which performs target detection on image frames in a target video to obtain at least one human body detection frame and at least one luggage detection frame, tracks the movement trajectories of the human body detection frame and the luggage detection frame, respectively, and in the process of tracking the trajectories, recognizes the time at which any of the luggage is dropped based on the tracked current movement trajectory for any of the luggage, obtains movement information for any of the luggage from the time at which it is dropped to the present time, and recognizes the sorting behavior of any of the luggage based on the movement information.

[0004] The present disclosure, when recognizing luggage sorting behavior, can effectively eliminate erroneous judgments caused by interference from people alone, luggage alone, people moving together with luggage, or irrelevant information such as the background, by determining the interaction relationship between the human body and luggage. It can also recognize the sorter's body and determine the sorting force level, making the judgment more accurate.

[0005] A second object of the present disclosure is to provide a device for recognizing baggage sorting behavior. A third object of the present disclosure is to provide an electronic device. A fourth object of the present disclosure is to provide a non-transitory computer-readable storage medium. The fifth object of the present disclosure is to M It is to provide 。

[0006] In order to achieve the above-mentioned object, an embodiment of a first aspect of the present disclosure provides a method for recognizing luggage sorting behavior, including: performing target detection on image frames in a target video to obtain at least one human body detection frame and at least one luggage detection frame; tracking the movement trajectories of the human body detection frame and the luggage detection frame, respectively; during the trajectory tracking process, recognizing the time at which any of the luggage is dropped based on the tracked current movement trajectory for any of the luggage; obtaining movement information for any of the luggage from the time at which it is dropped to the present time; and recognizing the sorting behavior of any of the luggage based on the movement information.

[0007] The present disclosure, when recognizing luggage sorting behavior, can effectively eliminate erroneous judgments caused by interference from people alone, luggage alone, people moving together with luggage, or irrelevant information such as the background, by determining the interaction relationship between the human body and luggage. It can also recognize the sorter's body and determine the sorting force level, making the judgment more accurate.

[0008] In order to achieve the above object, an embodiment of the second aspect of the present disclosure provides a luggage sorting behavior recognition device including: a first acquisition module that performs target detection on image frames in a target video and acquires at least one human body detection frame and at least one luggage detection frame; a trajectory tracking module that tracks the movement trajectories of the human body detection frame and the luggage detection frame, respectively; a second acquisition module that, during the trajectory tracking process, recognizes the time at which any of the luggage is dropped based on the tracked current movement trajectory for the luggage; and a behavior recognition module that acquires movement information of any of the luggage from the time at which it is dropped to the present time, and recognizes the sorting behavior of any of the luggage based on the movement information.

[0009] The luggage sorting behavior recognition device disclosed herein can effectively eliminate erroneous judgments caused by interference from people alone, luggage alone, people moving together with luggage, or irrelevant information such as the background, by determining the interaction relationship between the human body and luggage when recognizing luggage sorting behavior, and can also recognize the sorter's body and determine the sorting ability level, resulting in more accurate judgments.

[0010] To achieve the above object, an embodiment of a third aspect of the present disclosure provides an electronic device, the electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to perform the method for recognizing luggage sorting behavior described in the embodiment of the first aspect of the present disclosure.

[0011] To achieve the above object, an embodiment of a fourth aspect of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions for realizing the method for recognizing luggage sorting behavior described in the embodiment of the first aspect of the present disclosure.

[0012] To achieve the above object, the fifth embodiment of the present disclosure provides a computer program M Provide ,before When the computer program is executed by a processor, the computer program realizes the baggage sorting behavior recognition method according to the first aspect of the present disclosure. 。 [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of a method for recognizing baggage sorting behavior according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of the trajectory of a single person sorted baggage in one embodiment of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram of recognizing the time when any package is dropped in one embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram of recognizing the sorting behavior of any parcel in one embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram showing tracking of the movement trajectories of the human body detection frame and the baggage detection frame according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of a method for recognizing baggage sorting behavior according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram of a baggage sorting behavior recognition device according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, the embodiments of the present disclosure will be described in detail. Examples of the embodiments are shown in the drawings, in which the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are merely illustrative and are intended to help interpret the present disclosure, but should not be understood as limitations on the present disclosure.

[0015] FIG. 1 is an exemplary embodiment of a method for recognizing luggage sorting behavior according to the present disclosure. As shown in FIG. 1, the method for recognizing luggage sorting behavior includes the following steps S101-S104.

[0016] S101: Perform target detection on image frames in a target video to obtain at least one human body detection frame and at least one baggage detection frame.

[0017] A video to be analyzed for baggage sorting behavior is defined as a target video, where the target video may be a video of sorters sorting bags captured in real time or a locally stored or received video of sorters sorting bags. The target video is decoded to extract frames, image frames corresponding to the target video at multiple different time points are obtained, target detection is performed on all image frames corresponding to the target video, and a human body detection frame, denoted as Pbox(t1), corresponding to the sorter in each image frame and a baggage detection frame, denoted as Pbox(t2), corresponding to the sorted baggage, are obtained, where t1 and t2 are frame indexes of the image frame corresponding to the human body detection frame or the baggage detection frame, and each detection frame includes coordinate information of the detection frame in the image frame.

[0018] In some embodiments, the target detection algorithm may use an algorithm such as a Feature Pyramid Network (FPN), a Convolutional Neural Network (CNN), etc. Here, each image frame may have one or more human body detection frames and one or more baggage detection frames, and if there are multiple sorters in an image frame, there will be multiple human body detection frames in the image frame, and if there are multiple sorted bags in the image frame, there will be multiple baggage detection frames in the image frame.

[0019] S102: The movement trajectories of the human body detection frame and the baggage detection frame are tracked.

[0020] Label information is assigned to the human body detection frame and baggage detection frame in each acquired frame image, and the movement trajectories of the human body detection frame and baggage detection frame are tracked based on the label information of the human body detection frame and baggage detection frame to obtain the human body movement trajectory and baggage movement trajectory. The human body movement trajectory is described as Ptrack(i,t1), where i indicates the label information of the human body, t1 indicates the tracking state of the human body in the latest image frame t1, and the baggage movement trajectory is described as Ptrack(j,t2), where j indicates the label information of the baggage, and t2 indicates the tracking state of the baggage in the latest image frame t2. In some embodiments, target tracking can employ tracking algorithms such as nearest neighbor matching and a multi-target tracking algorithm (Simple Online And Realtime Tracking, SORT).

[0021] S103: In the trajectory tracking process, the time when any of the packages is dropped is recognized based on the tracked current movement trajectory of any of the packages.

[0022] In the trajectory tracking process, the tracking trajectory of each package is traversed to determine the time when each package is dropped. Figure 2 is a schematic diagram of a human body sorting packages. As shown in Figure 2, taking one package as an example, the image frames corresponding to the package are traced back from the current frame, i.e., from the image frame at time t shown in Figure 2, the image frames corresponding to times t-1, t-2, t-3, t-4, t-5, and t-6 are traced back in order. The image frames are analyzed to determine whether the package has been dropped by a sorter. If it is determined that the package has been dropped by a sorter, it is recognized that an image frame showing the package being dropped has appeared, and the time corresponding to the image frame showing the package being dropped is the drop time. As shown in Figure 2, at time t-4, the distance between the package detection frame and the human body detection frame of the package in the image frame is smaller than the distance threshold. That is, at time t-4, the sorter drops the package, and time t4 is the time when the package is dropped.

[0023] S104: Acquire movement information of any of the packages from the time they are dropped to the present time, and recognize the sorting behavior of any of the packages based on the movement information.

[0024] In Figure 2, at times t-6, t-5, and t-4, a sorter is moving with a package. At this time, the package is not detached from the person's body, and none of the movements during this time are considered as candidates for sorting behavior. After time t4, the package begins to detach from the person's body, and is then thrown. To eliminate erroneous judgments due to the sorter's large movements or the delivery and receiving behavior between sorters, movement information for the package from the time it is thrown to the present, such as acceleration, movement distance, average speed, etc., is acquired. Because there is a significant difference in the acceleration, movement distance, and average speed of a package when it is left alone and when it is violently thrown, movement physical quantities such as acceleration, movement distance, and average speed can reflect the sorter's sorting ability. Based on the acquired movement information, sorter behavior parameters are acquired, and the sorter's sorting behavior can be recognized by comparing the sorting behavior parameters with parameters in existing sorting behavior standards. In some embodiments, when acquiring the sorting behavior parameters of the sorter, the sorting behavior of the sorter can be recognized by comparing the parameters based on each piece of movement information. Here, the parameters in the sorting behavior model may differ based on different types of luggage scenes.

[0025] An embodiment of the present disclosure provides a method for recognizing luggage sorting behavior, which performs target detection on image frames in a target video to obtain at least one human body detection frame and at least one luggage detection frame, tracks the movement trajectories of the human body detection frame and the luggage detection frame, and during the trajectory tracking process, recognizes the time when each luggage is dropped based on the tracked current movement trajectory of each luggage, obtains movement information from the time when each luggage is dropped to the current time, and recognizes the sorting behavior of each luggage based on the movement information. When recognizing luggage sorting behavior, the present disclosure determines the interaction relationship between the human body and the luggage, which effectively eliminates erroneous judgments caused by interference from people alone, luggage alone, people moving together with luggage, or irrelevant information such as the background, and also recognizes the sorter's body and determines the sorting ability level, resulting in more accurate judgments.

[0026] FIG. 3 is an exemplary embodiment of a method for recognizing luggage sorting behavior according to the present disclosure. As shown in FIG. 3, for any luggage, recognizing the time at which any luggage is dropped based on the tracked current movement trajectory includes the following steps S301-S302. S301: A target baggage movement trajectory corresponding to any baggage is compared with each human body movement trajectory to obtain a target distance between the target baggage detection frame corresponding to any baggage at the time of detection and each human body detection frame.

[0027] Based on the target luggage movement trajectory, the current image frame is traced back in order from latest to earliest, and the position information of the target luggage detection frame in each image frame is obtained as first position information, and the position information of the human body detection frame in the same image frame is obtained as second position information.

[0028] In one embodiment of the present application, if there is only one sorter, i.e., if there is only one human body detection frame in the image frame, the target distance is calculated as the distance between the first position information of the target baggage detection frame in each image frame and the second position information of the human body detection frame of that one human body at the corresponding time.

[0029] In another embodiment of the present application, when there are multiple sorters, multiple human body detection frames are present in the same image frame, and the human body detection frames are large. Therefore, when calculating the distance between the first position information of the target baggage detection frame and the second position information of all human body detection frames in the image frame at the corresponding time, if a sorter throws a bag and it goes over another sorter and is thrown far away, the sorter who went over is first calculated in the image frame. In this case, there will be a deviation. Therefore, when there are multiple sorters, human body keypoint detection is performed on the human body detection frames in the image frame to obtain the position information of the human body's hands, and the distance between the first position information of the target baggage detection frame and the position information of the hand of all humans in the image frame at the corresponding time is obtained as the target distance. In some embodiments, the human body keypoint detection algorithm can adopt a human body skeleton keypoint detection algorithm, etc.

[0030] S302: The time when the target distance is smaller than the distance threshold for the first time is determined as the time when any of the packages is to be dropped.

[0031] A threshold distance is set in advance, and the time corresponding to the image frame in which the target distance is smaller than the distance threshold for the first time is determined as the time when the package is dropped, and at the same time, the sorter corresponding to the package can also be determined.

[0032] An embodiment of the present disclosure can recognize the time when any luggage is thrown based on the tracked current movement trajectory, determine the interaction relationship between the luggage and the human body, determine which person threw the luggage and when, and the throwing trajectory, and more accurately recognize luggage sorting behavior.

[0033] FIG. 4 is an exemplary embodiment of a method for recognizing luggage sorting behavior according to the present disclosure. As shown in FIG. 4, recognizing the sorting behavior of any luggage based on movement information includes the following steps S401-S402.

[0034] S401: Based on the movement information, a sorting force parameter for any of the packages is generated.

[0035] Each piece of luggage is treated as a target luggage, and movement information for each target luggage from the time it is dropped to the present time is obtained. In some embodiments, the movement information may include the distance value from the time it is dropped to the present time for each target luggage, the maximum speed, the average speed, and the movement speed and acceleration for each target luggage at each time from the time it is dropped to the present time for each target luggage.

[0036] In an embodiment of the present application, the numerical value of each piece of movement information can be used as the movement parameter of that piece of movement information. For example, if the distance value of a certain piece of luggage from the time it is dropped to the current time is 3 meters, 3 meters can be used as the movement parameter of the distance value from the time it is dropped to the current time. S402: Determine the sorting force level of the sorting behavior of any one of the packages based on the sorting force parameters.

[0037] Different sorting behavior standards can be set according to different scenarios. For example, if the luggage is perishable goods, the items inside the luggage are relatively fragile, and the distance value from the time the luggage is dropped to the current time is used as a parameter, the parameter range can be set relatively narrow. For example, if the distance value from the time the luggage is dropped to the current time is less than 0.2 meters, it is normal, if it is 0.2-0.4 meters it is mild violence, if it is 0.4-0.7 meters it is normal violence, and if it is 0.7-1 meter it is severe violence. The distance value from the time the luggage is dropped to the current time is compared with the parameters of the sorting behavior standard to recognize the sorting behavior of the sorter corresponding to the luggage.

[0038] In addition, for example, if the luggage is clothing, it is difficult for it to be deformed by being thrown, so the luggage perishables Regarding the sorting behavior standards for products, clothing products In such cases, the sorting behavior standards will be set more strictly.

[0039] Similarly, sorting behavior standards may be established and compared with other movement information, such as acceleration, average speed, etc., to recognize the sorting behavior of a sorter corresponding to a target baggage.

[0040] An embodiment of the present disclosure can provide detailed guidance for sorters when sorting luggage by determining the sorting ability level of the sorting behavior of any luggage based on movement information, and can set different standards for the sorting behavior of sorters in different work situations, thereby improving the accuracy and versatility of recognizing sorting behavior.

[0041] FIG. 5 is an exemplary embodiment of a method for recognizing luggage sorting behavior according to the present disclosure. As shown in FIG. 5, tracking the movement trajectories of the human body detection frame and the luggage detection frame respectively includes the following steps S501-S502.

[0042] S501: Based on first landmark information of the human body detection frame, the human body detection frame is tracked, and a human body movement trajectory corresponding to the human body detection frame is generated.

[0043] Each human body detection frame in the image frame is assigned a piece of marker information, and the human body detection frame in each image frame is tracked based on the first marker information corresponding to each human body detection frame to generate a human body movement trajectory corresponding to the human body detection frame. Optionally, the target tracking can be performed using a tracking algorithm such as nearest neighbor matching or a multi-target tracking algorithm (Simple Online and Realtime Tracking, SORT).

[0044] S502: Track the baggage detection frame based on the second sign information of the baggage detection frame, and generate a baggage movement trajectory corresponding to the baggage detection frame.

[0045] Each baggage detection frame in the image frame is assigned one piece of sign information, and the baggage detection frame in each image frame is tracked based on the second sign information corresponding to each baggage detection frame as second sign information, and a baggage movement trajectory corresponding to the baggage detection frame is generated. In some embodiments, the target tracking can employ a tracking algorithm such as nearest neighbor matching or a multi-target tracking algorithm (Simple Online And Realtime Tracking, SORT).

[0046] The embodiments of the present disclosure can obtain the human body movement trajectory and the luggage movement trajectory by tracking the movement trajectory of the human body detection frame and the luggage detection frame, thereby realizing the interaction between the human body and the luggage and providing a basis for obtaining the time when the luggage is thrown.

[0047] FIG. 6 is an exemplary embodiment of a method for recognizing luggage sorting behavior according to the present disclosure. As shown in FIG. 6, the method for recognizing luggage sorting behavior includes the following steps S601-S602. S601: Perform target detection on image frames in a target video to obtain at least one human body detection frame and at least one baggage detection frame.

[0048] S602: Track the human body detection frame based on the first landmark information of the human body detection frame, and generate a human body movement trajectory corresponding to the human body detection frame.

[0049] S603: Track the baggage detection frame based on the second sign information of the baggage detection frame, and generate a baggage movement trajectory corresponding to the baggage detection frame.

[0050] The implementation method of steps S602-S603 can be the same as that in the above-described embodiments of the present disclosure, and the description thereof will be omitted here.

[0051] S604: A target baggage movement trajectory corresponding to any baggage is compared with each human body movement trajectory to obtain a target distance between the target baggage detection frame corresponding to any baggage at the time of detection and each human body detection frame.

[0052] S605: The time when the target distance is smaller than the distance threshold for the first time is determined as the time when any of the packages is to be dropped.

[0053] The implementation method of steps S604-S605 can be the same as that in the above-described embodiments of the present disclosure, and the description thereof will be omitted here.

[0054] S606: Based on the movement information, a sorting force parameter for any of the packages is generated.

[0055] S607: Determine the sorting force level of the sorting behavior of any one of the packages based on the sorting force parameters.

[0056] An embodiment of the present disclosure provides a method for recognizing luggage sorting behavior, which performs target detection on image frames in a target video to obtain at least one human body detection frame and at least one luggage detection frame, tracks the movement trajectories of the human body detection frame and the luggage detection frame, and during the trajectory tracking process, recognizes the time when each luggage is dropped based on the tracked current movement trajectory of each luggage, obtains movement information from the time when each luggage is dropped to the current time, and recognizes the sorting behavior of each luggage based on the movement information. When recognizing luggage sorting behavior, the present disclosure determines the interaction relationship between the human body and the luggage, which effectively eliminates erroneous judgments caused by interference from people alone, luggage alone, people moving together with luggage, or irrelevant information such as the background, and also recognizes the sorter's body and determines the sorting ability level, resulting in more accurate judgments.

[0057] FIG. 7 is a schematic diagram of a luggage sorting behavior recognition device according to the present disclosure. As shown in FIG. 7, the luggage sorting behavior recognition device 700 includes a first acquisition module 71, a trajectory tracking module 72, a second acquisition module 73, and a behavior recognition module 74.

[0058] The first acquisition module 71 performs target detection on image frames in the target video to acquire at least one human body detection frame and at least one baggage detection frame.

[0059] The trajectory tracking module 72 tracks the movement trajectories of the human body detection frame and the baggage detection frame.

[0060] In the process of tracking the trajectory, the second acquisition module 73 recognizes the time when any of the packages is dropped based on the tracked current movement trajectory of any of the packages.

[0061] The behavior recognition module 74 acquires movement information of any of the packages from the time they are dropped to the present time, and recognizes the sorting behavior of any of the packages based on the movement information.

[0062] In some embodiments, the second acquisition module 73 compares the target luggage movement trajectory corresponding to any of the luggage with each human body movement trajectory to obtain the target distance between the target luggage detection frame corresponding to any of the luggage and each human body detection frame at the time of detection, and determines the time when the target distance is smaller than the distance threshold, which is used to determine the time when any of the luggage is dropped.

[0063] In some embodiments, the second acquisition module 73 is used to acquire first position information of the target luggage detection frame in each image frame from the target luggage movement trajectory in order from latest to latest, acquire second position information of the human body detection frame in the same image frame from the human body movement trajectory, and acquire the target distance based on the first position information and the second position information at the corresponding time.

[0064] In some embodiments, the second acquisition module 73 is used to determine the time point at which the package is dropped as the time point corresponding to the image frame in which the target distance is first smaller than the distance threshold.

[0065] In some embodiments, the second acquisition module 73 is used to extract an image area marked with the second position information from an image frame corresponding to the second position information, perform human body keypoint detection on the image area, acquire human body hand position information, and acquire the distance from the first position information to the human body hand position information as a target distance.

[0066] In some embodiments, the behavior recognition module 74 is used to generate sorting force parameters for any of the packages based on the movement information and to determine a sorting force level for the sorting behavior of any of the packages based on the sorting force parameters.

[0067] In some embodiments, the movement information in the behavior recognition module 74 includes a distance value from the time when any of the luggage was dropped to the current time, a maximum speed, an average speed, and a movement speed and acceleration at each time from the time when any of the luggage was dropped to the current time.

[0068] In some embodiments, the trajectory tracking module 72 is used to track the human body detection frame based on first sign information of the human body detection frame to generate a human body movement trajectory corresponding to the human body detection frame, and to track the luggage detection frame based on second sign information of the luggage detection frame to generate a luggage movement trajectory corresponding to the luggage detection frame.

[0069] In order to realize the above embodiments, an embodiment of the present disclosure further provides an electronic device 800, which, as shown in FIG. 8, includes a processor 801 and a memory 802 communicatively connected to the processor, the memory 802 storing instructions executable by at least one processor, and the instructions are executed by the at least one processor 801 to realize the method for recognizing luggage sorting behavior shown in the above embodiments.

[0070] To realize the above embodiments, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause a computer to realize the baggage sorting behavior recognition method shown in the above embodiments.

[0071] To realize the above embodiment, the embodiment of the present disclosure ,Ko Computer Program M Furthermore, a computer program is provided that, when executed by a processor, realizes the baggage sorting behavior recognition method shown in the above embodiment.

[0072] To realize the above embodiments, an embodiment of the present disclosure provides a computer program, which includes computer program code, which, when executed by a computer, causes the computer to realize the baggage sorting behavior recognition method shown in the above embodiments.

[0073] The interpretations and explanations of the embodiments of the method for recognizing baggage sorting behavior can also be applied to the apparatus, electronic device, non-transitory computer-readable storage medium, computer program product, and computer program in the above embodiments, and will not be repeated here.

[0074] Additionally, the terms "first" and "second" are merely descriptive and cannot be understood to indicate or imply relative importance or the number of the indicated technical features. Thus, a feature qualified as "first" or "second" can explicitly or implicitly include one or more features. In the description of this disclosure, "plurality" means two or more than two, unless otherwise specified.

[0075] In the description herein, terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that a specific feature, structure, material, or characteristic described with reference to the embodiment or example is included in at least one embodiment or example of the present disclosure. In the present specification, schematic expressions for the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics may be combined in an appropriate manner in any one or more embodiments or examples. Furthermore, unless mutually inconsistent, a person skilled in the art may combine or combine different embodiments or examples and features of different embodiments or examples described herein.

[0076] Although the embodiments of the present disclosure have been shown and described above, the above embodiments are illustrative and should not be construed as limitations on the present disclosure. Those skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. performing target detection on image frames in the target video to obtain at least one human body detection frame and at least one baggage detection frame; tracking the movement trajectories of the human body detection frame and the baggage detection frame; In the process of tracking the trajectory, for any of the luggage, recognizing the time of dropping the luggage based on the tracked current movement trajectory; Acquiring movement information of any of the packages from the time of being dropped to the present time, and recognizing the sorting behavior of any of the packages based on the movement information; The current movement trajectory includes a human body movement trajectory corresponding to the human body detection frame and a baggage movement trajectory corresponding to the baggage detection frame, and recognizing a time point at which any of the baggage is dropped based on the tracked current movement trajectory for any of the baggage, Acquiring first position information of the target baggage detection frame in each image frame in order from latest to latest from the target baggage movement trajectory; acquiring second position information of the human body detection frame in the same image frame from the human body movement trajectory; If there is only one sorter, obtaining a target distance based on the first position information and the second position information at the corresponding time; extracting an image area marked with the second position information from an image frame corresponding to the second position information when there are multiple sorters; performing human body keypoint detection on the image region to obtain human hand position information; acquiring a distance from the first position information to the human hand position information as the target distance; determining a time when the target distance becomes smaller than a distance threshold for the first time as a time when any of the luggage is to be dropped. A method for recognizing luggage sorting behavior.

2. Determining the time when the target distance is first smaller than the distance threshold as the time when any of the luggage is to be dropped, and determining a time point corresponding to an image frame in which the target distance is first smaller than the distance threshold as a time point at which the luggage is dropped.

2. The method of claim 1 .

3. Recognizing the sorting behavior of any of the packages based on the movement information includes: generating a sorting force parameter for any of the packages based on the movement information, and determining a sorting force level for the sorting behavior of any of the packages based on the sorting force parameter.

2. The method of claim 1 .

4. The movement information includes a distance value from the time when any of the luggage was dropped to the current time, a maximum speed, an average speed, and a movement speed and acceleration of any of the luggage at each time from the time when it was dropped to the current time.

4. The method of claim 3.

5. Tracking the movement trajectories of the human body detection frame and the baggage detection frame, respectively, tracking the human body detection frame based on first marker information of the human body detection frame, and generating a human body movement trajectory corresponding to the human body detection frame; tracking the baggage detection frame based on second sign information of the baggage detection frame, and generating a baggage movement trajectory corresponding to the baggage detection frame.

2. The method of claim 1 .

6. a first acquisition module that performs target detection on image frames in the target video to acquire at least one human body detection frame and at least one baggage detection frame; a trajectory tracking module that tracks the movement trajectories of the human body detection frame and the baggage detection frame; a second acquisition module for recognizing a time when any of the packages is dropped based on the tracked current movement trajectory during the trajectory tracking process; a behavior recognition module that acquires movement information of any of the packages from the time of being dropped to the present time, and recognizes the sorting behavior of any of the packages based on the movement information; the current movement trajectory includes a human body movement trajectory corresponding to the human body detection frame and a baggage movement trajectory corresponding to the baggage detection frame; The second acquisition module further comprises: From the target luggage movement trajectory, first position information of the target luggage detection frame in each image frame is acquired in order from latest to latest; acquiring second position information of the human body detection frame in the same image frame from the human body movement trajectory; If there is only one sorter, a target distance is obtained based on the first position information and the second position information at the corresponding time; If there are multiple sorters, extracting an image area marked with the second position information from an image frame corresponding to the second position information; Perform human body keypoint detection on the image region to obtain human hand position information; acquiring a distance from the first position information to the human hand position information as the target distance; The time when the target distance becomes smaller than the distance threshold for the first time is determined as the time when any of the packages is to be dropped. A luggage sorting behavior recognition device characterized by the above.

7. The second acquisition module further comprises: The time point corresponding to the image frame in which the target distance is first smaller than the distance threshold is determined as the time point at which the baggage is dropped.

7. The device according to claim 6.

8. The behavior recognition module further comprises: A sorting force parameter for any of the packages is generated based on the movement information, and a sorting force level of the sorting behavior of any of the packages is determined based on the sorting force parameter.

7. The device according to claim 6.

9. The movement information includes a distance value from the time when any of the luggage was dropped to the current time, a maximum speed, an average speed, and a movement speed and acceleration of any of the luggage at each time from the time when it was dropped to the current time.

9. The device according to claim 8.

10. The trajectory tracking module further comprises: tracking the human body detection frame based on first marker information of the human body detection frame, and generating a human body movement trajectory corresponding to the human body detection frame; Tracking the baggage detection frame based on second sign information of the baggage detection frame and generating a baggage movement trajectory corresponding to the baggage detection frame.

7. The device according to claim 6.

11. An electronic device, at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform the method of any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions are for causing a computer to carry out the method according to any one of claims 1 to 5.

13. When executed by a processor, it implements the method according to any one of claims 1 to 5. Computer program.

Images