Theft detection method, apparatus, equipment, and media based on edge devices

Abstract

This application relates to a method, apparatus, device, and medium for identifying theft behavior based on edge devices. The method includes: acquiring surveillance video streams from a retail scene and extracting video frame images; detecting the video frame images using a target detection model, outputting human body detection boxes and their action categories, hand detection boxes and their holding states, and head detection boxes and their orientation categories; tracking the human body detection boxes, assigning a unique ID to each tracked human body to generate a human body ID number and a behavior analysis queue; matching the head detection boxes and hand detection boxes to the corresponding human body IDs; grouping and tracking the hand detection boxes matched to each human body ID to distinguish different hand groups; determining the hand attribution for picking and hiding actions; and determining whether theft behavior has occurred based on information recorded in the behavior analysis queue to obtain a target judgment result. This application improves the accuracy of theft behavior identification and reduces model complexity.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for identifying theft behavior based on edge devices. Background Technology

[0002] With the increasing security demands of the retail industry, video surveillance systems are playing an increasingly important role in theft detection. Currently, computer vision-based theft detection methods mainly rely on deep learning models, such as those based on human skeleton points, temporal modeling, or end-to-end video understanding. While these methods can achieve high accuracy in behavior recognition under specific conditions, their model structures are usually complex, consuming significant computational resources, making it difficult to achieve low-latency, high-concurrency real-time detection on edge devices with limited computing power (such as embedded video analytics terminals).

[0003] Specifically, existing methods often employ multi-stage or high-parameter models, resulting in limited inference speed and failing to meet the actual needs for real-time alerts in retail monitoring scenarios. Furthermore, the deployment and maintenance costs of complex models are high, hindering large-scale application across a wide range of retail stores. Therefore, how to reduce model complexity and improve real-time processing capabilities on edge devices while ensuring recognition accuracy has become a pressing technical challenge in this field. Summary of the Invention

[0004] The purpose of this application is to propose a theft behavior identification method, device, equipment, and medium based on edge devices, so as to improve identification accuracy, reduce model complexity, and enhance real-time processing capabilities on edge devices.

[0005] To address the aforementioned technical problems, embodiments of this application provide a theft behavior identification method based on edge devices, comprising:

[0006] Acquire surveillance video streams from retail scenarios and extract video frame images from the surveillance video streams;

[0007] The video frame image is detected by the target detection model, and the human body detection box and human action category, hand detection box and hand holding state, and head detection box and head orientation category are output.

[0008] The human detection box in the monitoring video stream is tracked, a unique ID is assigned to each tracked human body, a human body ID is generated, and a behavior analysis queue is initialized for each human body ID.

[0009] Match the head detection box and the hand detection box to the corresponding human ID;

[0010] The hand detection boxes matched with each human ID are grouped and tracked to distinguish different hand groups;

[0011] The hand affixation for grasping and concealing actions is determined based on the human body movement category, the hand group, and the head orientation category.

[0012] Based on the human action sequence, hand state sequence, and head orientation sequence recorded in the behavior analysis queue, it is determined whether theft has occurred, and the target judgment result is obtained.

[0013] To address the aforementioned technical problems, embodiments of this application provide a theft behavior identification device based on an edge device, comprising:

[0014] The video stream acquisition module is used to acquire the monitoring video stream of the retail scene and extract the video frame images from the monitoring video stream;

[0015] The image detection module is used to detect the video frame image through the target detection model and output the human body detection box and human action category, the hand detection box and hand holding state, and the head detection box and head orientation category.

[0016] The human body ID allocation module is used to track human body detection boxes in the monitoring video stream, assign a unique ID to each tracked human body, generate a human body ID, and initialize a behavior analysis queue for each human body ID.

[0017] The detection box matching module is used to match the head detection box and the hand detection box to the corresponding human body ID;

[0018] The hand grouping module is used to group and track the hand detection boxes matched with each human ID, so as to distinguish the hand detection boxes into different hand groups;

[0019] The hand attribution determination module is used to determine the hand attribution for grasping and concealing actions based on the human body action category, the hand group, and the head orientation category.

[0020] The judgment result generation module is used to determine whether theft has occurred based on the human action sequence, hand state sequence and head orientation sequence recorded in the behavior analysis queue, and to obtain the target judgment result.

[0021] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, so that the one or more processors implement the theft behavior identification method based on edge devices as described in any one of the above-mentioned methods.

[0022] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the theft behavior identification method based on edge devices described in any one of the above-mentioned methods.

[0023] This invention provides a method, apparatus, device, and medium for identifying theft behavior based on edge devices. The method includes: acquiring a surveillance video stream from a retail scene and extracting video frame images from the surveillance video stream; detecting the video frame images using a target detection model, outputting human body detection boxes and human action categories, hand detection boxes and hand-holding states, and head detection boxes and head orientation categories; tracking the human body detection boxes in the surveillance video stream, assigning a unique ID to each tracked human body, generating a human body ID, and initializing a behavior analysis queue for each human body ID; matching the head detection boxes and hand detection boxes to the corresponding human body IDs; grouping and tracking the hand detection boxes matched to each human body ID to distinguish different hand groups; determining the hand attribution for taking and hiding actions based on the human action category, the hand group, and the head orientation category; and determining whether a theft behavior has occurred based on the human action sequence, hand state sequence, and head orientation sequence recorded in the behavior analysis queue, obtaining a target determination result. This invention uses a combination of multi-source feature information, such as human body movements, hand states, and head orientation, to determine theft behavior, which helps improve recognition accuracy. At the same time, it only uses a target detection model to identify the human body, hand, and head regions, without the need for additional deep learning models, thus reducing model complexity and improving real-time processing capabilities on edge devices. Attached Figure Description

[0024] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the implementation of the theft behavior identification method based on edge devices provided in this application embodiment;

[0026] Figure 2 This is an overall flowchart of the theft behavior identification method based on edge devices provided in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the implementation of the first sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0028] Figure 4 This is a flowchart illustrating the implementation of the second sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0029] Figure 5 This is a flowchart illustrating the implementation of the third sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0030] Figure 6 This is a flowchart illustrating the implementation of the fourth sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0031] Figure 7 This is a flowchart illustrating the implementation of the fifth sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0032] Figure 8 This is a flowchart illustrating the implementation of the sixth sub-process in the theft behavior identification method based on edge devices provided in this application embodiment;

[0033] Figure 9 This is a schematic diagram of a theft behavior identification device based on an edge device provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0038] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the theft behavior identification method based on edge devices provided in this application embodiment is generally executed by a server, and correspondingly, the theft behavior identification device based on edge devices is generally configured in the server.

[0040] Please see Figure 1 and Figure 2 , Figure 1 This paper illustrates a specific implementation of a theft behavior identification method based on edge devices. Figure 2 This is an overall flowchart of the theft behavior identification method based on edge devices provided in the embodiments of this application.

[0041] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 Limited to the order of the processes shown, this method includes the following steps:

[0042] S1: Acquire the surveillance video stream of the retail scene and extract the video frame images from the surveillance video stream.

[0043] Specifically, the system acquires surveillance video streams from retail scenarios and extracts video frame images from these streams. In one specific embodiment, a video decoding library (such as FFmpeg or OpenCV) is used to decode the surveillance video stream, extracting video frame images at a preset frame rate (such as 24 frames per second) to generate a continuous image sequence.

[0044] S2: Detect the video frame image using the target detection model, and output the human body detection box and human action category, the hand detection box and hand holding state, and the head detection box and head orientation category.

[0045] Specifically, the object detection model can be based on structures such as YOLO or RT-DETR. This model is used to determine the bounding box and category of human actions, the bounding box and the hand holding state, and the bounding box and head orientation. In one specific embodiment, the object detection model outputs 17 preset prediction boxes, specifically categorized as follows:

[0046] Human Body Frames (11 types): Standing, Walking, Bending over, Squatting, Standing to Pick Up, Bending over to Pick Up, Squatting to Pick Up, Putting into Pants Pocket (Concealing), Putting into Bag (Concealing), Putting into Basket, Stuffing into Clothes (Concealing);

[0047] Hand frame (2 types): Holding an item, not holding an item;

[0048] Head frame (4 types): front, back, left, right.

[0049] S3: Track the human detection box in the monitoring video stream, assign a unique ID to each tracked human body, generate a human body ID, and initialize a behavior analysis queue for each human body ID.

[0050] Specifically, tracking algorithms (DeepSort, ByteTrack, etc.) are used to track human detection boxes. Each human ID initializes an independent behavior analysis queue to record human action state, hand state, hand group, and head state. The behavior analysis queue follows a "first-in, first-out" principle, and the queue length is calculated as M * N, where M is the time period and N is the sampling frequency. For example, if M is 3 seconds, assuming there are 24 frames per second, and N is 8 (i.e., sampling once every 3 frames), then the behavior analysis queue length is 24.

[0051] S4: Match the head detection box and the hand detection box to the corresponding human ID.

[0052] Specifically, the matching rules for head detection boxes must follow: each head detection box can only be assigned to one human ID, and each human ID can be bound to at most one head box. The matching rules for hand detection boxes must follow: each hand detection box can only be assigned to one human ID, and each human ID can be bound to at most two hand boxes.

[0053] Please see Figure 3 , Figure 3 A specific implementation of step S4 is shown below:

[0054] S41: For any of the head detection frames, calculate the head overlap rate and height ratio of the head detection frame;

[0055] S42: Calculate the pairing score based on the head overlap rate and the height ratio to obtain the first pairing score, and match the head detection box to the corresponding human ID based on the first pairing score.

[0056] Specifically, the head box matching rule is mainly based on the head ratio and height ratio, and the calculation formulas are as follows:

[0057] ;

[0058] ;

[0059] The specific pairing process is as follows: For each head detection box, iterate through all normally tracked human IDs, skipping paired head detection boxes; skipping human detection boxes with bound heads; calculate the head ratio (if the head ratio ≤ 0, discard the head detection box) and height_ratio between the human detection box and the head detection box; calculate the pairing score according to the formula a * head ratio + b * height_ratio (a and b are weight parameters, which can be set to a=0.5, b=0.5), and assign the head detection box to the human detection box with the higher score.

[0060] S43: For any of the hand detection frames, calculate the hand overlap rate and normalized distance of the hand detection frame;

[0061] S44: Calculate the pairing score based on the hand overlap rate and the normalized distance to obtain a second pairing score, and match the hand detection box to the corresponding human ID based on the second pairing score.

[0062] Specifically, the hand frame pairing rules are mainly based on the hand ratio and the normalized distance. The calculation formulas are as follows:

[0063] ;

[0064] ;

[0065] ;

[0066] Here, cdist represents the distance between the center point of the human detection bounding box and the center point of the hand bounding box. This represents the normalized distance, where W and H represent the width and height of the image, respectively. and This represents the x and y coordinates of the center point of the human detection frame. and This represents the horizontal and vertical coordinates of the center point of the hand detection frame.

[0067] The specific pairing process is as follows: For each hand detection box, iterate through all normally tracked human IDs, skipping already paired hand detection boxes; skipping human detection boxes that have been bound to two hands; calculate the hand ratio between the human detection box and the hand detection box (if the hand ratio ≤ 0, discard the hand detection box) and... According to the formula c * hand ratio + d * (1 - ... Calculate the pairing score (c and d are weight parameters, which can be set to c=0.7 and d=0.3), and assign the hand detection box to the human body detection box with the higher score.

[0068] S5: Group and track the hand detection boxes that match each human ID to distinguish different hand groups.

[0069] Specifically, hand grouping and tracking rules are primarily based on the distance between the hand frame center and the head orientation. It's important to note that the fundamental principle of hand grouping is comparing the Euclidean distance between the hand center points in the current frame and historical frames; the closest distance belongs to the same group. The historical frame search method involves searching in reverse order for the most recent frame that has recorded the hand center point coordinates, group information, and head orientation. The maximum value for the reverse search is K frames. The Euclidean distance between the hand center points is calculated as follows:

[0070] ;

[0071] in, and This represents the x and y coordinates of the center point of the hand frame in the current frame. and The horizontal and vertical coordinates represent the center point of the hand frame in the historical frame.

[0072] Please see Figure 4 , Figure 4 A specific implementation of step S5 is shown below:

[0073] S51: Based on the number of hand detection boxes that match the same human ID in the current frame of the monitoring video stream, and the hand group and position information in the most recent K frames obtained from the historical behavior analysis queue of the human ID;

[0074] S52: If the number of matching hand detection boxes in the current frame is 0, then count the number of consecutive frames in which the hand disappears. If the number of consecutive disappearing frames exceeds a preset threshold, then determine that the human body ID has no valid historical hand information.

[0075] Specifically, if the number of matched hand detection boxes in the current frame is 0, the number of times "hand disappears" is counted. If the number of consecutive occurrences of "hand disappears" exceeds K, it indicates that there is no hand information in the historical frames, and therefore it is determined that the human ID has no valid historical hand information. It should be noted that the preset threshold is set according to the actual situation and is not limited here.

[0076] S53: If the number of matched hand detection boxes in the current frame is 1 or 2, then the hand detection boxes are distinguished into different hand groups based on the number of hands in the past, the distance between the center points of the hand detection boxes in the current frame and the past frames, the degree of overlap, and the head orientation.

[0077] Please see Figure 5 , Figure 5 A specific implementation of step S53 is shown below:

[0078] S531: If the number of matched hand detection boxes in the current frame is 1, then determine the number of historical hands.

[0079] S532: If the number of historical hands is 2, and the current head orientation and the historical head orientation in the head orientation category are both facing left or right, and the overlap of the detection boxes of the two historical hands is higher than the preset overlap threshold, then the current hand is assigned to the group of the hand with the larger center ordinate in the detection boxes of the two historical hands; otherwise, the current hand is assigned to the group of the hand with the closest center distance to the detection boxes of the two historical hands.

[0080] Specifically, if the number of matched hand detection boxes in the current frame is 1 and the number of historical hands is 2, then it is necessary to determine the current head orientation category and the historical head orientation category. If both the current head orientation and the historical head orientation are facing left or right, and the IoU (Intersection over Union) of the detection boxes of the two historical hands is higher than a preset overlap threshold (e.g., 60%), then the current hand is assigned to the group of the hand with the larger center ordinate in the historical two-hand detection boxes; otherwise, the current hand is assigned to the group of the hand with the closest center distance to the historical two-hand detection boxes. The formula for calculating the overlap of the historical two-hand detection boxes is as follows:

[0081] ;

[0082] in, This represents the intersection area of ​​the two hand detection frames. This represents the union area of ​​two hand detection boxes. The IoU value ranges from [0,1]. The larger the value, the more severe the overlap between the two hand detection boxes.

[0083] S533: If the number of historical hands is 1, and the current head orientation and the historical head orientation are both front or back, and the current hand detection frame and the historical hand detection frame are located on different sides of the human body centerline, then the current hand is assigned to the group opposite to the historical hand; otherwise, the group of the historical hand is inherited.

[0084] Specifically, if the number of matched hand detection boxes in the current frame is 1 and the number of historical hands is 1, and the current head orientation and the historical head orientation are both front or back, and the current hand detection box and the historical hand detection box are located on different sides of the human body centerline, then the current hand is assigned to the group opposite to the historical hand; otherwise, the group of the historical hand is inherited.

[0085] S534: If the historical number of hands is 0, then initialize the current hand group based on the current head orientation and the current position of the hand detection box relative to the human body detection box.

[0086] Specifically, if the number of matched hand detection boxes in the current frame is 1 and the number of historical hand detection boxes is 0, the current hand group is initialized based on the current head orientation and the position of the current hand detection box relative to the human body detection box. The hands are then grouped into two groups: group 0 for left hands and group 1 for right hands. Table 1 below shows the grouping criteria when the number of matched hand detection boxes in the current frame is 1 and the number of historical hand detection boxes is 0:

[0087] Table 1

[0088]

[0089] S535: If the number of matched hand detection boxes in the current frame is 2, then the hand detection boxes are distinguished into different hand groups based on the historical number of hands, the distance from the center point, and the head orientation.

[0090] Please see Figure 6 , Figure 6 A specific implementation of step S535 is shown below:

[0091] S5351: If the number of matched hand detection boxes in the current frame is 2, then determine the number of historical hands.

[0092] S5352: If the number of hands in the history is 2, then according to the principle that the Euclidean distance between the center point of the current two-hand detection box and the center point of the historical two-hand detection box is the closest, the corresponding historical group is inherited for the current two-hand detection box respectively.

[0093] Specifically, if the number of matched hand detection boxes in the current frame is 2 and the number of historical hands is 2, the corresponding historical groups are inherited for the current two hand detection boxes according to the principle that the Euclidean distance between the center point of the current two hand detection boxes and the center point of the historical two hand detection boxes is the closest.

[0094] S5353: If the number of hands in the history is 1, then the current hand detection box that is closer to the center point of the historical hand detection box is assigned to the historical group, and the other hand detection box is assigned to a different group.

[0095] Specifically, if the number of matched hand detection boxes in the current frame is 2 and the number of historical hands is 1, the current hand detection box that is closer to the center point of the historical hand detection box is assigned to the historical group, and the other hand detection box is assigned to a different group.

[0096] S5354: If the number of hands in the history is 0, then initialize the two-hand detection box group according to the current head orientation and the coordinate position of the current two-hand detection box in the image.

[0097] Specifically, if the number of matched hand detection boxes in the current frame is 2 and the number of historical hand detection boxes is 1, the two hand detection box groups are initialized based on the head orientation and the coordinate positions of the current two hand detection boxes in the image. The hands are grouped into two groups: group 0 for the left hand and group 1 for the right hand. Table 2 below shows the grouping criteria when the number of matched hand detection boxes in the current frame is 2 and the number of historical hand detection boxes is 0:

[0098] Table 2

[0099]

[0100] S6: Determine the hand affixation for the grasping and concealing actions based on the human body movement category, the hand group, and the head orientation category.

[0101] Specifically, determining the hand affixed to a grasping action involves selecting the hand group for grasping based on a set distance scoring rule when the grasping action occurs, and simultaneously determining the hand affixed to grasp the item based on the head orientation category. Conversely, when a concealment action occurs, the hand affixed to conceal the item is determined based on the head orientation. Furthermore, hand grouping is merely a result of hand tracking and does not explicitly indicate left or right hand affixation, whereas hand affixation clearly carries the meaning of left or right hand.

[0102] Please see Figure 7 , Figure 7 A specific implementation of step S6 is shown below:

[0103] S61: When the human body action category is the grasping action, if only one hand is detected, the hand belonging to the grasping action is determined according to the head orientation category and the hand's proximity to the human body boundary.

[0104] Specifically, when the human action category is a grasping action, if only one hand is detected, the hand classification for the grasping action is determined based on the head's orientation and the hand's proximity to the body boundary. When only one hand is detected, the analysis is performed according to Table 3:

[0105] Table 3

[0106]

[0107] S62: If two hands are detected, calculate the overall extension distance and horizontal distance between each hand and the center of the human body, normalize and weight the scores to obtain the scores, and determine the hand to which the grasping action belongs based on the scores and the head orientation category.

[0108] Specifically, if two hands are detected, the overall extension distance (e.g., Euclidean distance) and horizontal distance (lateral extension amplitude) of each hand to the center of the body are calculated. Then, the overall extension distance and horizontal distance are normalized and assigned different weights to calculate the score. The calculation process of the overall extension distance is as follows:

[0109] ;

[0110] in, and The horizontal and vertical coordinates of the center point of the human body frame are represented. and Indicates the first The x and y coordinates of the center point of each hand frame This is the overall extension distance, which reflects the overall extension range of the hand relative to the center of the body.

[0111] The horizontal distance is calculated as follows:

[0112] ;

[0113] in, The horizontal distance, considering only horizontal displacement, measures the extent to which the hand extends to the left and right. To eliminate the influence of different resolutions on distance calculation, both distances (overall extension distance and horizontal distance) are normalized. The calculation formula is as follows:

[0114] ;

[0115] Where W and H represent the width and height of the image, respectively;

[0116] ;

[0117] in, As a weight for the overall extension range, The weights for the lateral extension amplitude, and satisfying It can be set to: , After calculating the scores for both hands, the hand group corresponding to the hand with the higher score is selected as the hand to perform the grasping action.

[0118] ;

[0119] Then, based on the analysis of the positions of the contestants in Table 4 below, the hand corresponding to the grabbing action can be determined:

[0120] Table 4

[0121]

[0122] S63: When the human body action category is the hiding action, the hand belonging to the hiding action is determined according to the head orientation category and the position of the visible hand.

[0123] Specifically, when the human action category is a hiding action, it is only necessary to analyze the situation where only one hand is present (while the other hand is "hiding"). The hand group is selected by default, and then the hand classification corresponding to the "hiding" action is analyzed according to Table 5:

[0124] Table 5

[0125]

[0126] S7: Based on the human action sequence, hand state sequence and head orientation sequence recorded in the behavior analysis queue, determine whether theft has occurred and obtain the target judgment result.

[0127] Specifically, the behavior analysis queue has recorded the aforementioned human action sequence, hand state sequence, and head orientation sequence. In this embodiment, based on the human action sequence, hand state sequence, and head orientation sequence recorded in the behavior analysis queue, it is determined whether a theft has occurred, and a target determination result is obtained. This target determination result is divided into two categories: theft has occurred and theft has not occurred.

[0128] Please see Figure 8 , Figure 8 A specific implementation of step S7 is shown below:

[0129] S71: The first condition is that the last action in the behavior analysis queue is the hiding action, and there are no cases where two hands are detected during hiding, the hiding hand and the taking hand belong to the same group, or the hiding hand and the taking hand belong to different groups.

[0130] Specifically, the first condition is that the last human action in the behavior analysis queue is a hidden action and the following invalid situations do not occur: 1. Two hands appear; 2. The group of the hand detected in the hidden action frame is the same as the group of the hand selected in the take action frame (if the take action and the hidden action appear in the same group, it means that the probability of not hiding is relatively high); 3. The hand belonging to the hidden action frame is different from the hand belonging to the take action frame (if the take action and the hidden action use hands from different sides, it means that the probability of not hiding is relatively high).

[0131] S72: Take the number of the taking actions in the behavior analysis queue within a preset range as the second condition.

[0132] Specifically, the second condition: The number of taking actions in the behavior analysis queue is within a preset range. If the number of taking actions ≥ t1: reduce the impact of misjudgment of the taking action frames; if the number of taking actions ≤ t2: when theft occurs, the taking time will not be too long. Here, t1 < t2, and t1 and t2 can be set to 2 and 25.

[0133] S73: Take the number of the states of holding an item reaching a first threshold value between the first taking action and the last hiding action as the third condition.

[0134] Specifically, the third condition: The number of the states of holding an item between the first taking action and the last hiding action ≥ h1 (the first threshold value). Extract a subsequence from the behavior analysis queue, mainly including all the hand states corresponding to the first taking action to the last hiding action. This subsequence is called the hand voting sequence. Count the number of items held in the hand voting sequence. After this number reaches a certain value h1 (which can be set to 3), it can enter the subsequent theft judgment rules.

[0135] S74: Vote based on the hand state sequences after taking and after non-taking to obtain a voting result, and take the determination of holding an item in the voting result as the fourth condition.

[0136] Specifically, the fourth condition: Vote based on the hand state sequences after taking and after non-taking to obtain a voting result, and take the determination of holding an item in the voting result. The specific voting process is as follows:

[0137] 1. First, extract two hand state lists from the hand voting sequence: one is the hand state list T containing the hand states after the taking action, and the other is the hand state list O containing the hand states after the remaining non-taking actions. The above lists do not record the states where the hand is not detected.

[0138] 2. Respectively count the number of continuously repeated hand states in the two lists. The hand state with the largest number is the voting result (such as the state with the most continuous occurrences is "holding an item").

[0139] 3. If the voting results of both list T and list O are "not holding an item", it is considered that the customer is not holding an item. If there are other combinations of voting results, it is considered that the customer is holding an item.

[0140] S75: After the first condition, the second condition, the third condition and the fourth condition are all satisfied, within the preset delayed observation window, it is determined whether the proportion of effective hiding actions exceeds the second threshold, so as to determine whether theft has occurred, and the target judgment result is obtained.

[0141] Specifically, after the first, second, third, and fourth conditions are all met, a delayed observation window of duration P seconds needs to be set. P and L can be set to 3 seconds and 70%, respectively. The effective concealment judgment condition is:

[0142] 1. No two hands were visible during the concealment.

[0143] 2. The groups of hands that appear when the item is taken are different from the groups of hands that appear when the item is hidden;

[0144] 3. The hand that takes the food and the hand that hides it belong to the same person (meaning that the same hand is being used).

[0145] The length of the delayed observation window is fixed. For example, if 8 frames are taken per second (24 frames per second) and the duration is 3 seconds, then the length of the delayed observation window is 24.

[0146] In this embodiment, a target detection model is used to detect the video frame image, outputting human body detection boxes and human action categories, hand detection boxes and hand holding status, and head detection boxes and head orientation categories. The human body detection boxes in the monitoring video stream are tracked, a unique ID is assigned to each tracked human body, a human body ID is generated, and a behavior analysis queue is initialized for each human body ID. The head detection boxes and hand detection boxes are matched to the corresponding human body IDs. The hand detection boxes matched to each human body ID are grouped and tracked to distinguish different hand groups. Based on the human action category, the hand group, and the head orientation category, the hand attribution for taking and hiding actions is determined. Based on the human action sequence, hand state sequence, and head orientation sequence recorded in the behavior analysis queue, it is determined whether a theft has occurred, resulting in a target determination result. This invention uses a combination of multi-source feature information, such as human body movements, hand states, and head orientation, to determine theft behavior, which helps improve recognition accuracy. At the same time, it only uses a target detection model to identify the human body, hand, and head regions, without the need for additional deep learning models, thus reducing model complexity and improving real-time processing capabilities on edge devices.

[0147] While maintaining the accuracy of theft identification, the embodiments of this application significantly reduce model complexity and computational cost, and improve real-time detection performance on edge devices or ordinary video analysis terminals; and by comprehensively utilizing multi-source feature information such as human body movements, hand states and head orientation, it enhances the ability and stability of theft behavior discrimination, overcoming the shortcomings of existing methods such as coarse identification granularity, poor real-time performance and difficult deployment.

[0148] Please refer to Figure 9 As a response to the above Figure 1 The implementation of the method shown in this application provides an embodiment of a theft behavior recognition device based on an edge device. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0149] like Figure 9 As shown, the theft behavior recognition device based on edge devices in this embodiment includes: a video stream acquisition module 81, an image detection module 82, a human body ID allocation module 83, a detection box matching module 84, a hand group differentiation module 85, a hand attribution determination module 86, and a judgment result generation module 87, wherein:

[0150] The video stream acquisition module 81 is used to acquire the monitoring video stream of the retail scene and extract the video frame images from the monitoring video stream;

[0151] The image detection module 82 is used to detect the video frame image through the target detection model and output the human body detection box and human action category, the hand detection box and hand holding state, and the head detection box and head orientation category.

[0152] The human body ID allocation module 83 is used to track human body detection boxes in the monitoring video stream, assign a unique ID to each tracked human body, generate a human body ID, and initialize a behavior analysis queue for each human body ID.

[0153] The detection box matching module 84 is used to match the head detection box and the hand detection box to the corresponding human body ID;

[0154] The hand grouping module 85 is used to group and track the hand detection boxes matched with each human ID, so as to distinguish the hand detection boxes into different hand groups;

[0155] The hand attribution determination module 86 is used to determine the hand attribution for grasping and concealing actions based on the human body action category, the hand group, and the head orientation category.

[0156] The judgment result generation module 87 is used to determine whether theft has occurred based on the human action sequence, hand state sequence and head orientation sequence recorded in the behavior analysis queue, and to obtain the target judgment result.

[0157] Furthermore, the detection box matching module 84 includes:

[0158] The head overlap rate calculation unit is used to calculate the head overlap rate and height ratio of any of the head detection frames.

[0159] The first pairing score calculation unit is used to calculate a pairing score based on the head overlap rate and the height ratio, obtain a first pairing score, and match the head detection box to the corresponding human ID based on the first pairing score;

[0160] The hand overlap rate calculation unit is used to calculate the hand overlap rate and normalized distance of any of the hand detection frames.

[0161] The second pairing score calculation unit is used to calculate the pairing score based on the hand overlap rate and the normalized distance, obtain the second pairing score, and match the hand detection box to the corresponding human ID based on the second pairing score.

[0162] Furthermore, the hand group differentiation module 85 includes:

[0163] The location information acquisition unit is used to determine the number of hand detection boxes that match the same human ID in the current frame of the monitoring video stream, and the hand group and location information in the most recent K frames obtained from the historical behavior analysis queue of the human ID.

[0164] The first allocation unit is used to count the number of consecutively disappearing hands if the number of matching hand detection boxes in the current frame is 0, and to determine that the human body ID has no valid historical hand information if the number of consecutively disappearing hands exceeds a preset threshold.

[0165] The second allocation unit is used to, if the number of matched hand detection boxes in the current frame is 1 or 2, distinguish the hand detection boxes into different hand groups based on the number of historical hands, the distance between the center points of the hand detection boxes in the current frame and the historical frames, the degree of overlap, and the head orientation.

[0166] Furthermore, the second allocation unit includes:

[0167] The first historical hand count subunit is used to determine the historical hand count if the number of matched hand detection boxes in the current frame is 1.

[0168] The first allocation subunit is configured to, if the number of historical hands is 2, and the current head orientation and the historical head orientation in the head orientation category are both facing left or right, and the overlap of the detection boxes of the two historical hands is higher than a preset overlap threshold, then assign the current hand to the group of the hand with the larger center ordinate in the detection boxes of the two historical hands; otherwise, assign the current hand to the group of the hand with the closest center distance to the detection boxes of the two historical hands.

[0169] The second allocation unit is used to distinguish different hand groups based on the historical number of hands, the distance between the center points of the hand detection boxes in the current frame and the historical frames, the degree of overlap, and the head orientation if the number of matched hand detection boxes in the current frame is 1 or 2.

[0170] Furthermore, the second allocation unit includes:

[0171] The first historical hand count subunit is used to determine the historical hand count if the number of matched hand detection boxes in the current frame is 1.

[0172] The first allocation subunit is configured to, if the number of historical hands is 2, and both the current head orientation and the historical head orientation in the head orientation category are facing left or right, and the overlap of the detection frames of the two historical hands is higher than a preset overlap threshold, then assign the current hand to the group of the hand with the larger center ordinate in the detection frames of the two historical hands; otherwise, assign the current hand to the group of the hand with the closest center distance to the detection frames of the two historical hands.

[0173] The second allocation subunit is configured to assign the current hand to a group opposite to the historical hand if the number of historical hands is 1, the current head orientation and the historical head orientation are both front or back, and the current hand detection frame and the historical hand detection frame are located on different sides of the human body centerline; otherwise, it inherits the group of the historical hand.

[0174] The third allocation subunit is used to initialize the current hand group based on the current head orientation and the current position of the hand detection frame relative to the human body detection frame if the historical hand count is 0.

[0175] The third allocation unit is used to distinguish different hand groups based on the historical number of hands, the center point distance, and the head orientation if the number of matched hand detection boxes in the current frame is 2.

[0176] Furthermore, the third allocation unit includes:

[0177] The second historical hand count subunit is used to determine the historical hand count if the number of matched hand detection boxes in the current frame is 2.

[0178] The fourth allocation subunit is used to, if the number of hands in the history is 2, inherit the corresponding historical group for the current two-hand detection frame according to the principle of the closest Euclidean distance between the center point of the current two-hand detection frame and the center point of the historical two-hand detection frame;

[0179] The fifth allocation subunit is used to allocate the current hand detection frame that is closer to the center point of the historical hand detection frame to the historical group if the number of historical hands is 1, and allocate the other hand detection frame to a different group.

[0180] The sixth allocation subunit is used to initialize the two-hand detection box group according to the current head orientation and the coordinate position of the current two-hand detection box in the image if the number of hands in the history is 0.

[0181] Furthermore, the hand attribution determination module 86 includes:

[0182] The first hand attribution unit is used to determine the hand attribution of the grasping action if only one hand is detected when the human body action category is the grasping action. This is based on the head orientation category and the hand's proximity to the human body boundary.

[0183] The second hand attribution unit is used to calculate the overall extension distance and horizontal distance between each hand and the center of the human body if two hands are detected, perform normalization and weighted scoring to obtain a scoring result, and determine the hand attribution of the grasping action based on the scoring result and the head orientation category.

[0184] The third hand attribution unit is used to determine the hand attribution of the concealment action based on the head orientation category and the position of the visible hand when the human body action category is the concealment action.

[0185] Furthermore, the judgment result generation module 87 includes:

[0186] The first condition generation unit is used to take the last action in the behavior analysis queue as the hiding action, and the following situations do not occur: two hands are detected during hiding, the hiding hand and the taking hand belong to the same group, or the hiding hand and the taking hand belong to different groups.

[0187] The second condition generation unit is used to take the number of the taking actions in the behavior analysis queue as a second condition if it is within a preset range.

[0188] The third condition generation unit is used to take the number of handheld items between the first take-up action and the last hide-up action as a third condition;

[0189] The fourth condition generation unit is used to vote based on the hand state sequence after taking the item and after not taking the item, obtain the voting result, and use the hand-held item determined in the voting result as the fourth condition.

[0190] The target judgment result generation unit is used to determine whether the proportion of effective hiding actions exceeds a second threshold within a preset delayed observation window after the first condition, the second condition, the third condition, and the fourth condition are all satisfied, so as to determine whether theft has occurred and obtain the target judgment result.

[0191] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0192] Computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 9 with three components—memory 91, processor 92, and network interface 93—is shown. It should be understood that implementing all shown components is not required; more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0193] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0194] The memory 91 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 91 may also include both internal storage units and external storage devices of the computer device 9. In this embodiment, the memory 91 is typically used to store the operating system and various application software installed on the computer device 9, such as program code for a theft behavior identification method based on edge devices. In addition, the memory 91 may also be used to temporarily store various types of data that have been output or will be output.

[0195] In some embodiments, processor 92 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 92 is typically used to control the overall operation of the computer device 9. In this embodiment, processor 92 is used to run program code stored in memory 91 or process data, for example, to run the program code of the aforementioned edge device-based theft behavior recognition method to implement various embodiments of the edge device-based theft behavior recognition method.

[0196] The network interface 93 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0197] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described edge device-based theft behavior identification method.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0199] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. An edge device based theft behavior identification method, characterized in that, The method comprises: acquiring a monitoring video stream of a retail scene and extracting video frame images in the monitoring video stream; detecting the video frame images by a target detection model to output a human body detection frame, a human body action category, a hand detection frame, a hand holding object state, a head detection frame and a head orientation category; tracking the human body detection frame in the monitoring video stream, assigning a unique ID to each tracked human body, generating a human body ID and initializing an action analysis queue for each human body ID; matching the head detection frame and the hand detection frame to a corresponding human body ID; grouping and tracking each hand detection frame matched to the human body ID to distinguish the hand detection frames into different hand groups, wherein the hand groups are used to identify different hand tracking identifiers under the same human body ID; determining hand attribution of a taking action and a hiding action based on the human body action category, the hand group and the head orientation category, wherein the hand attribution is used to indicate whether the hand performing the taking or hiding action is a left hand or a right hand; judging whether a theft behavior occurs based on a human body action sequence, a hand state sequence and a head orientation sequence recorded in the action analysis queue to obtain a target judgment result; the judgment whether the theft behavior occurs based on the human body action sequence, the hand state sequence and the head orientation sequence recorded in the action analysis queue to obtain the target judgment result comprises: taking the last action in the action analysis queue as the hiding action and the following conditions as a first condition: the hiding hand and the taking hand are not detected at the same time, the hiding hand and the taking hand belong to the same group or the hiding hand and the taking hand have different attributions; taking the number of the taking actions in the action analysis queue within a preset range as a second condition; taking the number of the hand holding object states between the first taking action and the last hiding action reaching a first threshold as a third condition; voting based on the hand state sequence after taking and the hand state sequence without taking to obtain a voting result and taking the hand holding object determined in the voting result as a fourth condition; after the first condition, the second condition, the third condition and the fourth condition are all satisfied, judging whether a proportion of valid hiding actions exceeds a second threshold within a preset delayed observation window to judge whether the theft behavior occurs to obtain the target judgment result. 2.The edge device based theft behavior identification method of claim 1, wherein, the matching the head detection frame and the hand detection frame to the corresponding human body ID comprises: for any head detection frame, calculating a head coincidence rate and a height proportion of the head detection frame; calculating a matching score based on the head coincidence rate and the height proportion to obtain a first matching score and matching the head detection frame to the corresponding human body ID based on the first matching score; for any hand detection frame, calculating a hand coincidence rate and a normalized distance of the hand detection frame; calculating a matching score based on the hand coincidence rate and the normalized distance to obtain a second matching score and matching the hand detection frame to the corresponding human body ID based on the second matching score. 3.The edge device based theft behavior identification method of claim 1, wherein, The grouping and tracking of each human body ID matched hand bounding box are performed to distinguish different hand groups, including: According to the number of hand bounding boxes matched to the same human body ID in the current frame in the monitoring video stream, and the hand group and position information in the last K frames obtained from the historical behavior analysis queue of the human body ID; If the number of hand bounding boxes matched in the current frame is 0, the number of frames in which the hand disappears continuously is counted, and if the number of continuous disappearance frames exceeds a preset threshold, it is determined that the human body ID has no valid historical hand information; If the number of hand bounding boxes matched in the current frame is 1 or 2, the hand bounding boxes are distinguished into different hand groups according to the historical number of hands, the center point distance between the hand bounding boxes in the current frame and the historical frame, the overlap degree, and the head orientation. 4.The edge device based theft behavior identification method of claim 3, wherein, If the number of hand bounding boxes matched in the current frame is 1 or 2, the hand bounding boxes are distinguished into different hand groups according to the historical number of hands, the center point distance between the hand bounding boxes in the current frame and the historical frame, the overlap degree, and the head orientation, including: If the number of hand bounding boxes matched in the current frame is 1, the historical number of hands is determined; If the historical number of hands is 2, and the current head orientation and the historical head orientation in the head orientation category are both left or right, and the overlap degree of the historical two hand bounding boxes is higher than a preset overlap threshold, the current hand is assigned to the group to which the hand with a larger center vertical coordinate in the historical two hand bounding boxes belongs, otherwise the current hand is assigned to the group to which the hand with the closest center distance in the historical two hand bounding boxes belongs; If the historical number of hands is 1, and the current head orientation and the historical head orientation are both front or back, and the current hand bounding box and the historical hand bounding box are located on different sides of the human body center line, the current hand is assigned to the group opposite to the historical hand, otherwise the group of the historical hand is inherited; If the historical number of hands is 0, the current hand group is initialized according to the current head orientation and the position of the current hand bounding box relative to the human body bounding box; If the number of hand bounding boxes matched in the current frame is 2, the hand bounding boxes are distinguished into different hand groups according to the historical number of hands, the center point distance, and the head orientation. 5.The edge device based theft behavior identification method of claim 4, wherein, If the number of hand bounding boxes matched in the current frame is 2, the hand bounding boxes are distinguished into different hand groups according to the historical number of hands, the center point distance, and the head orientation, including: If the number of hand bounding boxes matched in the current frame is 2, the historical number of hands is determined; If the historical number of hands is 2, the current two hand bounding boxes respectively inherit the corresponding historical groups according to the principle of the Euclidean distance between the center points of the current two hand bounding boxes and the historical two hand bounding boxes being the closest. If the number of the historical hands is 1, the current hand bounding box closer to the center point of the historical hand bounding box is assigned to the historical group, and the other hand bounding box is assigned to a different group; If the number of the historical hands is 0, the two hand bounding boxes are initialized into different groups according to the current head orientation and the coordinate positions of the two hand bounding boxes in the image.

6. The edge device-based theft behavior identification method according to any one of claims 1 to 5, characterized in that, the hand attribution of the taking action and the concealing action is determined based on the human action category, the hand group, and the head orientation category, including: when the human action category is the taking action, if only one hand is detected, the hand attribution of the taking action is determined according to the head orientation category and the situation of the hand close to the human body boundary; if two hands are detected, the overall stretch distance and the horizontal distance between each hand and the human body center are calculated respectively, normalized and weighted to obtain a score result, and the hand attribution of the taking action is determined based on the score result and the head orientation category; when the human action category is the concealing action, the hand attribution of the concealing action is determined according to the head orientation category and the position of the visible hand.

7. An edge device based theft behavior identification apparatus, comprising: including: a video stream acquisition module configured to acquire a monitoring video stream of a retail scene and extract video frame images in the monitoring video stream; an image detection module configured to detect the video frame images by a target detection model, and output human body bounding boxes and human action categories, hand bounding boxes and hand holding state, head bounding boxes and head orientation categories; a human body ID assignment module configured to track the human body bounding boxes in the monitoring video stream, assign a unique ID to each tracked human body, generate a human body ID, and initialize a behavior analysis queue for each human body ID; a bounding box matching module configured to match the head bounding boxes and the hand bounding boxes to corresponding human body IDs; a hand group distinguishing module configured to group and track the hand bounding boxes matched to each human body ID to distinguish the hand bounding boxes into different hand groups, wherein the hand group is used to identify different hand tracking identifiers under the same human body ID; a hand attribution determination module configured to determine the hand attribution of the taking action and the concealing action based on the human action category, the hand group, and the head orientation category, wherein the hand attribution is used to indicate whether the hand performing the taking or concealing action is the left hand or the right hand; a judgment result generation module configured to determine whether a theft behavior occurs based on a human action sequence, a hand state sequence, and a head orientation sequence recorded in the behavior analysis queue, and obtain a target judgment result; the judgment result generation module includes: a first condition generation unit configured to take the last action in the behavior analysis queue as the concealing action, and take the following conditions as a first condition: two hands are detected when concealing, the concealing hand and the taking hand are in the same group, or the concealing hand and the taking hand have different attributions. a second condition generation unit configured to take the number of the taking actions in the behavior analysis queue within a preset range as a second condition; a third condition generation unit configured to take the number of the hand-held object states between the first taking action and the last hiding action reaching a first threshold value as a third condition; a fourth condition generation unit configured to vote based on the hand state sequences before and after taking, obtain a voting result, and take the determination of holding an object in the voting result as a fourth condition; a target judgment result generation unit configured to, after the first condition, the second condition, the third condition, and the fourth condition are all met, judge whether the proportion of valid hiding actions exceeds a second threshold value within a preset delayed observation window, to judge whether a theft behavior occurs, and obtain the target judgment result.

8. A computer device, comprising: An edge device based theft behavior identification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the edge device based theft behavior identification method according to any one of claims 1 to 6.

Images