A method for identifying theft in unmanned convenience stores

Abstract

The present invention provides a method for identifying theft in unmanned convenience stores, including feature processing, model training and a theft identification algorithm. The theft identification algorithm reads video frames in a surveillance video stream, extracts original features from the video frames using a YOLOv8pose model, extracts action-related features through a feature processing module, and inputs the features into an action classification model to obtain a total action classification result. The current video frame is marked as a key frame based on the total action classification result. The classification model is used to determine whether there is an object in the hand of the key frame to obtain a classification result. The queue of frames with hands close to the body and the queue of frames with hands away from the body are updated according to preset conditions. The theft is identified by combining the queue length and the classification result. The invention can assist backstage staff in determining whether a customer has theft behavior, thereby reducing the risk of product theft and reducing store losses.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned convenience stores, and in particular to a method for identifying theft in unmanned convenience stores. Background Art

[0002] With the development of the internet and the real economy, unmanned convenience stores have become a new trend in the retail industry. Traditional retail faces challenges such as high labor costs and high rent pressures. This is especially true for 24-hour convenience stores, which require staff to operate even at night when traffic is sparse, leading to high labor costs. Unmanned convenience stores significantly reduce operating costs by reducing labor costs, leading to widespread adoption and acceptance in various sectors, including catering, lifestyle services, and books.

[0003] However, one of the main challenges facing unmanned convenience stores is theft prevention. Due to the lack of on-site staff supervision, some unscrupulous customers may damage the RFID tags on items and conceal them on their person in an attempt to avoid payment. While on-site security personnel can use in-store cameras to monitor and prevent theft, this monitoring model presents numerous problems. First, in-store cameras cannot detect and prevent theft in real time, requiring security personnel to monitor video feeds around the clock, which can easily lead to sensory fatigue and missed or false detections. Second, online monitoring is labor-intensive, requiring security personnel to continuously monitor multiple video streams, resulting in high labor costs.

[0004] Based on the above situation, there is an urgent need to provide a method for identifying theft in unmanned convenience stores to assist backstage staff in determining whether customers are stealing, thereby reminding customers to pay for unpaid items or put them back in place, reducing the burden on backstage staff. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for identifying theft in unmanned convenience stores, so as to solve the problems of missed detection and false detection that are easy to occur when manually monitoring theft in unmanned convenience stores, as well as the problems of large workload and high labor cost.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] A method for identifying theft in unmanned convenience stores, comprising:

[0008] Acquire multiple surveillance video streams of the surveillance scene, create and initialize a hand-approaching-body frame queue and a hand-away-from-body frame queue;

[0009] Obtain video frames from each video stream and perform detection. If a customer is detected, obtain the original features of the video frame and perform data preprocessing.

[0010] Build a window and aggregate multiple video frames within the window; extract the action behavior features related to the hand approaching the body and the hand moving away from the body in the window based on the original features of the video frames after data preprocessing;

[0011] After dimensionality reduction processing on the action behavior features within the window, the model is input into the action classification model to obtain the action classification results of the video frame. Based on the action classification results of the video frames at the same moment in all the monitoring video streams, the total action classification results are recorded as hand approaching the body, hand away from the body, or other actions. Based on the total action classification results, it is decided whether to use the video frame at that moment as a key frame.

[0012] For each key frame of the surveillance video stream, input it into the classification model of whether there is an object in the hand to obtain the classification result of whether there is an object in the customer's hand;

[0013] According to the preset conditions, update the frame queue of the hand approaching the body and the frame queue of the hand away from the body;

[0014] The length of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether the customer has an object in their hand, are used to comprehensively determine whether the customer is suspected of stealing at the current moment.

[0015] Record the time when customers are suspected of stealing and send reminder messages.

[0016] Furthermore, the obtaining of original features of the video frame and performing data preprocessing includes:

[0017] The YOLOv8-Pose model is used to extract the coordinates of specific key points of the customer's human skeleton, as well as the normalized prediction box height and center point coordinates; the specific key points include the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip.

[0018] Furthermore, the action behavior features in the window include:

[0019] The key points of the original features include the relative coordinates of the normalized prediction frame center as the origin, the movement speed of the key points in two adjacent frames, the angle between the key points and the horizontal plane, the distance statistics from the wrist to the normalized prediction frame center, the distance statistics from the elbow to the normalized prediction frame center, the distance statistics from the wrist to the hip, the distance statistics from the elbow to the hip, the average arm length ratio, and the average angle between the shoulder, elbow and wrist.

[0020] Furthermore, the action classification model is obtained by training a training set, including:

[0021] The training set consists of video frames of a single person engaging in theft and corresponding action type annotations, which are divided into three action categories: hands close to the body, hands away from the body, and other actions. The YOLOv8-Pose model is used to obtain the original features in the video frames and preprocess them. The action behavior features that are highly correlated with hands close to the body and hands away from the body are extracted. The action behavior features of each window and the action type annotation of each video frame in the window are combined into two table files.

[0022] Loading the feature data and the action type annotation data from the table file and dividing them into a training set and a validation set;

[0023] The principal component analysis method is used to reduce the dimension of the input feature data, and the action behavior characteristics and corresponding action type labels after dimension reduction in a window are used as a training sample;

[0024] A stacked classifier is constructed using a multi-layer perceptron and a gradient boosting-based tree model XGBoost classifier as base classifiers, and logistic regression as a meta-learner. First, each base classifier is trained separately using the input training samples to obtain an independent model for each base classifier. Then, a 5-fold cross-validation method is used to predict the input feature data to generate a secondary training dataset. Finally, the secondary training dataset is used to train the meta-learner to construct the final action classification model.

[0025] Furthermore, whether to use the video frame at the moment as a key frame is determined based on the overall action classification result, including:

[0026] If the action classification results of the current video frames of more than half of the video streams at the same moment are hands approaching the body or hands away from the body, the total action classification result at that moment is recorded as hands approaching the body or hands away from the body, and the current video frames of all video streams are recorded as key frames; otherwise, the total action classification result is recorded as other actions.

[0027] Furthermore, according to preset conditions, updating the hand approaching the body frame queue and the hand away from the body frame queue includes:

[0028] The elements in the hand-close-to-body frame queue and the hand-away-from-body frame queue are lists, including the total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points;

[0029] If the previous total action classification result is not hand approaching the body and the current total action classification result is hand approaching the body, then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand approaching the body frame queue;

[0030] If the previous and current total action classification results are both "hand close to body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand close to body frame queue;

[0031] If the previous and current total action classification results are both "hands away from the body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue;

[0032] If the previous total action classification result is hand approaching the body and the current total action classification result is not hand approaching the body, then determine whether the length of the hand approaching the body frame queue is greater than or equal to 5. If it is greater than or equal to 5, retain the elements in the hand approaching the body frame queue; if it is less than 5, clear the hand approaching the body frame queue;

[0033] If the previous total action classification result is not the hand away from the body, and the current total action classification result is the hand away from the body, then determine whether the length of the hand close to the body frame queue is greater than 0. If it is greater than 0, then merge the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue; otherwise, clear the hand away from the body frame queue.

[0034] Furthermore, the length of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether the customer has an object in their hand, are combined to comprehensively determine whether the customer is stealing at the current moment, including:

[0035] If the previous total action classification result is "hands away from the body" and the current total action classification result is not "hands away from the body," then determine whether the length of the "hands approaching the body" frame queue is greater than 0 and whether the length of the "hands away from the body" frame queue is greater than or equal to 5. If the conditions are met, then record that the customer's hand approaching the body action and the hand away from the body action are successfully paired, and read the elements in the hand approaching the body frame queue and the hand away from the body frame queue at this time.

[0036] If the hand-approaching-body action is successfully paired with the hand-away-from-body action, read the elements in the hand-approaching-body frame queue and the hand-away-from-body frame queue at this time, calculate the total movement distance of the left wrist and the right wrist in the hand-approaching-body frame queue and the hand-away-from-body frame queue, and return the wrist with the larger total movement distance and the classification result of whether there is an object in the hand; the total movement distance is calculated and accumulated by the coordinates of the wrist key points of two adjacent elements in the queue;

[0037] The classification result is calculated as the number of frames with objects in the hand and without objects in the hand; if the number of frames with objects in the hand in the hand-close-to-body frame queue is greater than 4 and the number of frames without objects in the hand in the hand-away-from-body frame queue is greater than 4, the customer is considered to have committed suspected theft, and the hand-close-to-body frame queue and the hand-away-from-body frame queue are cleared.

[0038] Compared with the prior art, the present invention has the following technical features:

[0039] The present invention uses the YOLOv8pose model to extract raw features from video frames in surveillance video streams. It then uses a feature processing module to extract action-related features, which are then input into an action classification model to obtain a total action classification result. The current video frame is marked as a keyframe based on the total action classification result. The classification model determines whether the keyframe contains an object and obtains a classification result. Based on preset conditions, the queues of frames showing hands approaching the body and hands away from the body are updated. The queue length and classification results are combined to identify theft. The present invention can assist backstage staff in determining whether customers are stealing, thereby reminding them to pay for unpaid items or return them to their original location, alleviating the burden on backstage staff and effectively resolving the issues of missed and false detections that can easily occur when manually monitoring theft in unmanned convenience stores, as well as the high workload and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a method in one embodiment of the present invention;

[0041] Figure 2 A human body key point sequence number map predicted by the YOLOv8 Pose model according to an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of an algorithm for updating a hand-close-to-body frame queue and a hand-away-from-body frame queue according to an embodiment of the present invention;

[0043] Figure 4 This is a flow chart of an algorithm for determining whether a customer has committed theft at the current moment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] Referring to the accompanying drawings, the present invention provides a method for identifying theft in an unmanned convenience store, comprising:

[0045] Step 1: Set up the image reading thread and the theft behavior recognition algorithm thread

[0046] The image reading thread continuously reads video frames from the surveillance video stream according to the predefined RTSP address of the surveillance video stream and stores these video frames in a list; sets up a queue for storing the received video frames, and puts the frame list into the queue for use by the theft behavior recognition algorithm thread, and notifies the theft behavior recognition algorithm thread to start processing the video frames.

[0047] The monitoring video stream comes from the monitoring site, such as a real-time video stream obtained from an unmanned convenience store. When the algorithm starts executing, a hand-close-to-body frame queue and a hand-away-from-body frame queue are created and initialized for subsequent behavior judgment.

[0048] In step 2, the theft behavior recognition algorithm thread obtains video frames from the queue, scales them to a fixed size, and inputs them into the YOLOv8pose model. If a customer is detected in the video frame, the original features of the video frame are obtained and data preprocessing is performed on the original features.

[0049] The obtaining of original features and performing data preprocessing on the original features include:

[0050] The original features are 17 normalized human skeleton key point coordinates extracted using the YOLOv8-Pose model. The key points include nose, eyes, ears, shoulders, elbows, wrists, knees, hips and ankles, as well as normalized prediction box height and center point coordinates. These original features can all be used for model training. However, after actual research and verification by the inventor, the specific key points of "left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip" can well reflect the customer's real movements, thereby expressing their true intentions. Therefore, during actual model training, the present invention utilizes these specific key points + normalized prediction box height and center point coordinates. Of course, other combined features can also be selected.

[0051] The data preprocessing steps are:

[0052] During the training phase: the customer's original features are merged into a summarized original feature text file, which includes action sequence numbers, action labels, and original feature data for subsequent action feature processing; the action label refers to the action classification, which includes three categories: close to the body, hand away from the body, and other actions; when constructing the training set, the original features corresponding to each frame of the image are labeled according to its actual action type; the action sequence is a section of action process corresponding to the action label, usually represented by the action sequence start frame number and the action sequence end frame number; generally, in a theft scene, there may be multiple hands close to the body, hands away from the body, and other actions. If the action of hand close to the body appears ten times, then the action of hand close to the body will have 10 action sequence numbers, and so on for other categories. Read the text file after merging features and retain only the coordinates of specific key points such as left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, as well as the normalized prediction box height and center point coordinates; if there are key points missing in the current video frame, fill in the position of the missing key points based on the position of the key points in the previous frame relative to the normalized prediction box center point.

[0053] In the prediction stage: directly read the original features, and retain the above key point coordinates and the normalized prediction box height and center point coordinates; if there are key points missing in the current video frame, the position of the missing key points is also filled according to the position of the key points in the previous frame relative to the normalized prediction box center point.

[0054] Step 3: Input the raw features after data preprocessing into the feature processing module to extract nine action behavior features that are highly correlated with the hand approaching the body and the hand moving away from the body. Window construction is performed, and action behavior features of multiple video frames are aggregated within each window. Specifically, the following steps are included:

[0055] During the action classification model training phase, the feature processing module aggregates the raw features of multiple preprocessed frames into a window with a window size of 10 frames. If the window size is not met, the module continues to read the raw features of the next video frame in the aggregated raw feature text file. If the window size is met, the module extracts the action behavior features based on the raw features of each video frame in the window after data preprocessing. The features of each window and the action type label of each frame in the window are combined into two tables for subsequent action classification model training.

[0056] In the action classification model prediction stage, the feature processing module aggregates the preprocessed raw features into a window with a window size of 10 frames. If the window size is not met, the raw features of the next video frame are read. If the window size is met, the action behavior features are extracted based on the raw features of each video frame in the window after data preprocessing. Principal component analysis (PCA) is used to reduce the feature dimension to 50 and then input it into the trained action classification model. By sliding the window frame by frame, an action classification result is output for each video frame.

[0057] The nine action behavior features within a window that are highly correlated with the hand approaching the body and the hand moving away from the body specifically include: the relative coordinates of the key point with the center point of the normalized prediction frame as the origin, the movement speed of the key point in two adjacent frames, the angle between the key point and the horizontal plane, the distance statistics from the wrist to the center point of the normalized prediction frame, the distance statistics from the elbow to the center point of the normalized prediction frame, the distance statistics from the wrist to the hip, the distance statistics from the elbow to the hip, the average arm length ratio, and the average angle between the shoulder, elbow and wrist.

[0058] In step 4, the action behavior features within a window are subjected to dimensionality reduction processing using principal component analysis (PCA) to reduce the feature dimension to 50 dimensions, and then input into the action classification model to obtain the action classification results of the video frame; based on the action classification results of the video frames at the same moment of all video streams, the total action classification results are recorded as hand approaching the body, hand away from the body, or other actions, and based on the total action classification results, it is decided whether to use the video frame at that moment as a key frame.

[0059] The action classification model is obtained by training the training set, specifically:

[0060] The training set consists of video frames of a single person stealing and corresponding action type annotations, which are divided into three action categories: hands close to the body, hands away from the body, and other actions. The raw features in the video frames are obtained and pre-processed using the YOLOv8-Pose model and input into the feature processing module;

[0061] Extracting the action behavior features that are highly correlated with the hand approaching the body and the hand moving away from the body through the feature processing module, combining the action behavior features of each window and the action type label of each video frame in the window into two table files;

[0062] Loading the feature data and the action type annotation data from the table file, and dividing the data into a training set and a validation set in a ratio of 7:3;

[0063] To improve the efficiency and effect of model training, principal component analysis (PCA) is used to reduce the dimensionality of the input feature data to 50 dimensions. The action behavior features and corresponding action type labels after dimensionality reduction in a window are used as a training sample.

[0064] A stacked classifier is constructed using a multi-layer perceptron (MLP) and a gradient boosting-based tree model XGBoost classifier as base classifiers, and logistic regression as a meta-learner. First, each base classifier is trained separately using the input training samples to obtain an independent model for each base classifier. Then, a 5-fold cross-validation method is used to predict the input feature data to generate a secondary training dataset. Finally, the secondary training dataset is used to train the meta-learner to construct the final action classification model.

[0065] The decision of whether to set the current video frame as a key frame specifically includes:

[0066] If the action classification result of the current video frames of more than half of the video streams at the same moment is hand approaching the body, the total action classification result at that moment is recorded as hand approaching the body, and the current video frames of all video streams are recorded as key frames; if the action classification result of the current video frames of more than half of the video streams at the same moment is hand away from the body, the total action classification result at that moment is recorded as hand away from the body, and the current video frames of all video streams are recorded as key frames; otherwise, the total action classification result is recorded as other actions.

[0067] Step 5. For each key frame of the monitoring video stream, a hand image of size 100*100 is cropped with the wrist joint point of the key frame as the center. After resizing, center cropping, tensor conversion and normalized data preprocessing, it is input into the classification model of whether the customer has an object in the hand to obtain the classification result of whether the customer has an object in the hand; if the behavior classification result of the left hand image cropped from the key frame of a video stream is that there is an object in the hand, then it is marked that the customer's left hand has an object in the key frame at the current moment; otherwise, it is recorded that there is no object in the customer's left hand in the key frame at the current moment; if the classification result of the right hand image cropped from the key frame of a video stream is that there is an object in the hand, then it is recorded that there is an object in the customer's right hand in the key frame at the current moment; otherwise, it is recorded that there is no object in the customer's right hand in the key frame at the current moment.

[0068] Among them, the classification model for whether there is an object in the hand is obtained by training the neural network based on a dataset constructed from pictures of theft behavior.

[0069] Step 6: Update the hand-close-to-body frame queue and the hand-away-from-body frame queue according to preset conditions, specifically including:

[0070] The elements in the hand-approaching-the-body frame queue and the hand-away-from-the-body frame queue are lists, including the overall action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points (used for subsequent calculation of the movement distance of the left wrist and the right wrist). When the theft behavior recognition algorithm thread runs, the hand-approaching-the-body frame queue and the hand-away-from-the-body frame queue are created and initialized.

[0071] If the previous total action classification result is not hand approaching the body and the current total action classification result is hand approaching the body, then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand approaching the body frame queue;

[0072] If the previous and current total action classification results are both "hand close to body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand close to body frame queue;

[0073] If the previous and current total action classification results are both "hands away from the body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue;

[0074] If the previous total action classification result is hand approaching the body and the current total action classification result is not hand approaching the body, then determine whether the length of the hand approaching the body frame queue is greater than or equal to 5. If it is greater than or equal to 5, retain the elements in the hand approaching the body frame queue; if it is less than 5, clear the hand approaching the body frame queue;

[0075] If the previous total action classification result is not the hand away from the body, and the current total action classification result is the hand away from the body, then determine whether the length of the hand close to the body frame queue is greater than 0. If it is greater than 0, then merge the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue; otherwise, clear the hand away from the body frame queue.

[0076] Step 7: Combine the lengths of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether the customer has an object in their hand, to comprehensively determine whether the customer is suspected of stealing at the current moment; specifically:

[0077] If the previous total action classification result is hand away from body and the current total action classification result is not hand away from body, then determine whether the length of the hand approaching body frame queue is greater than 0 and whether the length of the hand away from body frame queue is greater than or equal to 5; if the conditions are met, then record that the customer's hand first approaches the body and then moves away from the body, that is, the hand approaching body action and the hand away from body action are successfully paired, and read the elements in the hand approaching body frame queue and the hand away from body frame queue at this time; otherwise, continue to read the original features of the next frame;

[0078] If the hand approaching the body action is successfully paired with the hand moving away from the body action, the elements in the hand approaching the body frame queue and the hand moving away from the body frame queue are read, and the total movement distance of the left wrist and the right wrist in the hand approaching the body frame queue and the hand moving away from the body frame queue are calculated. The wrist with the larger total movement distance and the classification result of whether there is an object in its hand are returned; the total movement distance is calculated and accumulated by the coordinates of the wrist key points of two adjacent elements in the queue.

[0079] The classification result is calculated as the number of frames with objects in the hand and without objects in the hand; if the number of frames with objects in the hand in the hand-close-to-body frame queue is greater than 4 and the number of frames without objects in the hand in the hand-away-from-body frame queue is greater than 4, the customer is considered to have committed suspected theft, and the hand-close-to-body frame queue and the hand-away-from-body frame queue are cleared; otherwise, the original features of the next frame are read.

[0080] Step 8: If the customer is suspected of stealing at this time, the time node is recorded and a reminder is sent to the back-end staff; otherwise, the original features of the next frame are read; the back-end staff can review the surveillance video stream according to the recorded time node to check whether the customer has stolen.

[0081] Example:

[0082] 1. Feature Processing Module

[0083] During the training phase, the feature processing module aggregates the preprocessed raw features into a window with a window size of 10 frames. If the window size is not met, the module continues to read the raw features of the next frame in the summarized raw feature text file. If the window size is met, the module extracts nine features that are highly correlated with the hand being close to the body and the hand being far away from the body based on the raw features in a window. The features of each window and the action type of each frame in the window are labeled and synthesized into two tables for subsequent training of the action classification model.

[0084] In the model prediction stage, the feature processing module aggregates the preprocessed original features into a window with a window size of 10 frames. If the window size is not met, the original features of the next frame are read; if the window size is met, the features of the current window are subjected to dimensionality reduction using principal component analysis (PCA). After reducing the feature dimension to 50 dimensions, they are input into the action classification model. By sliding the window frame by frame, an action classification result is output for each frame.

[0085] The key point numbers of the human body predicted by the YOLOv8 Pose model are as follows Figure 2 As shown, the key points used by the feature processing module will be Figure 2 The key point numbers in the window are represented, and the features within a window include:

[0086] Feature 1: The relative coordinates of the key point with the center of the normalized prediction box as the origin.

[0087] For the key point coordinates of the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, the normalized prediction box center point is used as the origin (0,0) for translation. The translated key points are divided by the normalized prediction box height and further normalized. The processed results are stored in a list to ensure that the coordinates of all key points are relative to the normalized prediction box center point. Assume that i is the frame number in the window and j is the key point number. The normalized prediction box center point coordinates in each frame are The normalized prediction box height is h i , the relative coordinates with the center point of the normalized prediction box as the origin are:

[0088]

[0089]

[0090] feature1=(x′ j,1 ,y′ j,1 ),…,(x′ j,10 ,y′ j,10 ),

[0091] i=1,…,10,j∈5,6,7,8,9,10,11,12

[0092] Feature 2: The movement speed of key points in two adjacent frames.

[0093] Assuming i is the frame number in the window and j is the key point number, the movement speed of the key points in two adjacent frames is:

[0094] Δ(x j,i ,y j,i )=(x j,i+1 ,yj,i+1 )-(x j,i ,y j,i )

[0095] feature2=Δ(x j,i ,y j,i ),i=1,…,9,j∈5,6,7,8,9,10,11,12

[0096] Feature 3: Angle between the key point and the horizontal plane.

[0097] Assume that i is the frame number in the window, and the normalized prediction box center coordinates in each frame are Then in a window, the angle between the key point and the horizontal plane is:

[0098]

[0099]

[0100] i=1,…,10

[0101]

[0102] Feature 4: Distance statistics from the wrist to the center of the normalized prediction box.

[0103] Initialize two lists left dist 、right dist Used to store the distance between the left and right wrists and the center point of the normalized prediction box. Traverse the pre-processed raw features of each frame in the window, extract the coordinates of the center point and the key points of the left and right wrists, calculate the Euclidean distance between the left and right wrists and the center point of each frame in the window, and then store these distances separately To the left of the list dist 、right dist Assume that the normalized prediction box center coordinates in each frame are i is the frame number in the window:

[0104]

[0105] Compare the mean distances from the left and right wrists to the center of the normalized prediction box in a window to determine which hand is farther out, and select the corresponding distance list to calculate its mean, standard deviation, and range, and return these statistics as feature vectors, where distances i is the i-th element in the distance list.

[0106]

[0107]

[0108] dist_range=max(distances i )-min(distances i )

[0109] i=1,…,10

[0110] Then the distance statistic from the wrist to the center of the normalized prediction box is:

[0111] feature4=(dist_mean,dist_std,dist_range)

[0112] Feature 5: Distance statistics from the elbow to the center of the normalized prediction box.

[0113] Initialize two lists left dist 、right dist Used to store the distance between the left and right elbows and the center point of the normalized prediction box. Traverse the pre-processed raw features of each frame in the window, extract the center point and the key point coordinates of the left and right elbows, calculate the Euclidean distance between the left and right elbows and the center point of each frame in the window, and then store these distances separately To the left of the list dist 、right dist Assume that the normalized prediction box center coordinates in each frame are i is the frame number in the window:

[0114]

[0115] Compare the mean distances from the left and right elbows to the center of the normalized prediction box in a window to determine which hand reaches farther, and select the corresponding distance list to calculate its mean, standard deviation, and range, and return these statistics as feature vectors, where distances i is the i-th element in the distance list.

[0116]

[0117] dist_range=max(distances i )-min(distances i )

[0118] i=1,…,10

[0119] Then the distance statistic from the elbow to the center of the normalized prediction box is:

[0120] feature5=(dist_mean,dist_std,dist_range)

[0121] Feature 6: Statistics of the distance from wrist to hip.

[0122] Initialize two lists left dist 、right dist Used to store the distances from the left wrist and right wrist to the left hip and right hip. Traverse the pre-processed raw features of each frame in the window, extract the key point coordinates of the left wrist and right wrist, left hip and right hip, calculate the Euclidean distances between the left wrist and left hip, and between the right wrist and right hip in each frame in the window, and then store these distances separately. To the left of the list dist 、right dist Assume that the normalized prediction box center coordinates in each frame are i is the frame number in the window:

[0123]

[0124] Compare the mean distances between the left wrist and left hip, and the right wrist and right hip in a window to determine which hand is farther out, and select the corresponding distance list to calculate its mean, standard deviation, and range, and return these statistics as a feature vector, where distances i is the i-th element in the distance list.

[0125]

[0126] dist_range=max(distances i )-min(distances i )

[0127] i=1,…,10

[0128] The statistic of the distance from wrist to hip is:

[0129] feature6=(dist_mean,dist_std,dist_range)

[0130] Feature 7: Statistics of the distance from elbow to hip.

[0131] Initialize two lists left dist 、right dist Used to store the distances from the left elbow and right elbow to the left hip and right hip. Traverse the pre-processed raw features of each frame in the window, extract the key point coordinates of the left elbow and right elbow, left hip and right hip, calculate the Euclidean distances between the left elbow and left hip, and between the right elbow and right hip in each frame in the window, and then store these distances separately. To the left of the list dist、right dist Assume that the normalized prediction box center coordinates in each frame are i is the frame number in the window:

[0132]

[0133] Compare the mean distances between the left elbow and left hip, and the right elbow and right hip in a window to determine which hand is farther out, and select the corresponding distance list to calculate its mean, standard deviation, and range, and return these statistics as a feature vector, where distances i is the i-th element in the distance list.

[0134]

[0135] dist_range=max(distances i )-min(distances i )

[0136] i=1,…,10

[0137] The statistic of the distance from the elbow to the hip is:

[0138] feature7=(dist_mean,dist_std,dist_range)

[0139] Feature 8: Average arm length proportions.

[0140] Traverse the pre-processed raw features of each frame in the window and extract the key point coordinates of the left and right elbows, left and right wrists, and left and right shoulders. i is the frame number in the window. Calculate the length and ratio of the left upper arm and left lower arm for each frame:

[0141]

[0142] Calculate the length and ratio of the right upper arm and right lower arm for each frame

[0143]

[0144] Aggregate the proportions of the left upper and lower arms, and the right upper and lower arms in a window:

[0145] left_win_ratio=left_ratio 1 ,…,left_ratio 10 ,i=1,…,10

[0146] right_win_ratio=right_ratio 1 ,…,right_ratio10 ,i=1,…,10

[0147] Calculate the Euclidean distance between the left and right wrists and the center of the prediction box in each frame in the window. Assume that the normalized coordinates of the center of the prediction box in each frame are i is the frame number in the window:

[0148]

[0149] Compare the mean distances between the left and right wrists and the center of the prediction box in a window to determine which hand is reaching farther, and select the arm length ratio of the hand reaching farther as a feature:

[0150]

[0151] Feature 9: The average angle between the shoulder, elbow, and wrist.

[0152] Traverse the pre-processed raw features of each frame in the window and extract the key point coordinates of the left and right elbows, left and right wrists, and left and right shoulders. i is the frame number in the window. The key point coordinates in each frame are as follows:

[0153] Left shoulder: Right shoulder:

[0154] Left elbow: Right elbow:

[0155] Left wrist: Right wrist:

[0156] Calculate the angles between the left shoulder, elbow, and wrist for each frame in the window:

[0157]

[0158] if but

[0159] Calculate the angles between the right shoulder, elbow, and wrist for each frame in the window:

[0160]

[0161] if but

[0162] Compute the average angle in the window:

[0163]

[0164] Calculate the Euclidean distance between the left and right wrists and the center of the prediction box in each frame in the window. Assume that the normalized coordinates of the center of the prediction box in each frame are i is the frame number in the window:

[0165]

[0166] Compare the average distances between the left and right wrists and the center of the prediction box in a window to determine which hand is extended further, and select the average angle between the shoulder, elbow, and wrist of the hand that extends further as the feature:

[0167]

[0168] 2. Model Training

[0169] (1) Load the feature data and action type annotation data from the table file obtained by the feature processing module, and divide the data into a training set and a validation set in a ratio of 7:3.

[0170] (2) In order to improve the efficiency and effect of model training, principal component analysis (PCA) is used to reduce the dimensionality of the input feature data and reduce the feature dimension to 50 dimensions.

[0171] (3) A stacked classifier is constructed using a multi-layer perceptron (MLP) and a gradient boosting-based tree model XGBoost classifier as base classifiers, and logistic regression as a meta-learner.

[0172] The parameters of the MLP classifier are set as follows: set three hidden layers, with 20 neurons in the first layer, 30 neurons in the second layer, and 40 neurons in the third layer (hidden_layer_sizes = (20, 30, 40)); the maximum number of iterations is 1000 (max_iter = 1000), and if the algorithm converges before reaching the maximum number of iterations, training will be stopped early; the optimization tolerance is 1e-5 (tol = 1e-5), and if the change in the loss function is less than this value, training will be stopped early; the learning rate is reduced inversely with the number of iterations (learning_rate = "invscaling"), the specific formula is: Where η0 is the initial learning rate, power t is a hyperparameter; when warm restart is set to True, the solution of the previous model is reused as the initial solution during training and more iterations are added instead of starting from scratch (warm_start=True).

[0173] The parameters of the XGBoost classifier are set as follows: 100 trees are built (n_estimators = 100); the learning rate is 0.1 (learning_rate = 0.1); and the maximum depth of each tree is 6 (max_depth = 6).

[0174] The parameters of the logistic regression linear model are set as follows: L2 regularization type (penalty = "l2"), dual method not used (dual = False), optimization tolerance of 0.0001 (tol = 0.0001), inverse regularization strength of 1 (C = 1), intercept calculated (fit_intercept = True), intercept scaled to 1 (intercept_scaling = 1), no class weights used (class_weight = None), random number generator not specified (random_state = None), LBFGS optimization algorithm (solver = "lbfgs"), maximum number of iterations of 100 (max_iter = 100), automatic selection of multi-class strategy (multi_class = "auto"), no warm start (warm_start = False), and no L1 ratio (l1_ratio = None).

[0175] (4) First, each base classifier is trained using the input feature data to obtain an independent model for each base classifier. Then, a 5-fold cross-validation method is used to predict the input feature data to generate a secondary training dataset. Finally, the meta-learner is trained using this secondary training dataset to construct the final classification model.

[0176] 4. Theft Behavior Recognition Algorithm

[0177] (1) To prevent the rapid reading of frame data from the surveillance video stream from conflicting with the theft recognition algorithm's processing of video frames, a picture reading thread and an algorithm thread are set up. The picture reading thread initializes a video capture object cv2.VideoCapture for each RTSP address of the predefined camera video stream. Each video capture object continuously reads frames from each video stream and stores them in a list. A queue is set up to store the received frames, and the frame list is placed in the queue for use by the theft recognition algorithm thread. The theft recognition algorithm thread is then notified to start processing the frames.

[0178] (2) The theft behavior recognition algorithm thread obtains video frames from the queue, scales them to a fixed size, and then inputs them into the YOLOv8pose model. If a customer is detected in the video frame, 17 normalized human skeleton key point coordinates, normalized prediction box height, and center point coordinates are obtained as raw features. If no customer is detected in the video frame, a null value is returned and the model does not participate in the subsequent algorithm process.

[0179] (3) The original features are preprocessed to retain only the keypoint coordinates of the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, as well as the normalized prediction box height and center coordinates. If a keypoint is missing in the current video frame, the position of the missing keypoint is filled in based on the position of the keypoint relative to the normalized prediction box center in the previous frame.

[0180] (4) The preprocessed original features are input into the feature processing module. The feature processing module aggregates the preprocessed original features into a window with a window size of 10 frames. If the window size is not met, the original features of the next frame are read. If the window size is met, 9 features highly correlated with the hand being close to the body and the hand being far away from the body are extracted, and the 9 features of the current window are merged into a vector. Principal component analysis (PCA) is used to reduce the dimensionality of the vector. After the feature dimension is reduced to 50, it is input into the action classification model. By sliding the window frame by frame, an action classification result is output for each frame.

[0181] (5) After obtaining the action classification results of the video frames, the action classification results of the video frames of multiple monitoring video streams are integrated to determine the total action classification result; if the action classification results of the video frames of more than half of the monitoring video streams are that the hand is close to the body, the total action classification result is recorded as the hand is close to the body, and the current video frame is recorded as a key frame; if the action classification results of the video frames of more than half of the monitoring video streams are that the hand is away from the body, the total action classification result is recorded as the hand is away from the body, and the current video frame is marked as a key frame; otherwise, the total action classification result is recorded as other actions.

[0182] (6) With the wrist joint of the key frame as the center, a hand image with a size of 100*100 is cropped, and after resizing, center cropping, tensor conversion and normalized data preprocessing, it is input into the classification model of whether there is an object in the hand to obtain the classification result; if the classification result of the left hand image cropped from a video frame is that there is an object in the hand, then it is recorded that there is an object in the left hand of the customer in the current video frame; otherwise, it is recorded that there is no object in the left hand of the customer in the current video frame; if the classification result of the right hand image cropped from a video frame is that there is an object in the hand, then it is recorded that there is an object in the right hand of the customer in the current video frame; otherwise, it is recorded that there is no object in the right hand of the customer in the current video frame;

[0183] (7) According to the preset conditions, update the hand close to the body frame queue and the hand away from the body frame queue. The specific process is as follows: Figure 3 As shown in the figure, the queue elements are lists, including the total action classification results, the classification results of whether an object is present in the hand, and the coordinates of the wrist key points. When the theft behavior recognition algorithm thread runs, it creates and initializes the queues for frames where the hand approaches the body and the hand moves away from the body.

[0184] If the total action classification result of the previous frame is not hand approaching the body and the total action classification result of the current video frame is hand approaching the body, then the total action classification result of the current video frame, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points are merged into a list and added to the end of the hand approaching the body frame queue.

[0185] If the total action classification results of the previous frame and the current video frame are both "hand close to the body", then the total action classification result of the current video frame, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points are merged into a list and added to the end of the hand close to the body frame queue.

[0186] If the total action classification results of the previous frame and the current video frame are both "hands away from the body", then the total action classification result of the current video frame, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points are merged into a list and added to the end of the hand away from the body frame queue.

[0187] If the total action classification result of the previous frame is that the hand is close to the body and the total action classification result of the current video frame is not that the hand is close to the body, then determine whether the length of the hand close to the body frame queue is greater than or equal to 5. If it is less than 5, clear the hand close to the body frame queue. If it is greater than or equal to 5, retain the elements in the hand close to the body frame queue.

[0188] If the total action classification result of the previous frame is not hand away from the body, and the total action classification result of the current video frame is hand away from the body, then determine whether the length of the hand close to the body frame queue is greater than 0. If it is greater than 0, then merge the total action classification result of the current video frame, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue; otherwise, clear the hand away from the body frame queue.

[0189] (8) Combined with the length of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether there is an object in the hand, a comprehensive judgment is made as to whether the customer has committed theft at the current moment. The specific process is as follows: Figure 4 shown.

[0190] If the total action classification result of the previous frame is hands away from the body and the total action classification result of the current video frame is not hands away from the body, then determine whether the length of the hands approaching the body frame queue is greater than 0 and whether the length of the hands away from the body frame queue is greater than or equal to 5; if the conditions are met, then record that the customer's hand first approaches the body and then moves away from the body, that is, the hand approaching the body action and the hand away from the body action are successfully paired, and read the elements in the hand approaching the body frame queue and the hand away from the body frame queue at this time; otherwise, continue to read the original features of the next frame.

[0191] If the hand-approaching-body action is successfully paired with the hand-away-from-body action, read the elements in the hand-approaching-body frame queue and the hand-away-from-body frame queue at this time, calculate the total movement distance of the left wrist and the right wrist in the hand-approaching-body frame queue and the hand-away-from-body frame queue, and return the wrist with the larger total movement distance. The classification result of whether there is an object in the hand is returned, where the total movement distance is calculated and accumulated by the coordinates of the wrist key points of the two adjacent elements in the queue. Calculate the number of frames with and without objects in the hand;

[0192] If the number of frames with an object in the hand in the near-body frame queue exceeds 4, and the number of frames without an object in the far-body frame queue exceeds 4, the customer is suspected of stealing. The near-body and far-body frame queues are cleared, the time is recorded, and a reminder is sent to the back-end staff. Otherwise, the original feature of the next frame will be read. The back-end staff can review the surveillance video stream based on the recorded time points to check whether the customer has stolen.

[0193] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for identifying theft in unmanned convenience stores, characterized by: include: Acquire multiple surveillance video streams of the surveillance scene, create and initialize a hand-approaching-body frame queue and a hand-away-from-body frame queue; Obtain video frames from each video stream and perform detection. If a customer is detected, obtain the original features of the video frame and perform data preprocessing. Build a window and aggregate multiple video frames within the window; extract the action behavior features related to the hand approaching the body and the hand moving away from the body in the window based on the original features of the video frames after data preprocessing; After dimensionality reduction processing on the action behavior features within the window, the model is input into the action classification model to obtain the action classification results of the video frame. Based on the action classification results of the video frames at the same moment in all the monitoring video streams, the total action classification results are recorded as hand approaching the body, hand away from the body, or other actions. Based on the total action classification results, it is decided whether to use the video frame at that moment as a key frame. For each key frame of the surveillance video stream, input it into the classification model of whether there is an object in the hand to obtain the classification result of whether there is an object in the customer's hand; According to the preset conditions, update the frame queue of the hand approaching the body and the frame queue of the hand away from the body; The length of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether the customer has an object in their hand, are used to comprehensively determine whether the customer is suspected of stealing at the current moment. Record the time when customers are suspected of stealing and send reminder messages; The obtaining of original features of the video frame and performing data preprocessing includes: Use the YOLOv8-Pose model to extract the coordinates of specific key points of the customer's human skeleton and the normalized prediction box height and center point coordinates; the specific key points include the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip; Action behavior characteristics in the window include: The key points of the original features include the relative coordinates of the normalized prediction frame center as the origin, the movement speed of the key points in two adjacent frames, the angle between the key points and the horizontal plane, the distance statistics from the wrist to the normalized prediction frame center, the distance statistics from the elbow to the normalized prediction frame center, the distance statistics from the wrist to the hip, the distance statistics from the elbow to the hip, the average arm length ratio, and the average angle between the shoulder, elbow and wrist.

2. The method for identifying theft in an unmanned convenience store according to claim 1, characterized in that: The action classification model is obtained by training the training set, including: The training set consists of video frames of a single person engaging in theft and corresponding action type annotations, which are divided into three action categories: hands close to the body, hands away from the body, and other actions. The YOLOv8-Pose model is used to obtain the original features in the video frames and preprocess them. The action behavior features that are highly correlated with hands close to the body and hands away from the body are extracted. The action behavior features of each window and the action type annotation of each video frame in the window are combined into two table files. Loading the feature data and the action type annotation data from the table file and dividing them into a training set and a validation set; The principal component analysis method is used to reduce the dimension of the input feature data, and the action behavior characteristics and corresponding action type labels after dimension reduction in a window are used as a training sample; A stacked classifier is constructed using a multi-layer perceptron and a gradient boosting-based tree model XGBoost classifier as base classifiers, and logistic regression as a meta-learner. First, each base classifier is trained separately using the input training samples to obtain an independent model for each base classifier. Then, a 5-fold cross-validation method is used to predict the input feature data to generate a secondary training dataset. Finally, the secondary training dataset is used to train the meta-learner to construct the final action classification model.

3. The method for identifying theft in an unmanned convenience store according to claim 1, characterized in that: The decision on whether to use the video frame at that moment as a key frame is made based on the overall action classification result, including: If the action classification results of the current video frames of more than half of the video streams at the same moment are hands approaching the body or hands away from the body, the total action classification result at that moment is recorded as hands approaching the body or hands away from the body, and the current video frames of all video streams are recorded as key frames; otherwise, the total action classification result is recorded as other actions.

4. The method for identifying theft in an unmanned convenience store according to claim 1, characterized in that: Based on preset conditions, update the frame queues for when the hand approaches the body and the frame queues for when the hand moves away from the body, including: The elements in the hand-close-to-body frame queue and the hand-away-from-body frame queue are lists, including the total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points; If the previous total action classification result is not hand approaching the body and the current total action classification result is hand approaching the body, then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand approaching the body frame queue; If the previous and current total action classification results are both "hand close to body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand close to body frame queue; If the previous and current total action classification results are both "hands away from the body", then combine the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue; If the previous total action classification result is hand approaching the body and the current total action classification result is not hand approaching the body, then determine whether the length of the hand approaching the body frame queue is greater than or equal to 5. If it is greater than or equal to 5, retain the elements in the hand approaching the body frame queue; if it is less than 5, clear the hand approaching the body frame queue; If the previous total action classification result is not the hand away from the body, and the current total action classification result is the hand away from the body, then determine whether the length of the hand close to the body frame queue is greater than 0. If it is greater than 0, then merge the current total action classification result, the classification result of whether there is an object in the hand, and the coordinates of the wrist key points into a list and add it to the end of the hand away from the body frame queue; otherwise, clear the hand away from the body frame queue.

5. The method for identifying theft in an unmanned convenience store according to claim 1, characterized in that: Combined with the length of the hand-close-to-body frame queue and the hand-away-from-body frame queue, as well as the classification results of whether the customer has an object in their hand, a comprehensive judgment is made as to whether the customer is stealing at the current moment, including: If the previous total action classification result is "hands away from the body" and the current total action classification result is not "hands away from the body," then determine whether the length of the "hands approaching the body" frame queue is greater than 0 and whether the length of the "hands away from the body" frame queue is greater than or equal to 5. If the conditions are met, then record that the customer's hand approaching the body action and the hand away from the body action are successfully paired, and read the elements in the hand approaching the body frame queue and the hand away from the body frame queue at this time. If the hand-approaching-body action is successfully paired with the hand-away-from-body action, read the elements in the hand-approaching-body frame queue and the hand-away-from-body frame queue at this time, calculate the total movement distance of the left wrist and the right wrist in the hand-approaching-body frame queue and the hand-away-from-body frame queue, and return the wrist with the larger total movement distance and the classification result of whether there is an object in the hand; the total movement distance is calculated and accumulated by the coordinates of the wrist key points of two adjacent elements in the queue; The classification result is calculated as the number of frames with objects in the hand and without objects in the hand; if the number of frames with objects in the hand in the hand-close-to-body frame queue is greater than 4 and the number of frames without objects in the hand in the hand-away-from-body frame queue is greater than 4, the customer is considered to have committed suspected theft, and the hand-close-to-body frame queue and the hand-away-from-body frame queue are cleared.

6. A system for identifying theft in unmanned convenience stores, comprising a processor, a memory, and a computer program stored in the memory; characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium storing a computer program; wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

Images