Behavior recognition method and device, equipment and medium

Abstract

The invention relates to the field of behavior recognition, in particular to a behavior recognition method, device and equipment and a medium, and the method comprises the steps: carrying out the target recognition of a video frame of a monitoring region, and obtaining the recognition information including personnel information, cash detection frame information and cashbox detection frame information; and then subsequent behavior identification processing is carried out based on the identification information in combination with the preset cashier area information to determine whether a cash private receipt behavior exists, so that whether the cash private receipt behavior exists can be identified, and the cashier risk is managed and controlled.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition, and in particular to a behavior recognition method, device, equipment and medium. Background Art

[0002] Although modern cashier technology has made significant progress, mobile payment has become popular, and traditional cashier methods have been challenged, there are still loopholes. Customers lack understanding and attention to the cashier process, and may not notice the cashier's abnormal operation, which makes theft successful; with the diversification of payment methods, cashiers may need to handle various payment methods and promotions, which may increase the complexity of operations and provide opportunities for stealing money.

[0003] Therefore, it is necessary to design intelligent and automated cash collection technology to achieve cashier risk management and regulate employee behavior. Summary of the invention

[0004] The purpose of this application is to provide a behavior recognition method, device, equipment and medium, which can identify whether there is private cash collection behavior and realize the management and control of cash register risks.

[0005] In a first aspect, a behavior recognition method is provided, comprising:

[0006] Obtain the video to be tested in the cashier monitoring area;

[0007] Performing target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame, wherein the recognition information includes: personnel information, cash detection frame information and cash box detection frame information, and the personnel information includes: cashier detection frame information and cashier posture;

[0008] Determine whether there is any private cash collection behavior based on the preset cashier area information in the video frame and the identification information.

[0009] In a preferred example, the present application can be further configured as follows: performing target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame, including:

[0010] The yolov8-pose model is used to perform target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame.

[0011] In a preferred example, the present application may be further configured as follows: determining whether there is a cash collection behavior based on the preset cashier area information in the video frame and the identification information includes:

[0012] Determine the position detection information according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame;

[0013] When the position detection information satisfies the first condition, determining whether there is a cash-handling action according to the cashier's posture, if there is no cash-handling action, determining that there is cash-handling behavior, otherwise there is no cash-handling behavior;

[0014] When the location detection information satisfies the second condition, it is determined that there is an act of privately collecting cash;

[0015] The first condition is that the cash is in the preset cashier area, the cash is outside the cash box detection frame information, and the cashier is in the preset cashier area;

[0016] The second condition is that the cash is in the preset cashier area and there is no cash box detection frame.

[0017] In a preferred example, the present application can be further configured to: determine the position detection information according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame, including:

[0018] Determine whether the cash is within the preset cash register area according to the preset cash register area and cash detection frame information in the video frame;

[0019] If the cash is in the preset cashier area, determining whether the cash is outside the cash box detection frame according to the cash box detection frame information and the cash detection frame information;

[0020] If the cash is outside the cash box detection frame, it is determined whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame.

[0021] In a preferred example, the present application may be further configured as follows: determining whether the cashier is in the preset cashier area according to the preset cashier area and cashier detection frame information in the video frame, including:

[0022] Determine whether there is a person in a preset customer area in the video frame;

[0023] If there is a person in the preset customer area, determining whether the cashier is in the preset cashier area according to the preset cashier area and cashier detection frame information in the video frame;

[0024] Otherwise, it is determined that there is behavior of private collection of cash.

[0025] In a preferred example, the present application can be further configured as follows: the second condition also includes: the cash is in the preset checkout area, and the cash is outside the cash box detection frame information, and there are no customers in the preset customer area.

[0026] In a preferred example, the present application can be further configured as follows: the cashier posture includes: shoulder joint position, elbow joint position, wrist joint position,

[0027] Determining whether there is a reaching action according to the cashier's posture includes:

[0028] Determining an elbow joint angle according to the shoulder joint position, the elbow joint position, and the wrist joint position;

[0029] Determine whether there are a preset second number of elbow joint angles greater than the preset elbow joint angle in a continuous preset first number of video frames, and the preset first number is greater than the preset second number.

[0030] In a second aspect, a behavior recognition device is provided, comprising:

[0031] An acquisition module is used to acquire the video to be detected in the cashier monitoring area;

[0032] The recognition module is used to perform target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame, wherein the recognition information includes: personnel information, cash detection frame information and cash box detection frame information, and the personnel information includes: cashier detection frame information and cashier posture;

[0033] The behavior confirmation module is used to determine whether there is any private cash collection behavior based on the preset cashier area information in the video frame and the identification information.

[0034] In a third aspect, an electronic device is provided, including:

[0035] one or more processors;

[0036] Memory;

[0037] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the behavior recognition method shown in any possible implementation of the first aspect.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded by a processor and executes the steps of the behavior recognition method shown in any possible implementation method of the first aspect.

[0039] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements operations corresponding to the behavior recognition method shown in any possible implementation manner in the first aspect.

[0040] In summary, the method provided by this application includes the following beneficial technical effects:

[0041] This solution performs target recognition on the video frames of the monitoring area and obtains recognition information including personnel information (cashier detection frame information, cashier posture), cash detection frame information, and cash box detection frame information. Based on the recognition information and the preset cashier area information, subsequent behavior recognition processing is performed to determine whether there is any private cash collection behavior. It can identify whether there is any private cash collection behavior, realize the control of cashier risks, and then manage the standardization of employee services and evaluate service quality conveniently and comprehensively based on the recognized behavior.

[0042] In addition, the present application also provides a behavior recognition device, equipment and medium, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flowchart of a behavior recognition method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a cashier scene provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of COCO human posture data provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the positional relationship between a target T and a region Q provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of another behavior recognition method provided in an embodiment of the present application;

[0049] Figure 6 A flowchart of another behavior recognition method provided in an embodiment of the present application;

[0050] Figure 7 is a structural diagram of a behavior recognition device provided in an embodiment of the present application;

[0051] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, a person skilled in the art may make non-creative modifications to the present embodiment as needed, but such modifications are protected by the patent law as long as they are within the scope of the present application.

[0053] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0056] The present application belongs to the field of image processing and computer vision technology, and involves technologies such as deep learning and posture estimation. It aims at realizing a method for detecting the behavior of privately collecting cash in cashier scene videos.

[0057] The present application embodiment provides a behavior recognition method, such as Figure 1As shown, the method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device is a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The terminal device and the electronic device can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes:

[0058] S101, obtaining a video to be detected in the cashier monitoring area;

[0059] In the embodiment of the present application, the video to be detected is a video including a cashier area and a customer area captured by a camera device in a merchant cashier scene. The merchant cashier scene refers to a specific environment or place where a cashier and a customer settle transactions in a commercial activity.

[0060] A camera device is a device that can capture the merchant's checkout scene. It may be a fixed-mounted camera or a portable camera device used to capture and record images or videos.

[0061] In the merchant cashier scene, the cashier area refers to the area dedicated to processing transaction settlement, usually including cashier counters, cash registers and other equipment, and is the core area of ​​transaction activities. The customer area refers to the area where customers wait, queue or conduct transactions in the merchant cashier scene. It may be adjacent to the cashier area or around the cashier area. For example, see Figure 2 , Figure 2 A schematic diagram of a cashier scene provided in an embodiment of the present application. Figure 2 The cashier area (such as a cashier counter, a cash register including a cash box, and a cashier), the customer area (such as customers waiting in line), and the camera device (such as a camera installed on the ceiling or wall) are included. The shooting range and angle of the camera device are no longer limited in the embodiment of the present application. When the camera parameters are determined, the customer area and the cashier area can be determined. During the merchant's cashier process, when the camera device is in a working state and continuously collects video, the video content including the cashier area and the customer area is collected by the camera device, and these video contents are stored or transmitted as videos to be detected, so that the electronic device can obtain the video to be detected.

[0062] S102, performing target recognition on the video frame of the video to be detected, and obtaining recognition information corresponding to the video frame, the recognition information including: personnel information, cash detection frame information and cash box detection frame information, the personnel information including: cashier detection frame information and cashier posture;

[0063] Input the video to be detected, read each video frame in turn, and obtain the identification information corresponding to the video frame.

[0064] Cash detection frame information refers to the information about the location of cash detected by video analysis technology, expressed in the form of a rectangular frame, used to indicate the location and range of cash in the video frame. Cash box detection frame information refers to the information about the location of the cash box detected by video analysis technology, expressed in the form of a rectangular frame, used to indicate the location and range of the cash box in the video frame. Cashier detection frame information refers to the information about the location of the cashier detected by video analysis technology, expressed in the form of a rectangular frame, used to indicate the location and range of the cashier in the video frame. Cashier posture refers to the posture information of the cashier in the video frame, specifically referring to posture information related to behavior.

[0065] The method for determining the identification information is not limited in the present embodiment, and the user can set it according to actual needs.

[0066] In one feasible manner, a YOLOv8-Pose model is used to perform target recognition on a video frame of a video to be detected, and recognition information corresponding to the video frame is obtained.

[0067] YOLOv8-Pose is a pose estimation technology based on deep learning. It combines the YOLOv8 target detection algorithm and the Pose Estimation pose estimation algorithm. It can simultaneously detect human cash box targets and estimate human poses. It has few model parameters and fast reasoning speed. The model is built using a deep convolutional neural network (DCNN). It estimates human poses by extracting features from images and associating key point coordinates.

[0068] Specifically, the YOLOv8-Pose model is used to obtain the human posture information, cash and cash box and their respective position information, as well as the corresponding confidence level for each frame. Regarding the target: cashier, cash and cash box, their corresponding information can be in the same format or in different formats. Figure 3 , Figure 3 A schematic diagram of COCO human posture data provided for an embodiment of the present application. For the embodiment of the present application, the human posture information can focus on six joint points: shoulder, elbow, and wrist (5, 6, 7, 8, 9, 10).

[0069] Exemplarily, when the formats of cashier, cash and cash box are the same, they are all (label, x, y, w, h, x5, y5, p5, x6, y6, p6, x7, y7, p7, x8, y8, p8, x9, y9, p9, x10, y10, p10), where label represents the category number of each target object, the subsequent coordinates are normalized coordinates, x, y represent the center position of the target object in the image, w, h represent the width and height of the target object; the joint points are represented by three values, for example, x5, y5, p5 represent the x coordinate, y coordinate and visibility flag of the 5th joint point respectively (0 means unlabeled, 1 means labeled but blocked, 2 means labeled and unblocked). For cash and cash box, only (label, x, y, w, h) are marked, and the joint point information is filled with (0, 0, 0). Exemplarily, the cashier label is 1, the cash label is 2 and the cash box label is 3.

[0070] In one possible approach, the training process of the YOLOv8-Pose model includes: selecting the public pre-trained network of the YOLOv8-Pose model as the basic network; using pre-labeled human posture and cash box data sets to train the pre-trained network. The pre-trained network refers to a neural network model that has been trained on a large-scale data set, has learned many common feature representations, and has a certain behavior detection capability.

[0071] S103: Determine whether there is any private cash collection behavior based on the preset cashier area information and identification information in the video frame.

[0072] It can be seen that in the embodiment of the present application, the target recognition is performed on the video frame, and the recognition information including personnel information (cashier detection frame information, cashier posture), cash detection frame information and cash box detection frame information is obtained. Then, based on the recognition information and the preset cashier area information, subsequent behavior recognition processing is performed to determine whether there is cash collection behavior; this embodiment can identify whether there is cash collection behavior, realize the control of cashier risks, and then can conveniently and comprehensively manage the standardization of employee services and evaluate service quality based on the recognized behavior.

[0073] Specifically, determining whether there is any private cash collection behavior based on the preset cashier area information and identification information in the video frame includes:

[0074] Determine the position detection information according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame;

[0075] When the position detection information satisfies the first condition, it is determined whether there is a cash-handling action according to the cashier's posture. If there is no cash-handling action, it is determined that there is cash-handling behavior; otherwise, there is no cash-handling behavior;

[0076] When the location detection information meets the second condition, it is determined that there is an act of privately collecting cash;

[0077] The first condition is: the cash is in the preset cashier area, the cash is outside the cash box detection frame information, and the cashier is in the preset cashier area;

[0078] The second condition is that the cash is in the preset cashier area and there is no cash box detection frame.

[0079] It is understandable that the embodiment of the present application does not limit the execution order of step a1 determining whether the cash is in the preset cash register area, step a2 determining whether the cash is outside the cash box detection frame information, and step a3 determining whether the cashier is in the preset cash register area. The order of step a1, step a2, and step a3 can be any order, or they can be executed simultaneously, or they can not be executed completely simultaneously.

[0080] Among them, combined Figure 4 , Figure 4 A schematic diagram of the position relationship between a target T and a region Q provided in an embodiment of the present application, a method for determining whether the target T is in the region Q may include:

[0081] Determine the center point coordinates (CeX, CeY) of target T;

[0082] Determine the coordinates of the four vertices of region Q (clockwise), recorded as (AX, AY), (BX, BY), (CX, CY), (DX, DY);

[0083] The directed areas of the four triangles are obtained by calculation. If the directed areas of all the triangles are positive or negative, it is determined that the cash is located in a preset cashier area in the image frame;

[0084] The calculation formula includes a=(AX-DX)*(CeY-DY)-(AY-DY)*(CeX-DX);

[0085] b=(BX-AX)*(CeY-AY)-(BY-AY)*(CeX-AX);

[0086] c=(CX-BX)*(CeY-BY)-(CY-BY)*(CeX-BX);

[0087] d=(DX-CX)*(CeY-CY)-(DY-CY)*(CeX-CX).

[0088] When the target T is cash, area Q is the preset cashier area; when the target T is cash, area Q is the cash box area corresponding to the cash box detection frame information; when the target T is a cashier, area Q is the preset cashier area. Exemplarily, assuming that the target T is cash, the coordinates of the center point of the cash detection frame are (CeX1, CeY1), where the four vertex coordinates of the cash detection frame constitute the cash detection frame information; the four vertex coordinates of the preset cashier area (clockwise) are (AX1, AY1), (BX1, BY1), (CX1, CY1), (DX1, DY1).

[0089] Furthermore, the second condition also includes: the cash is in a preset cashier area, the cash is outside the cash box detection frame information, and there is no customer in the preset customer area.

[0090] It can be seen that in the embodiment of the present application, the position detection information is determined based on the preset checkout area, cash detection box information, cash box detection box information and cashier detection box information; according to the different conditions satisfied by the position detection information, different judgment strategies are adopted to determine whether there is any private cash collection behavior.

[0091] A possible implementation of the embodiment of the present application is to determine the position relationship according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame, including:

[0092] Determine whether the cash is in the preset cash register area according to the preset cash register area and cash detection frame information in the video frame;

[0093] If the cash is not in the preset cashier area, the process ends;

[0094] If the cash is in the preset cashier area, determine whether the cash is outside the cash box detection frame based on the cash box detection frame information and the cash detection frame information;

[0095] If the cash is outside the cash box detection frame, it is determined whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame.

[0096] If the cash is in the cash box detection frame, a normal event is generated and reported.

[0097] In the embodiment of the present application, the preset cashier area is pre-set and used to identify the area where the cashier operation should occur. It is determined whether the detected cash is within the preset cashier area to confirm whether the cash appears within the normal cashier operation area; if the cash is not detected, the identification process is terminated; if the detected cash is not within the preset cashier area, it indicates that the cashier operation has not been performed, and the identification process is terminated at this time; if the detected cash is within the preset cashier area, it is determined whether the cash is outside the cash box detection frame to confirm whether the cash has been correctly placed in the cash box; if the cash is outside the cash box detection frame, it is determined whether the cashier is still within the preset cashier area to confirm whether the cashier is present and may have participated in abnormal cash handling behavior; if the cash is within the cash box detection frame, it is determined that this operation is a normal cashier operation, so a normal event is generated and reported to the relevant management system or personnel. If the cash is outside the cash box detection frame and the cashier is within the preset cashier area, a warning or abnormal event may be generated at this time, indicating that there may be private cash collection or other improper behavior, so as to determine whether there is a handover action through the cashier's posture, and further determine whether there is private cash collection behavior.

[0098] It can be seen that in the embodiment of the present application, firstly, it is determined whether the cash is in the preset cashier area to screen out cash transactions that may be required; by determining whether the cash is outside the cash box detection frame, it focuses on the final destination of the cash; if the cash is not in the cash box, it is determined whether the cashier is in the preset cashier area to confirm whether someone may be involved in abnormal cash handling. By analyzing the position relationship in the above sequence, a preliminary check is conducted to determine whether there is a possibility of private collection, thereby reducing the possibility of false alarms and missed alarms.

[0099] A possible implementation of the embodiment of the present application is to determine whether the cashier is in the preset cashier area according to the preset cashier area and cashier detection frame information in the video frame, including:

[0100] Determine whether there is a person in a preset customer area in the video frame;

[0101] If there is a person in the preset customer area, determine whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame;

[0102] Otherwise, it is determined that there is behavior of private collection of cash.

[0103] Among them, if there is no person in the preset customer area, it is directly determined that there is cash collection behavior. In the embodiment of the present application, the normal cashier process usually has customers waiting or conducting transactions in the customer area. If there is a person in the preset customer area, it is determined whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame; if there is no person in the preset customer area, cash is detected outside the cash box, indicating that the cashier performs cashier operations without the presence of customers, which is likely to be illegal or abnormal. At this time, it is determined that there is abnormal cash collection behavior.

[0104] It can be seen that in the embodiment of the present application, before judging whether the cashier is in the preset checkout area, the personnel situation in the preset customer area is also considered, thereby avoiding misjudgment caused by the presence of customers.

[0105] In a possible implementation of the embodiment of the present application, the cashier's posture includes: shoulder joint position, elbow joint position, wrist joint position,

[0106] Determine whether there is a handover action based on the cashier's posture, including:

[0107] Determine the elbow joint angle according to the shoulder joint position, elbow joint position and wrist joint position;

[0108] Determine whether there are a preset second number of elbow joint angles greater than the preset elbow joint angle in a continuous preset first number of video frames, and the preset first number is greater than the preset second number.

[0109] Specifically, it is determined whether there is a person detection frame in the current frame image, and if there is no person detection frame, the process ends.

[0110] The human detection frame is filtered into the customer area and the cashier area. If a person is in the customer area, it means there is a customer. If a person is in the cashier area, it means there is a cashier staff. It is determined whether there is a cashier staff member who meets the preset first number (such as 5, 7, 8, etc.) of elbow-shoulder angles on the same side within the preset second number (such as 2, 3, 4) of frames greater than 120 degrees. If it is met, it is determined that the cashier staff has extended the hand. For the preset first number and the preset second number, the user can set them according to actual needs.

[0111] Among them, human body posture estimation determines whether there is a handover behavior:

[0112] A. First, filter the joint points and only keep the data with confidence greater than 0.5.

[0113] B. Determine the positional relationship between the joint points and the human target frame, and only retain the joint points within the target frame.

[0114] C. Calculate the ipsilateral wrist, elbow, and shoulder angles.

[0115] Take the left 5, 7, and 9 joint points as an example, assuming that the 5, 7, and 9 joint points are A(x1, y1), B(x2, y2), and C(x3, y3) respectively. To calculate the angle of ∠ABC, you can use the vector and vector It is found by taking the dot product of .

[0116] First, calculate the vector and vector :

[0117] =(x1−x2,y1−y2);

[0118] =(x3−x2,y3−y2);

[0119] Then, we calculate the dot product of the two vectors:

[0120] =(x1−x2)(x3−x2)+(y1−y2)(y3−y2);

[0121] Next, calculate the magnitude of the two vectors:

[0122] ;

[0123] ;

[0124] Finally, use the law of cosines to calculate the cosine of ∠ABC:

[0125] ;

[0126] Since the angle range is 0° to 180°, we can use the inverse cosine function to get the angle:

[0127] ;

[0128] The result is in radians and needs to be converted to degrees using the following conversion formula:

[0129] ;

[0130] D. Determine whether there is any delivery behavior.

[0131] After obtaining the angles of the wrist, elbow and shoulder on the same side, they are stored in the list angleQueue. If the sliding judgment is that 2 frames out of every 5 frames satisfy the angle of any side greater than 120 degrees, it is considered that there is a delivery behavior. The specific code implementation is as follows.

[0132] bool currentFrameValid = (angle1 > 120.0f || angle2 > 120.0f);

[0133] angleQueue.push(currentFrameValid);

[0134] if (angleQueue.size() > 5)

[0135] {

[0136] angleQueue.pop();

[0137] }

[0138] / / Calculate the number of valid frames in the queue

[0139] int validCount = 0;

[0140] for (auto it = angleQueue; !it.empty(); it.pop())

[0141] {

[0142] if (it.front())

[0143] {

[0144] ++validCount;

[0145] }

[0146] }

[0147] if (validCount >= 2)

[0148] {

[0149] handCash2Cust = 1;

[0150] }

[0151] It can be seen that in the embodiment of the present application, when judging whether the cashier has extended the cashier's hand, the elbow joint angle is calculated based on the cashier's posture (shoulder joint position, elbow joint position, wrist joint position), and it is judged whether there are a preset second number of elbow joint angles greater than the preset elbow joint angle in a consecutive preset first number of video frames. The cashier's extended hand action can be accurately identified, thereby further confirming whether there is any behavior of privately collecting cash, thereby improving the judgment accuracy of the system.

[0152] Furthermore, when there is cash private collection, the current transaction information can be retrieved to determine whether the current transaction participates in preferential activities such as point redemption or coupons. If it participates, a first prompt message is generated, and if it does not participate, a second prompt message is generated. It can be understood that if a preferential activity is participated in and cash private collection is identified, it is very likely that cash private collection actually exists, and the first prompt message is used to prompt the administrator to pay attention and take timely measures; if the activity is not participated in, but cash private collection occurs, it may be a case of identification error, and the second prompt message is used to prompt the administrator to pay attention to the incident.

[0153] Based on any of the above embodiments, the embodiment of the present application discloses a method for detecting cash embezzlement based on posture estimation. The method is mainly deployed on a mobile terminal, mainly for real-time monitoring of cashier scenes. The method divides the cashier scene into a cashier area and a customer area, and mainly relies on two parts: the YOLOv8-Pose module and the post-processing module. Figure 5 .

[0154] Among them, the YOLOv8-Pose module performs high-precision estimation of human postures and objects in the video to obtain human posture points and cash and cash box detection results; the post-processing module compares the area where the cash is located and its position relationship with the cash box, calculates the joint point angle to determine whether the clerk has extended the cash, and thus determines whether there is an incident of private cash collection.

[0155] The method proposed in the embodiment of the present application can perform real-time detection on the cashier scene, provide automatic and intelligent cashier risk management, as well as convenient and comprehensive standardized management of employee services and service quality evaluation.

[0156] This application embodiment provides a specific cash private collection detection method based on posture estimation, see Figure 6 ,include:

[0157] 1. Determine whether there is a cash detection frame in the current frame image, if not, end the process.

[0158] 2. Filter the cash detection frame into customer area and cashier area.

[0159] Cash is in the customer area.

[0160] The cash is in the cashier area, and it is determined whether there is a cash box detection frame in the cashier area.

[0161] 3. Cash appears in the customer area. Report normal events. If cash appears in the cashier area, no cash box is detected, and there are no customers in the customer area, report private cash collection. Otherwise, go to step 4.

[0162] 4. Compare the position of the cash and the cash box detection frame. If the cash is in the cash box detection frame, report a normal event. Otherwise, proceed to step 5.

[0163] 5. If cash is outside the cash box detection frame, there are customers in the customer area, and the cashier is handing over cash, report it as a normal event, otherwise report it as private cash collection.

[0164] The embodiment of the present application proposes a method for detecting cash embezzlement based on posture estimation. The detection model and posture estimation are implemented in parallel. The model has few parameters and fast reasoning, which meets the real-time requirements. Through customer area cashier area filtering, the position relationship between the cash detection frame and the cash box detection frame is calculated, and human posture estimation assists in judging whether there is a handover behavior to avoid false detection, and quickly and effectively determine cash embezzlement events. The method proposed by the present invention can perform real-time detection on cashier scenes, provide automatic and intelligent cashier risk management, as well as convenient and comprehensive standardized management of employee services and service quality evaluation.

[0165] The following is an introduction to a behavior recognition device provided in an embodiment of the present application. The behavior recognition device described below and the behavior recognition method described above can be referred to each other. The behavior recognition device of this embodiment is set in an electronic device. Figure 7 , Figure 7 : is a structural block diagram of a behavior recognition device according to one embodiment of the present application, comprising:

[0166] An acquisition module 210 is used to acquire a video to be detected in a cashier monitoring area;

[0167] The recognition module 220 is used to perform target recognition on the video frame to be detected, and obtain recognition information corresponding to the video frame, the recognition information includes: personnel information, cash detection frame information and cash box detection frame information, the personnel information includes: cashier detection frame information and cashier posture;

[0168] The behavior confirmation module 230 is used to determine whether there is any private cash collection behavior based on the preset cashier area information and identification information in the video frame.

[0169] In one achievable manner, target recognition is performed on a video frame of a video to be detected to obtain recognition information corresponding to the video frame, including:

[0170] The yolov8-pose model is used to perform target recognition on the video frames of the video to be detected, and the recognition information corresponding to the video frames is obtained.

[0171] In a preferred example, the present application may be further configured as: a behavior confirmation module 230 for:

[0172] Determine the position detection information according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame;

[0173] When the position detection information satisfies the first condition, it is determined whether there is a cash-handling action according to the cashier's posture. If there is no cash-handling action, it is determined that there is cash-handling behavior; otherwise, there is no cash-handling behavior;

[0174] When the location detection information meets the second condition, it is determined that there is an act of privately collecting cash;

[0175] The first condition is: the cash is in the preset cashier area, the cash is outside the cash box detection frame information, and the cashier is in the preset cashier area;

[0176] The second condition is that the cash is in the preset cashier area and there is no cash box detection frame.

[0177] In a preferred example, the present application may be further configured as: a behavior confirmation module 230 for:

[0178] Determine whether the cash is in the preset cash register area according to the preset cash register area and cash detection frame information in the video frame;

[0179] If the cash is in the preset cashier area, determine whether the cash is outside the cash box detection frame based on the cash box detection frame information and the cash detection frame information;

[0180] If the cash is outside the cash box detection frame, it is determined whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame.

[0181] In a preferred example, the present application may be further configured as: a behavior confirmation module 230 for:

[0182] Determine whether there is a person in a preset customer area in the video frame;

[0183] If there is a person in the preset customer area, determine whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame;

[0184] Otherwise, it is determined that there is behavior of private collection of cash.

[0185] In a preferred example, the present application can be further configured as follows: the second condition also includes: the cash is in the preset cashier area, and the cash is outside the cash box detection frame information, and there are no customers in the preset customer area.

[0186] In a preferred example, the present application can be further configured as follows: the cashier posture includes: the shoulder joint position, the elbow joint position, the wrist joint position,

[0187] The behavior confirmation module 230 is used to:

[0188] Determine the elbow joint angle according to the shoulder joint position, elbow joint position and wrist joint position;

[0189] Determine whether there are a preset second number of elbow joint angles greater than the preset elbow joint angle in a continuous preset first number of video frames, and the preset first number is greater than the preset second number.

[0190] An electronic device is provided in an embodiment of the present application, such as Figure 8 As shown, Figure 8 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0191] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0192] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0193] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0194] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0195] Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0196] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0197] An embodiment of the present application provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0198] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0199] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A behavior recognition method, characterized in that: include: Obtain the video to be tested in the cashier monitoring area; Performing target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame, wherein the recognition information includes: personnel information, cash detection frame information and cash box detection frame information, and the personnel information includes: cashier detection frame information and cashier posture; Determine whether there is any private cash collection behavior based on the preset cashier area information in the video frame and the identification information.

2. The behavior recognition method according to claim 1, characterized in that: Performing target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame includes: The YOLOv8-Pose model is used to perform target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame.

3. The behavior recognition method according to claim 1, characterized in that: The determining whether there is a behavior of privately collecting cash according to the preset cashier area information in the video frame and the identification information includes: Determine the position detection information according to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame; When the position detection information satisfies the first condition, determining whether there is a cash-handling action according to the cashier's posture, if there is no cash-handling action, determining that there is cash-handling behavior, otherwise there is no cash-handling behavior; When the location detection information satisfies the second condition, it is determined that there is an act of privately collecting cash; The first condition is that the cash is in the preset cashier area, the cash is outside the cash box detection frame information, and the cashier is in the preset cashier area; The second condition is that the cash is in the preset cashier area and there is no cash box detection frame.

4. The behavior recognition method according to claim 3, characterized in that: According to the preset cashier area, cash detection frame information, cash box detection frame information and cashier detection frame information in the video frame, the position detection information is determined, including: Determine whether the cash is within the preset cash register area according to the preset cash register area and cash detection frame information in the video frame; If the cash is in the preset cashier area, determining whether the cash is outside the cash box detection frame according to the cash box detection frame information and the cash detection frame information; If the cash is outside the cash box detection frame, it is determined whether the cashier is in the preset cashier area based on the preset cashier area and cashier detection frame information in the video frame.

5. The behavior recognition method according to claim 4, characterized in that: Determining whether the cashier is in the preset cashier area according to the preset cashier area and the cashier detection frame information in the video frame includes: Determine whether there is a person in a preset customer area in the video frame; If there is a person in the preset customer area, determining whether the cashier is in the preset cashier area according to the preset cashier area and cashier detection frame information in the video frame; Otherwise, it is determined that there is behavior of private collection of cash.

6. The behavior recognition method according to claim 5, characterized in that: The second condition also includes: the cash is in the preset cashier area, the cash is outside the cash box detection frame information, and there is no customer in the preset customer area.

7. The behavior recognition method according to claim 3, characterized in that: The cashier's posture includes: shoulder joint position, elbow joint position, wrist joint position, Determining whether there is a reaching action according to the cashier's posture includes: Determining an elbow joint angle according to the shoulder joint position, the elbow joint position, and the wrist joint position; Determine whether there are a preset second number of elbow joint angles greater than the preset elbow joint angle in a continuous preset first number of video frames, and the preset first number is greater than the preset second number.

8. A behavior recognition device, characterized in that: include: An acquisition module is used to acquire the video to be detected in the cashier monitoring area; The recognition module is used to perform target recognition on the video frame of the video to be detected to obtain recognition information corresponding to the video frame, wherein the recognition information includes: personnel information, cash detection frame information and cash box detection frame information, and the personnel information includes: cashier detection frame information and cashier posture; The behavior confirmation module is used to determine whether there is any private cash collection behavior based on the preset cashier area information in the video frame and the identification information.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: execute the steps of the behavior recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the steps of the behavior recognition method according to any one of claims 1 to 7.

Images