Detection method, device, equipment and medium
Through the object detection model, the image frame is detected by cash and cash box to determine its position relationship, which solves the problems of high cost and omission of abnormal cash collection in the prior art, realizes automatic detection, and improves efficiency and accuracy.
Patent Information
- Application Number
- CN202411139326.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-15
Smart Images

Figure CN118968415A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of detection technology, and in particular to a detection method, device, equipment and medium. Background Art
[0002] With the diversification of payment methods and promotional activities, cashiers need to handle various payment methods and promotional activities when performing checkout, which increases the complexity of operations. When customers pay in cash, there may be abnormal cash collection situations.
[0003] At present, the existing method of identifying abnormal cash collection is usually to set up monitoring equipment and manually identify whether there is abnormal cash collection through monitoring video. This has high labor costs and is easy to miss. Therefore, how to reduce the cost of detecting abnormal cash collection and avoid missing is a problem that needs to be solved at present. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a detection method, device, equipment and medium, which can reduce the cost of detecting abnormal cash collection situations and avoid omissions. The specific scheme is as follows:
[0005] In a first aspect, the present application discloses a detection method, comprising:
[0006] Obtain image frames in the video to be detected;
[0007] Detect cash and cash box on the image frame using a target detection model;
[0008] When cash is detected, determining whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash;
[0009] If the cash is located in the preset cashier area, it is determined whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box.
[0010] Optionally, after performing cash and cash box detection on the image frame using the target detection model, the method further includes:
[0011] Track the cash in the image frame based on the first position information to obtain a tracking mark corresponding to the cash;
[0012] Accordingly, the determining whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box includes:
[0013] The distance between the cash and the cash box is calculated, and if the distance corresponding to any of the tracking marks in a preset number of consecutive image frames shows an increasing trend, it is determined that there is an abnormal cash collection situation.
[0014] Optionally, tracking the cash in the image frame based on the first position information to obtain a tracking identifier corresponding to the cash includes:
[0015] extracting a target feature vector corresponding to the cash from the image frame based on the first position information;
[0016] If the image frame is the first frame of the video to be detected, then creating tracking information corresponding to the cash, wherein the tracking information includes the target feature vector and a tracking identifier corresponding to the cash;
[0017] If the image frame is not the first frame of the video to be detected, the tracking information of the previous frame is matched with the target feature vector, and the tracking identifier of the previous frame corresponding to the cash whose tracking information is successfully matched with the target feature vector is determined as the tracking identifier of the cash in the image frame.
[0018] Optionally, the tracking information further includes the first location information, and the method further includes:
[0019] Predicting predicted position information corresponding to the cash in the next frame based on the tracking information;
[0020] After matching the tracking information based on the previous frame with the target feature vector, the method further includes:
[0021] Calculate the overlap between the predicted area corresponding to the predicted position information and the detection area corresponding to the first position information based on the predicted position information corresponding to the unmatched tracking information and the first position information corresponding to the target feature vector that is not matched successfully;
[0022] Determine the cash tracking mark in the previous frame corresponding to the detection area whose overlap degree meets the preset overlap degree condition as the cash tracking mark in the image frame;
[0023] A tracking mark of the cash in the image frame is created for the cash corresponding to the detection area whose overlap does not satisfy the preset overlap condition.
[0024] Optionally, judging whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash includes:
[0025] Determine the center point of the image area corresponding to the cash based on the first position information;
[0026] Determine the directed area of a triangle formed by the center point and any two adjacent vertices of the preset checkout area;
[0027] Based on the directed area of each of the triangles, it is determined whether the cash is located in a preset cashier area in the image frame.
[0028] Optionally, the determining whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box includes:
[0029] Determining whether the image area corresponding to the cash is located within the image area corresponding to the cash box based on the first position information and the second position information of the cash box;
[0030] If the image area corresponding to the cash is within the image area corresponding to the cash box, it is determined that there is no abnormal cash collection situation.
[0031] Optionally, also include:
[0032] Acquire the target detection model;
[0033] The target detection model is trained using an image training sample set, and the image training samples in the image training sample set include the location and category labeling information of cash and cash boxes.
[0034] In a second aspect, the present application discloses a detection device, comprising:
[0035] An image frame acquisition module is used to acquire image frames in the video to be detected;
[0036] An image detection module, used for performing cash and cash box detection on the image frame using a target detection model;
[0037] An area determination module, configured to determine whether the cash is located in a preset cashier area in the image frame based on first position information of the cash when the cash is detected;
[0038] The abnormality judgment module is used to determine whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box if the area judgment module determines that the cash is located in the preset cashier area.
[0039] In a third aspect, the present application discloses an electronic device, including a memory and a processor, wherein:
[0040] The memory is used to store the computer program;
[0041] The processor is used to execute the computer program to implement the aforementioned detection method.
[0042] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned detection method when executed by a processor.
[0043] It can be seen from the above scheme that the present invention provides a detection method, including: acquiring image frames in a video to be detected; using a target detection model to perform cash and cash box detection on the image frames; when cash is detected, judging whether the cash is located in a preset cash register area in the image frame based on the first position information of the cash; if the cash is located in the preset cash register area, determining whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box.
[0044] It can be seen that the beneficial effects of the present application are: cash and cash box detection are performed on the image frames in the video to be detected through the target detection model. When cash is detected and in the pre-divided checkout area, it is determined whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box, thereby realizing the automation of abnormal cash collection detection, reducing the cost of abnormal cash collection detection and avoiding omissions.
[0045] Correspondingly, a detection device, equipment and readable storage medium provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0047] Figure 1 A flow chart of a detection method provided in an embodiment of the present application;
[0048] Figure 2 A cash collection abnormality detection flow chart provided in an embodiment of the present application;
[0049] Figure 3 A flow chart of another detection method provided in an embodiment of the present application;
[0050] Figure 4 A specific detection method process provided in the embodiment of the present application;
[0051] Figure 5 A schematic diagram of the structure of a detection device provided in an embodiment of the present application;
[0052] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0054] See also Figure 1 As shown, the embodiment of the present application discloses a detection method, comprising:
[0055] Step S11: Acquire image frames in the video to be detected.
[0056] The detection method provided in this application can be applied to monitoring equipment or to a backend server, and the backend server obtains the video to be detected collected by the monitoring equipment. The monitoring equipment can be deployed based on user needs, and the image collection range covers the cashier area.
[0057] Step S12: Utilize the target detection model to perform cash and cash box detection on the image frame.
[0058] In the embodiment of the present application, the target detection model can be obtained; wherein the target detection model is obtained by training using an image training sample set, and the image training samples in the image training sample set include the location of cash and cash boxes and category labeling information. It is understandable that the target detection model can be obtained before obtaining the image frame in the video to be detected, and it is not necessary to obtain it separately for each image frame.
[0059] It is understandable that the target detection model can be used to detect whether there is cash and a cash box in the image frame, and determine the location information of the cash and the cash box.
[0060] Among them, the target detection model can be understood as a model that automatically detects and locates targets in images or videos through computer vision technology. The target detection model can be a two-stage target detection model or a one-stage target detection model. The two-stage target detection model is divided into two main stages: the first stage: generating candidate target areas. The second stage: classifying and refining the positioning of candidate areas. In this stage, each candidate area will be classified and judged in more detail, and its location information will be corrected to obtain the final target detection result. The one-stage target detection model can be understood as a model that directly inputs an image and simultaneously outputs the object category and corresponding location information in the image without first extracting the candidate area. Objects that may exist in various positions in the image can be captured through preset anchor frames, and these anchor frames can be classified and regressed.
[0061] In this embodiment, a general multi-category target detection model can be flexibly selected according to specific usage requirements, such as a two-stage target detection model Faster R-CNN (i.e., Faster Region-Convolutional Neural Networks, fast region-based convolutional neural network), SPPNet (Spatial Pyramid Pooling Network), or a one-stage target detection model YOLOv3 / v4 / v5 (i.e., You Only Look Once, third edition / fourth edition / fifth edition), SSD (i.e., Single Shot MultiBox Detector), and RetinaNet (a target detection network based on deep learning), etc., and the public pre-trained networks of these models are selected as the basic network of the target detection model, and the pre-trained network is trained using a pre-labeled cash box data set. Obtain the video to be detected, read each frame in turn, and use the trained target detection model to obtain the cash, cash box, and their respective position information in each frame of the video to be detected.
[0062] Furthermore, in an optional implementation, after the cash and cash box detection is performed on the image frame using the target detection model, the method may also include: tracking the cash in the image frame based on the first position information to obtain a tracking mark corresponding to the cash.
[0063] Among them, the target feature vector corresponding to the cash can be extracted from the image frame based on the first position information; if the image frame is the first frame of the video to be detected, the tracking information corresponding to the cash is created, wherein the tracking information includes the target feature vector and the tracking identifier corresponding to the cash; if the image frame is not the first frame of the video to be detected, the tracking information of the previous frame is matched with the target feature vector, and the tracking identifier of the previous frame corresponding to the cash in which the tracking information and the target feature vector are successfully matched is determined as the tracking identifier of the cash in the image frame. It can be understood that this embodiment can use the historical target feature vector in the tracking information to match the target feature vector extracted from the image frame. For example, the Euclidean distance, cosine distance, etc. can be used to determine whether the feature vectors match. In an optional embodiment, the target feature vectors of a preset number of historical image frames can be retained in each tracking information, such as the target feature vectors of the past 5 frames. A cost matrix is constructed for each tracking information. The number of rows in the cost matrix is the number of target feature vectors in the tracking information, and the number of columns is the number of cash detected in the image frame. For example, if 5 cash are detected, there are 5 detection frames, the column is the number of detection frames, and the element is the cosine distance between the target feature vector in the tracking information and the target feature line in the current image frame. Matching based on the cost matrix can be performed, for example, using the Hungarian algorithm for fast matching.
[0064] Further, the tracking information also includes the first position information. The embodiment of the present application may also include: predicting the predicted position information corresponding to the cash in the next frame based on the tracking information; after matching the tracking information of the previous frame with the target feature vector, further including: calculating the overlap between the predicted area corresponding to the predicted position information and the detection area corresponding to the first position information based on the predicted position information corresponding to the unmatched tracking information and the first position information corresponding to the unmatched target feature vector; determining the tracking mark of the cash in the previous frame corresponding to the detection area whose overlap meets the preset overlap condition as the tracking mark of the cash in the image frame; creating the tracking mark of the cash in the image frame for the cash corresponding to the detection area whose overlap does not meet the preset overlap condition, specifically, creating the corresponding tracking information. It can be understood that if the detection area does not match the predicted area, it indicates that a new cash target appears. The preset overlap condition can be that the overlap is higher than a preset threshold, that is, the detection area matches the predicted area. Among them, Kalman filtering can be used to predict the predicted position information corresponding to the cash in the next frame. Among them, the overlap can be calculated using IOU (i.e., Intersection over Union).
[0065] In addition, the embodiment of the present application can also create a status identifier for the tracking information. The status identifier corresponding to the newly created tracking identifier is an unconfirmed status. After matching the detection area (ie, the detection box), it is changed to a confirmed status. For tracking information that does not meet the preset overlap condition, if the status identifier is a confirmed status, it can be retained based on participating in the tracking of the next frame. If the status identifier is an unconfirmed status, the tracking information is marked as a deleted status. Furthermore, in an optional embodiment, the tracking information in the confirmed status can participate in the matching with the target feature vector and the matching based on the overlap; the tracking information in the unconfirmed status can only participate in the matching based on the overlap.
[0066] Step S13: When cash is detected, it is determined whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash.
[0067] Moreover, the embodiments of the present application can divide the cashier area from the image frame. For example, before acquiring the image frame in the video to be detected, a frame of image is captured, and in response to the user marking operation, the cashier area is determined based on the cashier area range corresponding to the user marking operation. In this way, the cashier area can be determined based on the cashier area range for subsequent image frames. That is, after installing the monitoring device, the user can set the cashier area based on the image captured by the monitoring device. Furthermore, in an optional embodiment, the cashier area and the customer area can be divided. The customer area can be the entire area outside the cashier area in the image frame, or it can be the customer area determined based on the customer area range corresponding to the user marking operation, or it can be a part of the entire area outside the cashier area in the image frame.
[0068] In the embodiment of the present application, the center point of the image area corresponding to the cash can be determined based on the first position information; and it can be determined whether the center point is located in a preset cashier area in the image frame.
[0069] In an optional implementation, the directed area of the triangle formed by the center point and any two adjacent vertices of the preset cashier area can be determined; based on the directed area of each triangle, it is determined whether the cash is located in the preset cashier area in the image frame. If the directed areas of all the triangles are positive or negative, it is determined that the cash is located in the preset cashier area in the image frame, otherwise it is determined that the cash is not located in the preset cashier area in the image frame. Any two adjacent vertices can be clockwise adjacent.
[0070] For example, the center point coordinates (CeX, CeY) of the image area corresponding to the cash (that is, the detection box corresponding to the cash);
[0071] CeX=(x1+x2) / 2;
[0072] CeY=(y1+y2) / 2;
[0073] Among them, (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and lower right corner of the detection box respectively.
[0074] The coordinates of the four vertices of the cash register area (clockwise) are (AX, AY), (BX, BY), (CX, CY), and (DX, DY).
[0075] a=(AX-DX)*(CeY-DY)-(AY-DY)*(CeX-DX);
[0076] b=(BX-AX)*(CeY-AY)-(BY-AY)*(CeX-AX);
[0077] c=(CX-BX)*(CeY-BY)-(CY-BY)*(CeX-BX);
[0078] d=(DX-CX)*(CeY-CY)-(DY-CY)*(CeX-CX);
[0079] Calculate the directed areas of the four triangles and check whether the four triangles are in the same direction (that is, if a, b, c, and d are all positive or negative, they are in the same direction, and the detection box corresponding to the cash is in the cashier area; otherwise, they are not in the same direction, and the detection box corresponding to the cash is not in the cashier area).
[0080] It is understandable that if multiple cash locations are detected, it is determined based on the first position information of each cash location whether each cash location is located in a preset cashier area in the image frame.
[0081] If the cash is not in the preset cashier area, it is determined that no abnormal cash collection occurs.
[0082] Step S14: If the cash is in the preset cashier area, determine whether there is any abnormal cash collection situation based on the positional relationship between the cash and the cash box.
[0083] In this embodiment, it can be determined whether the image area corresponding to the cash is located in the image area corresponding to the cash box based on the first position information and the second position information of the cash box; if the image area corresponding to the cash is located in the image area corresponding to the cash box, it is determined that there is no abnormal cash collection situation. That is, it is determined whether the detection box corresponding to the cash is within the detection box corresponding to the cash box.
[0084] Furthermore, this embodiment can also determine whether there is a cashbox in the preset checkout area before determining whether the image area corresponding to the cash is located within the image area corresponding to the cashbox based on the first position information and the second position information of the cashbox. If so, it triggers the determination of whether the image area corresponding to the cash is located within the image area corresponding to the cashbox based on the first position information and the second position information of the cashbox. If there is no cashbox in the preset checkout area, it is determined that there is an abnormal cash collection situation.
[0085] Furthermore, the present embodiment may also calculate the distance between the cash and the cash box, and if the distance corresponding to any of the tracking marks in a preset number of consecutive image frames shows an increasing trend, it is determined that there is an abnormal cash collection situation.
[0086] Furthermore, this embodiment can trigger the step of calculating the distance between the cash and the cash box when the image area corresponding to the cash is not within the image area corresponding to the cash box. If the distance corresponding to any tracking mark in a preset number of consecutive image frames shows an increasing trend, it is determined that there is an abnormal cash collection situation.
[0087] For further information, see Figure 2 As shown, Figure 2 A flow chart for detecting abnormal cash collection provided in an embodiment of the present application. Determine whether there is a cash detection frame in the current frame image, if not, end the process. Filter the cash detection frame for the customer area and the cashier area: if the cash is in the customer area, a normal event is reported. If the cash is in the cashier area, determine whether there is a cash box detection frame in the cashier area. Furthermore, if cash appears in the cashier area and the cash box is not detected, report the private collection of cash. If the cash box is detected, compare the positional relationship between the cash and the cash box detection frame. If the cash is in the cash box detection frame, report the normal event. Otherwise, calculate the distance L between the center point of the cash detection frame and the center point of the cash box detection frame, and compare L of the three frames with the same tracking mark. If L continues to increase, report the private collection of cash, otherwise report the normal event.
[0088] It can be seen that the embodiment of the present application performs cash and cash box detection on the image frames in the video to be detected through the target detection model. When cash is detected and in the case that the cash is in the pre-divided checkout area, it determines whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box, thereby realizing the automation of abnormal cash collection detection, which can reduce the cost of abnormal cash collection detection and avoid omissions.
[0089] For further information, see Figure 3 As shown, Figure 3 This is a flow chart of another detection method disclosed in an embodiment of the present application.
[0090] The real-time video stream is used as the video to be detected, and each frame is read in turn to divide the cashier area and the customer area. The detection module is used to obtain the cash, cash box and cash and cash box location information of each frame. Among them, the detection module uses the target detection model to detect each frame of the image. The target detection model is trained using the cash cash box data set. The cash cash box data set contains image data labeled with cash, cash box location and category information. The target detection model can be a one-stage target detection model or a two-stage target detection model. The one-stage target detection model can be understood as a model that outputs the location and type of the target at one time through the input image. Its advantage is that it is fast and can meet the requirements of real-time detection, such as SSD, YOLO series, etc. The two-stage target detection model obtains more accurate positioning by adding prior knowledge. It can use candidate boxes, ROI (i.e., Region of Interest) pooling layers, RPN (i.e., Region Proposal Network), ROIAlign (i.e., Region of Interest alignment) layers, etc. Through candidate boxes and multi-scale detection methods, detection of targets of different sizes is achieved. In the two-stage target detection model, RPN generates the offset of the candidate box through the neural network, and then filters the regression box to initially obtain a detection box with a relatively high confidence. The ROIAlign layer performs more precise positioning and classification on the preliminary detection results. The deployment of mobile terminals (i.e. monitoring devices) places computing and storage resources on platforms such as embedded systems to reduce the burden and latency of cloud computing. Therefore, the deployment of mobile terminals is mainly based on single-stage target detection models. Moreover, with the development of research, many single-stage target detection models have surpassed some two-stage models in terms of target detection result accuracy while maintaining speed.
[0091] Furthermore, the tracking module can be used to output the tracking identifier corresponding to the cash. For example, the cash position information obtained by the target detection model in the video frame can be used as input, passed into the target tracking model, and the ID (i.e., tracking identifier) corresponding to each target can be output. The target tracking model is a model that continuously tracks the target in the video sequence through computer vision technology. The main task of the target tracking model is to accurately locate and track the target in the video sequence, and maintain the continuity of tracking in complex situations such as target occlusion, target disappearance, and target reappearance. The target tracking model can be a tracking model based on feature extraction and a tracking model based on deep learning. Tracking model based on feature extraction: Track the target by extracting the appearance features (such as color, texture, etc.) and motion features (such as speed, direction, etc.) of the target. Feature extraction methods include HOG (i.e., Histogram of Oriented Gradients), SIFT (i.e., Scale-Invariant Feature Transform), etc. The advantage is high computational efficiency and is suitable for real-time tracking scenarios. The tracking model of deep learning uses a deep neural network to learn the feature representation of the target, thereby achieving accurate tracking of the target. Commonly used deep learning tracking models include Siamese networks (i.e. twin neural networks), Deepsort (i.e. Deep Simple Online and Realtime Tracking, simple online real-time tracking based on deep learning), etc. The advantage is that feature representation can be automatically learned with high accuracy and robustness. For example, the target tracking model can be a Deepsort model. Specifically, the following steps may be included: 1) Target feature extraction: extract the feature representation of the target from the target position detected in the first frame (the position corresponding to the cash). A deep learning model such as CNN (i.e. Convolutional Neural Networks) can be used to extract the feature vector of the target. For example, a pre-trained image classification model such as ResNet (i.e. Residual Network), VGG (i.e. Visual Geometry Group, Visual Geometry Group Network), etc. or a feature extraction network dedicated to target tracking can be used to extract the feature vector of the target. 2) Target matching: In subsequent video frames, the target feature vector is matched with the target feature vector in the previous frame to determine the position of the target object in the current frame. The Hungarian algorithm can be used to solve the matching problem between multiple targets. Various distance measurement methods can be used for matching, such as Euclidean distance, cosine similarity, etc. 3) Target trajectory management: According to the results of target matching, the target trajectory is managed and updated. The Kalman filter is used to predict and update the target position.
[0092] Furthermore, the post-processing module is called to compare whether the cash is in the cashier area or the customer area, and the area relationship with the cash box to determine whether there is an incident of private cash collection.
[0093] The solution provided in the embodiment of the present application can be deployed on a mobile terminal (such as a monitoring device) to monitor the cash register scene in real time. The cash register scene is divided into a cash register area and a customer area, relying on a detection module, a tracking module, and a post-processing module. Among them, the detection module is obtained by targeted training of a general pre-trained model of a multi-target category on an actual cash register data set to obtain cash and cash boxes and detection results; the tracking module can use a deepsort algorithm to track all cash that appears; the post-processing module compares the area where the cash is located and its positional relationship with the cash box to determine whether there is an incident of private cash collection. In this way, real-time detection can be performed on the cash register scene, providing automatic and intelligent cash register risk management and control, as well as convenient and comprehensive standardized management of employee services and service quality evaluation.
[0094] See also Figure 4 As shown, Figure 4 A specific detection method process disclosed in an embodiment of the present application.
[0095] The target detection model is used to output the detection frame information of cash and cash boxes, that is, the location information. For example, the Yolov5 detection model can be used to output the detection frame information of cash and cash boxes. Using YOLOv5 as the detection model, on the basis of the pre-trained model, the cashier scene data set is collected for annotation. The annotation format is as follows: (label, x, y, w, h), where label represents the category, x, y is the center point divided by the width and height of the picture, and w, h is the width and height of the box divided by the width and height of the picture. The obtained cash detection results are stored in the set as (x1, y1, x2, y2). If there is a cash box detection result, its location information is saved, where (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and lower right corner of the detection box, respectively. In this way, general multi-category target detection is used for target detection of specific categories in specific scenes. Using yolov5 as the detection model, the model has fewer parameters and faster inference speed, and can obtain better detection results at a lower training cost.
[0096] Furthermore, the target tracking model is used to track cash and output ID information (i.e., tracking identification). For example, the Deepsort model is used to track cash and output the ID information of each target. This embodiment can adopt the following steps: (1) Record the detection results of the first frame and create its corresponding Tracks (i.e., tracking information). The Kalman filter is initialized, and Tracks is used to predict the box corresponding to the detection result of the first frame, that is, the predicted box of the next frame. (2) For non-first frames, the cosine distance calculation can be performed between the feature vector corresponding to the box of the target detection of this frame and the feature vector saved in the Tracks of the previous frame, and the cosine distance is used as the cost to generate a cost matrix. (3) All the obtained cost matrices are brought into the Hungarian algorithm to obtain linear matching results. There are three types of results. The first is that the Tracks matching fails (Unmatched Tracks), and the unmatched Tracks are directly deleted; the second is that the Detections (i.e., the detection box) does not match (Unmatched Detections), which are initialized and called new Tracks (new Tracks); the third is that the detection box and the predicted box are successfully paired, which means that the previous frame and the next frame are tracked successfully, and the corresponding Detections are updated through the Kalman filter to their corresponding Tracks variables. (4) Repeat steps (2)-(3) until the end of the video frame. In addition, in an optional implementation, when the match fails, the prediction box and the detection box can be used to perform IOU matching one by one to further determine whether the match is successful. For Tracks that have not been successfully matched with IOU, if they are in the confirmed state, they will be retained and continue to participate in the predicted Tracks of the next frame; if they are in the unconfirmed state, the Tracks will be marked as deleted; for the detection box that has not been successfully matched with IOU, new Tracks will be created for them, and the new Tracks will be assigned values according to the data of the detection box. Each Track has two states: confirmed and unconfirmed. Tracks in the confirmed state can participate in cascade matching and IOU matching; tracks in the unconfirmed state can only participate in IOU matching. The state of the initial Tracks (newly created Tracks) is unconfirmed.
[0097] This embodiment uses a cascade matching data association method, that is, the predicted track Tracks are matched with the detections in the current frame through the Hungarian algorithm. The Hungarian algorithm is a combinatorial optimization algorithm used to solve the assignment problem, which can be divided into four steps. The first and second steps will be implemented once, and the third and fourth steps will be repeatedly implemented to obtain the optimal allocation result. The algorithm requires an n*n matrix as input, which can only be 0 or a positive number. The entire workflow of the algorithm is as follows: (1) Find the element with the minimum value in each row of the matrix, and subtract this element value from all elements in the row. (2) Find the element with the minimum value in each column of the matrix, and subtract this element value from all elements in the column. (3) Cover all 0 elements of the matrix with horizontal and vertical lines, requiring the minimum number of lines. If n lines are required, it means that there is an optimal solution among these 0s, and the algorithm ends. If the number of required lines is less than n, execute the fourth step. (4) In the matrix of step 3, find the smallest element in the row and column that is not covered by the line, denoted as k. Subtract k from all elements that have not been covered, and add k to the elements that have been covered twice.
[0098] In this embodiment, the tracking ID of the cash and the location information obtained by target detection can be stored in a set as (x1, y1, x2, y2, m), where m represents the tracking ID.
[0099] Furthermore, this embodiment can compare the positional relationship between cash and the cashier area and the customer area, and save the position information. The obtained position information is stored in a set as (x1, y1, x2, y2, m, p, q), where p and q represent the positional relationship between the detection frame and the cashier area and the customer area, respectively. The positional relationship between the cash in the cashier area and the cashbox detection frame is compared based on the cashbox detection frame information. The cashier event is determined based on the comparison result. In this way, by adding the cashbox detection item as well as the customer area and the cashier area, through regional filtering and detection frame center point distance calculation, the cash private collection event can be effectively determined.
[0100] It can be seen that the embodiment of the present application realizes real-time cash private collection detection through the detection module, tracking module and post-processing module. The detection model has few parameters and fast reasoning, which meets the real-time requirements; the tracking module adopts the deepsort algorithm, which has good cash tracking effect and stable tracking ID; through the customer area cashier area filtering, the distance between the cash detection frame and the cash box detection frame center point is calculated to determine whether the cash is privately collected. It can perform real-time detection for cashier scenes, provide automatic and intelligent cashier risk management, as well as convenient and comprehensive standardized management of employee services and service quality evaluation.
[0101] See also Figure 5 As shown, the embodiment of the present application discloses a detection device, including:
[0102] An image frame acquisition module 11 is used to acquire image frames in the video to be detected;
[0103] An image detection module 12, used for performing cash and cash box detection on the image frame using a target detection model;
[0104] An area determination module 13 is used to determine whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash when the cash is detected;
[0105] The abnormality judgment module 14 is used to determine whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box if the area judgment module determines that the cash is located in the preset cashier area.
[0106] It can be seen that the embodiment of the present application performs cash and cash box detection on the image frames in the video to be detected through the target detection model. When cash is detected and in the case that the cash is in the pre-divided checkout area, it determines whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box, thereby realizing the automation of abnormal cash collection detection, which can reduce the cost of abnormal cash collection detection and avoid omissions.
[0107] The detection device further includes a tracking module, configured to track the cash in the image frame based on the first position information to obtain a tracking mark corresponding to the cash;
[0108] Correspondingly, the abnormality determination module 14 is specifically used to calculate the distance between the cash and the cash box, and if the distance corresponding to any of the tracking marks in a preset number of consecutive image frames shows an increasing trend, it is determined that there is an abnormal cash collection situation.
[0109] Furthermore, the tracking module is specifically used to extract the target feature vector corresponding to the cash from the image frame based on the first position information; if the image frame is the first frame of the video to be detected, then create tracking information corresponding to the cash, wherein the tracking information includes the target feature vector and a tracking identifier corresponding to the cash; if the image frame is not the first frame of the video to be detected, then match the target feature vector based on the tracking information of the previous frame, and determine the tracking identifier of the previous frame corresponding to the cash in which the tracking information is successfully matched with the target feature vector as the tracking identifier of the cash in the image frame.
[0110] Optionally, the tracking information further includes the first location information, and the tracking module is further configured to:
[0111] Predicting predicted position information corresponding to the cash in the next frame based on the tracking information;
[0112] After matching the tracking information based on the previous frame with the target feature vector, the method further includes:
[0113] Calculate the overlap between the predicted area corresponding to the predicted position information and the detection area corresponding to the first position information based on the predicted position information corresponding to the unmatched tracking information and the first position information corresponding to the target feature vector that is not matched successfully;
[0114] Determine the cash tracking mark in the previous frame corresponding to the detection area whose overlap degree meets the preset overlap degree condition as the cash tracking mark in the image frame;
[0115] A tracking mark of the cash in the image frame is created for the cash corresponding to the detection area whose overlap does not satisfy the preset overlap condition.
[0116] Optionally, the area determination module 13 specifically includes:
[0117] A center point determination submodule, configured to determine a center point of an image area corresponding to the cash based on the first position information;
[0118] A directed area determination submodule, used to determine the directed area of a triangle formed by the center point and any two adjacent vertices of the preset checkout area;
[0119] The area determination submodule is used to determine whether the cash is located in a preset cashier area in the image frame based on the directed area of each of the triangles.
[0120] Optionally, the abnormality judgment module 14 is specifically used to: judge whether the image area corresponding to the cash is located within the image area corresponding to the cash box based on the first position information and the second position information of the cash box; if the image area corresponding to the cash is located within the image area corresponding to the cash box, it is determined that there is no abnormality in cash collection.
[0121] Furthermore, the device also includes:
[0122] The model acquisition module is used to acquire the target detection model; wherein the target detection model is trained using an image training sample set, and the image training samples in the image training sample set include the location of cash and cash boxes and category labeling information.
[0123] See also Figure 6As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; and the processor 21 is used to execute the computer program, the detection method disclosed in the above embodiment.
[0124] For the specific process of the above detection method, please refer to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.
[0125] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.
[0126] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0127] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the detection method disclosed in the aforementioned embodiment when executed by a processor.
[0128] For the specific process of the above detection method, please refer to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.
[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0130] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0131] The above is a detailed introduction to a detection method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A detection method, characterized in that: include: Obtain image frames in the video to be detected; Detect cash and cash box on the image frame using a target detection model; When cash is detected, determining whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash; If the cash is located in the preset cashier area, it is determined whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box.
2. The detection method according to claim 1, characterized in that: After the cash and cash box detection is performed on the image frame using the target detection model, the method further includes: Track the cash in the image frame based on the first position information to obtain a tracking mark corresponding to the cash; Accordingly, the determining whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box includes: The distance between the cash and the cash box is calculated, and if the distance corresponding to the tracking mark of any cash in a preset number of consecutive image frames shows an increasing trend, it is determined that there is an abnormal cash collection situation.
3. The detection method according to claim 2, characterized in that: Tracking the cash in the image frame based on the first position information to obtain a tracking mark corresponding to the cash includes: extracting a target feature vector corresponding to the cash from the image frame based on the first position information; If the image frame is the first frame of the video to be detected, then creating tracking information corresponding to the cash, wherein the tracking information includes the target feature vector and a tracking identifier corresponding to the cash; If the image frame is not the first frame of the video to be detected, the tracking information of the previous frame is matched with the target feature vector, and the tracking identifier of the previous frame corresponding to the cash whose tracking information is successfully matched with the target feature vector is determined as the tracking identifier of the cash in the image frame.
4. The detection method according to claim 3, characterized in that: The tracking information also includes the first location information, and the method further includes: Predicting predicted position information corresponding to the cash in the next frame based on the tracking information; After matching the tracking information based on the previous frame with the target feature vector, the method further includes: Calculate the overlap between the predicted area corresponding to the predicted position information and the detection area corresponding to the first position information based on the predicted position information corresponding to the unmatched tracking information and the first position information corresponding to the target feature vector that is not matched successfully; Determine the cash tracking mark in the previous frame corresponding to the detection area whose overlap degree meets the preset overlap degree condition as the cash tracking mark in the image frame; A tracking mark of the cash in the image frame is created for the cash corresponding to the detection area whose overlap does not satisfy the preset overlap condition.
5. The detection method according to claim 1, characterized in that: The determining whether the cash is located in a preset cashier area in the image frame based on the first position information of the cash includes: Determine the center point of the image area corresponding to the cash based on the first position information; Determine the directed area of a triangle formed by the center point and any two adjacent vertices of the preset checkout area; It is determined whether the cash is located in a preset cashier area in the image frame based on the directed area of each of the triangles.
6. The detection method according to claim 1, characterized in that: The determining whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box includes: Determining whether the image area corresponding to the cash is located within the image area corresponding to the cash box based on the first position information and the second position information of the cash box; If the image area corresponding to the cash is within the image area corresponding to the cash box, it is determined that there is no abnormal cash collection situation.
7. The detection method according to any one of claims 1 to 6, characterized in that: Also includes: Acquire the target detection model; The target detection model is trained using an image training sample set, and the image training samples in the image training sample set include the location and category labeling information of cash and cash boxes.
8. A detection device, characterized in that: include: An image frame acquisition module is used to acquire image frames in the video to be detected; An image detection module, used for performing cash and cash box detection on the image frame using a target detection model; An area determination module, configured to determine whether the cash is located in a preset cashier area in the image frame based on first position information of the cash when the cash is detected; The abnormality judgment module is used to determine whether there is an abnormal cash collection situation based on the positional relationship between the cash and the cash box if the area judgment module determines that the cash is located in the preset cashier area.
9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the detection method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method and apparatus for detecting suspicious activity using video analysis
CN101268478A
Cashier loss prevention method and system based on cloud computing
CN106558061A
Cashier loss prevention method and system
CN113537034A
Method and device for detecting object abnormity, electronic equipment and storage medium
CN113947591A
Cashier loss prevention method and device, equipment and storage medium
CN115880611A
Cited By
Behavior recognition method and device, equipment and medium
CN119206873A
Method and system for identifying private cashier, computer equipment and storage medium
CN119399707A
