Self-checkout risk identification method, device and system
By acquiring video information from self-checkout devices and identifying user scanning behavior and product information, a dual risk identification method is adopted to solve the problem of theft in self-checkout devices, achieving automated risk identification and cost reduction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA (CHINA) CO LTD
- Filing Date
- 2022-01-17
- Publication Date
- 2026-05-19
AI Technical Summary
Self-checkout machines are susceptible to theft, whether intentional or unintentional, by customers, leading to high loss prevention costs for merchants. Existing technologies are insufficient to effectively identify and mitigate these risks.
By acquiring video information of customers in the self-checkout area, the system identifies users' scanning behavior and product information, employing a dual risk identification approach, including abnormal behavior identification and unscanned product identification, and outputs the risk identification results.
It has enabled automated risk identification, reduced merchants' manual loss prevention costs, improved the accuracy and efficiency of risk identification, and simplified the risk detection process.
Smart Images
Figure CN114529850B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a risk identification method, apparatus and system for self-service checkout. Background Technology
[0002] With the continuous breakthroughs and mature implementation of information technology and artificial intelligence, the new retail industry is also constantly developing towards convenience and intelligence. For example, many supermarkets and stores at home and abroad have adopted self-checkout equipment, allowing customers to settle their bills themselves. This not only provides convenience for customers but also saves merchants a lot of labor costs for cashiers, thereby increasing the overall net profit of operations.
[0003] However, self-checkout also exposes merchants to serious payment theft problems. Customers using self-checkout may intentionally fail to scan barcodes, swap product labels for scanning, or pretend to scan, while unintentionally missing scans due to failed scans or excessive purchases. This places immense pressure on merchants' loss prevention efforts and incurs high labor costs. Therefore, there is an urgent need for an automated risk identification method for self-checkout to reduce merchants' loss prevention costs. Summary of the Invention
[0004] In view of this, this application provides a risk identification method, device and system for self-service checkout, thereby automatically identifying risks in self-service checkout and reducing loss prevention costs for merchants.
[0005] This application provides the following solution:
[0006] According to the first aspect, a risk identification method for self-service checkout is provided, including:
[0007] Obtain video information of customers in the self-checkout area, identify abnormal behavior of users scanning goods in the video information, and determine the first risk information based on the identification results;
[0008] If payment success information is obtained, the unscanned product information is determined based on the product information identified from the video information and the scanned product information; second risk information is determined based on the unscanned product information.
[0009] Based on the first risk information and the second risk information, output the risk identification result.
[0010] According to an embodiment of this application, before identifying abnormal behavior in the user's scanning behavior of goods in the video information, the method further includes:
[0011] If a customer is detected entering the self-checkout area in the video information, the detection status is set to "not in use and there is a customer in front of the device," the product information in the video information is identified, and the attribute of each product is initialized to "not scanned"; or,
[0012] If scanning start information is obtained when the device is not in use and there is a customer in front of it, the detection status is set to "in use". In the "in use" status, the step of identifying abnormal behavior of the user scanning the product in the video information is executed, and the attribute of the product that has been successfully scanned is set to "scanned".
[0013] According to an embodiment of this application, identifying product information in the video information includes:
[0014] Based on the product trajectory information and product feature information in the video information, the information of each product is identified.
[0015] According to an embodiment of this application, the method further includes: before detecting that a customer has entered the self-checkout area in the video information, setting the detection status to unused and no customer in front of the device;
[0016] If a product is detected at the self-checkout device from the video information when the device is not in use and there are no customers in front of it, an alert message will be output.
[0017] According to an embodiment of this application, the method further includes:
[0018] If you receive information that you have entered the payment page, set the detection status to "Payment in Progress";
[0019] If, while in the payment process, the payment success information is obtained and the video information indicates that the customer has not left the self-service checkout area, then the detection status is set to "payment completed and customer has not left".
[0020] The step of determining unscanned product information based on product information identified from the video information and scanned product information is executed while the payment is completed and the customer has not left.
[0021] According to an embodiment of this application, the method further includes:
[0022] If, after payment is completed and the customer has not left, a new customer is detected entering the self-checkout area from the video information, the detection status is set to "not in use and there is a customer in front of the device"; and / or,
[0023] If, after payment is completed and the customer has not left, the video information detects that the customer has left the self-checkout area and no new customer has entered the self-checkout area, then the detection status is set to "not in use and no customer in front of the device".
[0024] According to an embodiment of this application, the step of identifying abnormal behavior in the user's scanning behavior of goods in the video information and determining the first risk information based on the identification result includes:
[0025] The product location information and / or human posture information in the video information are identified, and at least one of the following is performed based on the identification result:
[0026] If no successful scanning message is received within the set time period after the product's trajectory passes through the area in front of the scanning device, it is determined that there is a risk that the product has passed through the scanning area without being scanned.
[0027] If a successful scan is obtained, the product features corresponding to the product label information contained in the successful scan are compared with the product features identified from the video information. If the similarity is lower than the preset similarity threshold, it is determined that there is a risk of product label swapping during scanning.
[0028] If the product's trajectory is in the area in front of the scanning device and the customer does not receive a successful scan notification within the set time period for multiple scans of the product, then there is a risk of multiple failed scans of the product.
[0029] If the product's trajectory does not pass through the area in front of the scanning device and an abnormal movement trajectory is observed, and no information indicating that the product was successfully scanned is received, then there is a risk of abnormal movement of the product that has not been scanned.
[0030] If a product is successfully scanned and then removed from the purchase list, or if the product count is 0, then there is a risk that the product will be removed after scanning.
[0031] If no payment success information is received after scanning the code and the video information detects that the customer has left the self-service checkout area, then it is determined that there is a risk of leaving without paying.
[0032] According to an embodiment of this application, determining the second risk information based on the unscanned product information includes:
[0033] The similarity between scanned and unscanned products is calculated, and unscanned products with a similarity greater than a preset similarity threshold are filtered out. The similarity is calculated based on the product's trajectory information and / or feature information.
[0034] If unscanned products still exist after filtering, then a second risk is identified.
[0035] According to an embodiment of this application, in one feasible manner, the risk identification result is output based on the first risk information and the second risk information, including:
[0036] Based on the level of the first risk information and the level of the second risk information, a recording, reminder, or warning method appropriate to the corresponding level is adopted.
[0037] According to the second aspect, a risk identification device for self-checkout is also provided, comprising:
[0038] The video acquisition unit is configured to acquire video information of customers in the self-checkout area;
[0039] The first risk analysis unit is configured to identify abnormal behavior of users scanning goods in the video information and determine the first risk information based on the identification results.
[0040] The second risk analysis unit is configured to, if payment success information is obtained, determine unscanned product information based on product information identified from the video information and scanned product information; and determine second risk information based on the unscanned product information.
[0041] The risk result output unit is configured to output a risk identification result based on the first risk information and the second risk information.
[0042] According to the third aspect, a risk identification system for self-service checkout is provided, including video capture equipment, self-service checkout equipment, monitoring equipment, and the risk identification device provided in the second aspect above:
[0043] The video capture device is configured to capture video information of customers in the self-checkout area and provide the video information to the risk identification device.
[0044] The self-service checkout equipment includes a barcode scanner, a shopping counter, and a payment device;
[0045] The scanning device is configured to scan the product under the operation of the customer and provide the scanned product information to the risk identification device.
[0046] The shopping counter is used to store goods;
[0047] The payment device is configured to provide an interactive payment interface to the customer and send payment-related information to the risk identification device based on the customer's payment operation on the interface.
[0048] The monitoring equipment is configured to acquire and display the risk identification results output by the risk identification device.
[0049] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.
[0050] According to a fifth aspect, an electronic device is provided, characterized in that it comprises:
[0051] One or more processors; and
[0052] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.
[0053] According to the specific embodiments provided in this application, this application can have the following advantages:
[0054] 1) This application can automatically identify risks in self-service checkout, reducing merchants' manual loss prevention costs.
[0055] 2) This application adopts a dual risk identification method to reduce the probability of missed detection caused by simply analyzing the user's specific scanning behavior and improve the accuracy of risk identification results.
[0056] 3) This application only requires focusing on the area in front of the scanning device in the self-service checkout area, without the need to divide other functional areas, making it simpler and more universal.
[0057] 4) The risk detection status of the self-service checkout machine is divided into different stages, making the risk identification logic clearer and more accurate.
[0058] 5) A more comprehensive classification of primary risk identification will be implemented so that merchants can adopt flexible handling methods based on different types of primary risks.
[0059] 6) In the second risk identification process, unscanned products are filtered based on product similarity to improve the accuracy of risk identification and reduce the probability of misidentification.
[0060] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1An exemplary system architecture for a self-service checkout system to which embodiments of this application can be applied is shown;
[0063] Figure 2 A main flowchart of the risk identification method for self-service checkout provided in the embodiments of this application;
[0064] Figure 3 This is a schematic diagram of the first risk type provided in the embodiments of this application;
[0065] Figure 4 This is a schematic diagram of the detection state transition provided in an embodiment of this application;
[0066] Figure 5 This is a schematic block diagram illustrating a risk identification device according to one embodiment;
[0067] Figure 6 An example is shown of the architecture of an electronic device. Detailed Implementation
[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0069] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0070] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0071] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."
[0072] To facilitate understanding of this application, the self-service checkout system provided in this application will first be described.
[0073] Figure 1 An exemplary system architecture for a self-service checkout system applicable to embodiments of this application is shown, such as... Figure 1 As shown, the system may include a self-service checkout device 100, a video capture device 110, a risk identification device 120, and a monitoring device 130.
[0074] The self-checkout machine 100 can be installed in any location requiring checkout, such as supermarkets, shopping malls, stores, bookstores, etc. It can be placed at the exits or other locations in these establishments. Customers can check out their items at the self-checkout machine 100.
[0075] The self-checkout device 100 may specifically include a barcode scanner 101, a shopping counter 102, and a payment device 103. The barcode scanner 101 scans the barcodes of goods at the customer's request and provides the scanned product information to the risk identification device 120. The shopping counter 102 stores the goods. The payment device 103 provides an interactive payment interface to the customer and sends payment-related information to the risk identification device 103 based on the customer's payment actions on the interface.
[0076] For example, after selecting items, a customer comes to the self-checkout machine 100, places the items on the shopping counter 102, and then scans the items using a barcode scanner 101. After scanning, an interactive payment interface displays a purchase list (containing all scanned items), and the user can trigger components on the interactive payment interface to make the payment. Upon entering the payment page, the payment device 103 sends information about entering the payment page to the risk identification device 120. Upon successful payment, the payment device 103 sends a payment success message to the risk identification device 120. Upon payment failure, the payment device 103 sends a payment failure message to the risk identification device 120. And so on.
[0077] The aforementioned risk identification device 120 is used to identify risks throughout the customer's entire self-checkout process and output the risk identification results. This risk identification device 120 can, for example... Figure 1 The setup shown is on the server side, such as a risk identification server. The server can be a single server or a server cluster consisting of multiple servers, and the server can be a cloud server. However, in addition to this method, it can also be set up on other devices, such as computer terminals with strong computing power.
[0078] The monitoring device 130 is used to acquire and display the risk identification results output by the risk identification device 120, or it can further send control information to the self-service checkout device 100. This monitoring device can be any type of terminal device, such as a monitor, smart mobile device, wearable device, or PC (Personal Computer). Smart mobile devices can include devices such as mobile phones, tablets, laptops, and PDAs (Personal Digital Assistants). Wearable devices can include devices such as smartwatches, smart glasses, and smart bracelets.
[0079] It should be noted that the system architecture only schematically includes the main equipment required for risk identification; other equipment related to self-checkout, such as third-party settlement servers, is not shown.
[0080] Figure 2 This is a flowchart illustrating the main process of a risk identification method for self-service checkout provided in this application embodiment. The method comprises... Figure 1 The risk identification device 120 in the system architecture shown is executed. For example... Figure 2 As shown, the method mainly includes the following steps:
[0081] Step 201: Obtain video information of customers in the self-checkout area, identify abnormal behavior of users scanning goods in the video information, and determine the first risk information based on the identification results.
[0082] Step 202: If payment success information is obtained, determine the unscanned product information based on the product information identified from the video information and the scanned product information; determine the second risk information based on the unscanned product information.
[0083] Step 203: Based on the first risk information and the second risk information, output the risk identification result.
[0084] As can be seen, this application can automatically identify risks in self-service checkouts, reducing merchants' manual loss prevention costs. Furthermore, this application employs a dual risk identification method, reducing the probability of missed detections caused by simply analyzing users' specific scanning behavior, and improving the accuracy of risk identification results.
[0085] The steps described above are described in detail below. First, step 201, namely "acquiring video information of customers in the self-service checkout area, identifying abnormal behavior of users scanning goods in the video information, and determining the first risk information based on the identification results," will be described in detail with reference to the embodiment.
[0086] In this embodiment of the application, it can be by Figure 1The system architecture shown includes an image acquisition device for collecting video information. This device can be installed in specific locations such as shopping malls, supermarkets, and retail stores, so that it faces the self-checkout area and captures video information of customers in that area. During deployment, the image acquisition device must ensure unobstructed and clear capture of customer behavior in the self-checkout area, as well as information such as product features and human characteristics. The image acquisition device can be a camera, video camera, or similar equipment, capable of transmitting the captured video information as a video stream to the risk identification device.
[0087] In this step, the risk identification device can identify the product location information and / or human posture information in the video information, and identify the first risk information based on the identification results.
[0088] It should be noted that the terms "first" and "second" used in the embodiments of this application do not have limitations in terms of size, order, or quantity, but are only used to distinguish them in name. For example, "first risk information" and "second risk information" are used to distinguish the risks in two stages. The first risk information reflects the intermediate risks in the self-service checkout process, and the second risk information reflects the final risks at the end of the self-service checkout process.
[0089] On one hand, the risk identification device acquires the results of identifying goods in the video information. Specifically, it can identify each goods based on the goods' motion information and feature information in the video information. The goods' motion information mainly includes trajectory information and posture information determined based on the goods' position. Goods position identification can be achieved using image detection algorithms; this application does not limit the specific algorithm used, but only utilizes the results of goods position detection. Goods feature information can be implemented using deep learning algorithms; this application also does not limit the specific algorithm used, but only utilizes the extracted goods feature information.
[0090] On the other hand, the risk identification device acquires the results of human posture recognition in video information. Specifically, human feature information can be extracted from video information using network algorithms such as ResNet (Residual Network) for human tracking and customer identification. Further, multi-task detection algorithms such as YOLO (You Only Look Once, an object detection algorithm) and Faster R-CNN (Fast Region Convolutional Neural Network) can be combined to identify human body parts such as hands. Then, algorithms such as OpenPose and HRnet (High Resolution Network) are used to detect and extract human key points. These key points can be used to calculate the fine position of the human body and analyze the interaction information between the hands and goods. Then, based on the human position information, human feature information, and human key point information, the customer's behavioral trajectory is analyzed to identify the human posture. Similarly, this application does not limit the specific algorithm used, but only utilizes the identified human posture information.
[0091] like Figure 3 As shown, based on the analysis of product location information and human posture information, the following first risk types can be obtained, but are not limited to:
[0092] 1) Risk of goods not being scanned after passing through the scanning area:
[0093] If no successful scan notification is received within a set time period after the product's trajectory passes in front of the scanning device, it is determined that there is a risk that the product passed through the scanning area but was not scanned. For example, after identifying the product's trajectory, it is found that although the product's trajectory passed in front of the scanning device, no successful scan notification was received within the set time period. This means that the user may have pretended to scan the product, only moving the product in front of the scanning device with the scanning action, but without actually scanning it. Therefore, this situation poses a certain risk and is considered one of the first types of risks.
[0094] 2) Risk of product label swapping during barcode scanning:
[0095] If a successful scan is obtained, the product features corresponding to the product label information contained in the successful scan are compared with the product features identified from the video information. If the similarity is lower than the preset similarity threshold, it is determined that there is a risk of product label swapping during scanning.
[0096] In some cases, customers may tear off the tags of high-priced items and replace them with tags of lower-priced items. To detect this, the product tag information obtained from scanning can be searched in a product feature database to find the corresponding product features. Then, the similarity between these features and the features of the recently scanned product identified from video footage can be calculated to see if they are the same item. For example, suppose a customer scans a product, and the product tag information obtained from the scan reveals in the product feature database that the product is a bag of red-packaged potato chips. However, by comparing the features of the recently scanned product in the video footage, if the features of the recently scanned product in the video footage are not red-packaged, then the similarity between the two is low, indicating a risk of product tag tampering during scanning, which can be identified as one of the first risk types. This risk and the corresponding scanned product tag information can be recorded and output as one of the risk identification results in subsequent steps to assist supermarkets in investigating specific risk information.
[0097] 3) Risk of multiple failed product scans:
[0098] If the product's trajectory is in the area in front of the scanning device and the user does not receive a successful scan notification within the set time limit for multiple scans of the product, then there is a risk of multiple failed scans of the product.
[0099] This situation involves analyzing both the product's trajectory and the user's posture. It was found that the product's trajectory was in the area in front of the scanning device, and the user attempted to scan the product multiple times without success. In such cases, the user's failure to scan is usually not intentional, but may be due to missing or uneven labels. Identifying this risk is primarily to provide a risk warning so that the merchant can intervene quickly to resolve the issue and improve the user experience.
[0100] 4) Risk of abnormal product movement without scanning:
[0101] If the product's trajectory does not pass through the area in front of the scanning device and an abnormal movement trajectory is observed, and no information indicating that the product was successfully scanned is received, then there is a risk of abnormal product movement and failure to scan.
[0102] This situation mainly occurs when, after analyzing the product trajectory, it is found that the product trajectory does not pass through the area in front of the barcode scanner and is not scanned. For example, the product is placed on the right side of the shopping counter after bypassing the area in front of the barcode scanner from the left side of the shopping counter, or the product is directly packaged after passing through the area in front of the barcode scanner from the left side of the shopping counter, but is not scanned. This may be because the customer is intentionally avoiding scanning, which is one of the first types of risks.
[0103] 5) Risk of product barcodes being deleted after scanning:
[0104] If a product is successfully scanned and then removed from the purchase list, or if the product count is 0, then there is a risk that the product will be removed after scanning.
[0105] This situation could occur when a customer scans a product and then intentionally removes it from their shopping list, potentially resulting in them taking the item without paying for it. Therefore, this could be considered one of the primary risk types. However, it's also possible that the customer scans the barcode but genuinely no longer wants the item and removes it from their shopping list. In this case, it's necessary to identify the customer and further determine whether the item was ultimately taken to prevent missed detection.
[0106] 6) Risk of leaving without payment:
[0107] If no payment is made after scanning the code and the video information detects that the customer has left the self-checkout area, then it is determined that there is a risk of leaving without paying.
[0108] If a user's scanning action is detected or a message indicating successful product scanning is received, but no payment success message is received, yet the video footage shows the customer leaving the self-checkout area, then it is necessary to prevent the user from taking the goods without paying. Therefore, this can be considered one of the primary risk types, and merchants should be reminded to investigate this risk.
[0109] The following describes in detail step 202, namely, "If payment success information is obtained, determine the unscanned product information based on the product information identified from the video information and the scanned product information; determine the second risk information based on the unscanned product information," with reference to the embodiments.
[0110] Since all product information in the self-checkout area can be identified from the video information, after obtaining the payment success information, the product information that has not been scanned can be determined by using the product information that has been scanned and the product information identified from the video information.
[0111] As one feasible approach, the second risk information can be determined using all unscanned product information, that is, the second risk information can be output as all unscanned product information.
[0112] However, in some accidental situations, unintentional obstruction of goods by customers or other objects can interrupt the product's trajectory information, potentially leading to the same product being identified as two or more items. Similarly, the way the product is displayed or other factors can also cause the same product to be identified as two or more items. To address these situations, a better implementation method is provided: similarity calculation is performed on scanned and unscanned products, and unscanned products with a similarity greater than or equal to a preset similarity threshold are filtered. If unscanned products still exist after filtering, a second risk is identified, and the filtered unscanned product information can be included in the second risk information. Specifically, the similarity calculation between scanned and unscanned products can be based on the product's trajectory information and / or feature information.
[0113] For example, suppose we identify products A, B, and C from video information. After receiving payment success information, it means the user has completed the payment, but only products B and C were scanned. In this case, we can calculate the similarity between the unscanned product A and products B and C respectively. If product A and product B are highly similar in both trajectory and features, we can consider product A and product B to be the same product, but they were mistakenly identified as two separate products. Therefore, product A can be filtered out from the unscanned products. It can be seen that this filtering process improves the accuracy of risk identification and reduces the probability of false identification.
[0114] The following describes step 203, namely "outputting risk identification results based on the first risk information and the second risk information," in detail with reference to the embodiments.
[0115] In this step, the risk identification result can be output by combining the first risk information and the second risk information.
[0116] As an approach, if second risk information exists, both the first and second risk information can be included in the risk identification result output.
[0117] As another feasible approach, if no secondary risk information exists, then there can be no risk, and the absence of risk can be output as the risk identification result. The absence of secondary risk indicates that all goods in the video information have been scanned and paid for, therefore, there can be no risk.
[0118] As another feasible approach, if no second risk information exists, all or part of the first risk information can be included in the risk identification result output. In some cases, the first and second risk information may contradict each other. For example, the first risk may identify that a product passed through the scanning area without being scanned, but the second risk may indicate that all products were scanned and paid for. In such cases, the second risk can be taken as the standard, or the first risk information can be output for the merchant to further determine whether a risk truly exists. Therefore, all first risk information can be included in the risk identification result output, or only some important first risk information can be included.
[0119] Regarding the method of outputting risk results, recording, reminders, or early warning methods can be adopted according to the level of the first risk information and the level of the second risk information, and the corresponding level can be used.
[0120] For example, if a second risk exists, the risk identification result can be recorded and an alarm can be triggered to output the risk identification result.
[0121] For example, if the system identifies primary risks such as products passing through the scanning area without being scanned, products having their labels swapped before scanning, products moving abnormally without being scanned, or products leaving without payment, it can record these primary risks and include them in the risk identification results as a reminder, or use an early warning method.
[0122] For example, if multiple scans of a product fail, it is very likely due to issues such as uneven or damaged product labels. Often, customers need assistance. In such cases, the self-checkout device can be activated by flashing lights or screens, or by sending messages to the monitoring equipment, to alert the merchant's staff to help the customer. This will not significantly impact the customer's experience.
[0123] In addition to the methods mentioned above, other measures can be flexibly adopted. For example, if the first risk is identified, such as the product passing through the scanning area without being scanned or the product moving abnormally without being scanned, the user can be prompted on the payment interface that a missed scan has occurred and the user can be prohibited from making the payment.
[0124] To make the risk identification process for self-checkout clearer and more accurate, the entire process can be divided into five detection states: not in use and no customer in front of the device, not in use and customer in front of the device, in use, paying in progress, and payment completed and customer not yet left. These five states correspond to different switching conditions, and each state requires different identification processing. The following detailed description of the entire risk identification process is provided in conjunction with an example.
[0125] like Figure 4 As shown, the initial state is set to state 1: not in use and no customers in front of the device.
[0126] In this detection state, the following procedure can be performed: If a product is detected at the self-checkout machine via video feed, it is likely left behind by a previous customer. To prevent this product from interfering with subsequent customers' self-checkout, a notification message can be output to remind the merchant's staff to remove the product. This notification message can be sent to the monitoring equipment via text message, broadcast to the merchant's staff via voice, or use flashing lights to alert them that a product needs to be removed from the self-checkout machine. After the product is removed, the merchant's staff can disable the notification message through the security equipment.
[0127] If condition 2 is detected based on state 1, i.e., a customer enters the self-checkout area in the video information, set the detection state to state 2: not in use and there is a customer in front of the device.
[0128] In this state, process 2 can begin: identifying product information in the video and initializing the attribute of each product to "unscanned". Additionally, in this state, product trajectories and human postures can be identified to recognize customer actions.
[0129] If condition 3 is detected based on state 2, i.e., the scanning start information is obtained, then the detection state is set to state 3: in use.
[0130] In this state, process 3 is executed: Abnormal behavior identification is performed on the user's scanning behavior of goods in the video information, and the first risk information is determined based on the identification results. Furthermore, the attribute of the successfully scanned goods is set to "scanned".
[0131] For information on determining the first risk information in section 3, please refer to [link / reference]. Figure 2 The specific description of step 201 in the illustrated embodiment will not be repeated here.
[0132] If condition 1 is detected based on state 2, i.e., the customer leaves without carrying any goods, then the detection state is set to state 1: not used and no customer in front of the device.
[0133] If condition 4 is detected based on state 3, i.e., information indicating entry to the payment page is obtained, then the detection state is set to state 4: Payment in progress.
[0134] If condition 5 is detected based on state 4, i.e., payment success information is obtained and the customer has not left the self-checkout area is detected from the video information, then the detection state is set to state 5: payment completed and customer has not left.
[0135] In this state, process 4 can be executed: Based on the product information identified from the video information and the scanned product information, determine the unscanned product information, and determine the second risk information based on the unscanned product information. For details on determining the second risk information in process 4, please refer to [link to relevant documentation]. Figure 2 The specific description of step 202 in the illustrated embodiment will not be repeated here.
[0136] If condition 6 is detected based on state 5, i.e., a customer is detected leaving the self-checkout area from the video information and no new customer is detected entering the self-checkout area, then the detection state is set to state 1: not in use and no customer in front of the device.
[0137] If condition 7 is detected based on state 5, and the scanning start information is obtained, it may be that the customer discovers that there is a missing item that has not been scanned, and then scans the code to pay. In this case, the detection state is set to state 3: in use.
[0138] If condition 8 is detected based on state 5, that is, a new customer is detected entering the self-checkout area from the video information, this situation often means that the previous customer has not completely left and the new customer is already eager to enter the self-checkout area. Therefore, the detection state can be directly set to state 2: not in use and there is a customer in front of the device.
[0139] It should be noted that the above-mentioned detection states and the transition methods between detection states are only a preferred embodiment provided by the present application. Any deletions, additions, modifications, etc., of the detection states or the transition methods between detection states based on this are within the protection scope of the present application if they are within the spirit and principles of the present application.
[0140] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0141] According to another embodiment, a risk identification device for self-checkout is provided. Figure 5 The diagram illustrates a schematic block diagram of a risk identification device according to one embodiment. This device can be an application located on a server side, or it can be a functional unit such as a plugin or software development kit (SDK) within an application located on a server side, or it can be located on a computer device with strong computing power. This embodiment of the invention does not particularly limit its capabilities. Figure 5As shown, the device 500 includes: a video acquisition unit 501, a first risk analysis unit 502, a second risk analysis unit 503, and a risk result output unit 504. It may further include an information acquisition unit 505, a product feature library unit 506, a human body detection unit 507, a human posture estimation unit 508, a product detection unit 509, a product feature extraction unit 510, and a status control unit 511. The main functions of each component are as follows:
[0142] The video acquisition unit 501 is configured to acquire video information of customers in the self-checkout area.
[0143] The first risk analysis unit 502 is configured to identify abnormal behavior of users scanning goods in video information and determine the first risk information based on the identification results.
[0144] The second risk analysis unit 503 is configured to, if payment success information is obtained, determine unscanned product information based on product information identified from video information and scanned product information; and determine second risk information based on unscanned product information.
[0145] The risk result output unit 504 is configured to output risk identification results based on the first risk information and the second risk information.
[0146] In addition to the video acquisition unit 501 mentioned above, the embodiments of this application may also include acquisition end modules such as information acquisition unit 505 and commodity feature library unit 506.
[0147] The information acquisition unit 505 is configured to acquire scanning-related information from the scanning device and payment-related information from the payment device.
[0148] Product feature library unit 506 is configured to pre-store product feature information and construct a product feature library. For example, for a self-checkout system in a supermarket, product feature library unit 506 can construct a product feature library from the feature information of all products in the supermarket. It can be searched and queried using product labels, product barcodes, etc.
[0149] The aforementioned first risk analysis unit 502 and second risk analysis unit 503 belong to the strategy analysis module. The first risk analysis unit 502 can be specifically configured to identify product location information and / or human posture information in the video information, and perform at least one of the following actions based on the identification result:
[0150] If no successful scanning message is received within the set time period after the product's trajectory passes through the area in front of the scanning device, it is determined that there is a risk that the product has passed through the scanning area without being scanned.
[0151] If a successful scan is obtained, the product features corresponding to the product label information contained in the successful scan are compared with the product features identified from the video information. If the similarity is lower than the preset similarity threshold, it is determined that there is a risk of product label swapping during scanning.
[0152] If the product's trajectory is in the area in front of the scanning device and the customer does not receive a successful scan notification within the set time period for multiple scans of the product, then there is a risk of multiple failed scans of the product.
[0153] If the product's trajectory does not pass through the area in front of the scanning device and an abnormal movement trajectory is observed, and no information indicating that the product was successfully scanned is received, then there is a risk of abnormal movement of the product that has not been scanned.
[0154] If a product is successfully scanned and then removed from the purchase list, or if the product count is 0, then there is a risk that the product will be removed after scanning.
[0155] If no payment is made after scanning the code and the video information detects that the customer has left the self-checkout area, then it is determined that there is a risk of leaving without paying.
[0156] In a preferred embodiment, the second risk analysis unit 503 may be specifically configured to perform similarity calculation on scanned and unscanned products, filter unscanned products with similarity greater than a preset similarity threshold, wherein the similarity is calculated based on the product's trajectory information and / or feature information; if unscanned products still exist after filtering, a second risk is determined to exist.
[0157] To support strategy analysis, some visual algorithm modules are needed as a foundation. For example... Figure 6 As shown, it may include units such as human body detection unit 507, human body posture estimation unit 508, commodity detection unit 509, and commodity feature extraction unit 510.
[0158] The human detection unit 507 can be configured to extract human feature information from video information using network algorithms such as ResNet, and further combine it with multi-task detection algorithms such as YOLO series algorithms and Faster R-CNN algorithms to identify the human body or specific human body parts, such as the hand. It can also use algorithms such as OpenPose and HRnet to detect and extract human key points.
[0159] The human posture estimation unit 508 can be configured to analyze the customer's behavioral trajectory based on human position information, human feature information, and human key point information, thereby identifying the human posture for analyzing the customer's behavior.
[0160] The product feature extraction unit 510 can be configured to extract product feature information from video information using a deep learning algorithm.
[0161] The product detection unit 509 can be configured to identify the position information of the product in the video information through an image detection algorithm, thereby forming the product's trajectory information.
[0162] It should be noted that units such as the human pose estimation unit 508 and the product detection unit 509 mentioned above can be further broken down into multiple functional modules for implementation. Alternatively, multiple units such as the human detection module 507 and the product detection module 509 can be combined into a single module, using a single algorithm model to output different detection results. Each unit in the aforementioned visual algorithm module can be implemented using currently mature algorithms; this application may utilize only the output results of the aforementioned visual algorithm module.
[0163] The aforementioned first risk analysis unit 502 and second risk analysis unit 503 belong to the strategy analysis module. The strategy analysis module may further include a state control unit 511, specifically configured as follows:
[0164] Initially, before detecting a customer entering the self-checkout area in the video information, the detection state is set to "not in use and no customer in front of the device." In this "not in use and no customer in front of the device" state, if the video information detects items at the self-checkout device, the risk result output unit 504 can be triggered to output an alert message.
[0165] If a customer is detected entering the self-checkout area in the video information, the detection status is set to "not in use and there is a customer in front of the device". The product information in the video information is identified and the attribute of each product is initialized to "not scanned".
[0166] If scanning start information is obtained when the device is not in use and there is a customer in front of it, the detection status is set to "in use". In the "in use" status, the first risk analysis unit 502 is triggered to perform abnormal behavior identification on the user's scanning behavior of the product in the video information, and the attribute of the product that has been successfully scanned is set to "scanned".
[0167] If information indicating entry to the payment page is received, the detection status is set to "Payment in Progress". If, while in the "Payment in Progress" status, payment success information is received and the video footage indicates the customer has not left the self-checkout area, the detection status is set to "Payment Completed and Customer Has Not Left".
[0168] When payment is completed and the customer has not left, the second risk analysis unit 503 is triggered to determine the processing of unscanned product information based on the product information identified from the video information and the scanned product information.
[0169] If a new customer is detected entering the self-checkout area from the video information while the payment is completed and the customer has not left, the detection status is set to "not in use and there is a customer in front of the device".
[0170] If, after payment is completed and the customer has not left, the video information detects that the customer has left the self-checkout area and no new customer has entered the self-checkout area, then the detection status is set to "not in use and no customer in front of the device".
[0171] It should be noted that the first risk analysis unit 502 described above can be further divided into multiple functional modules for execution. For example, it can be further divided into a module for analyzing customer interaction behavior with products. Similarly, the second risk analysis unit 503 described above can also be further divided into multiple functional modules for execution. For example, it can be further divided into a module for retrieving product features, a module for filtering unscanned products based on product similarity, and so on. Furthermore, the visual algorithm module can be implemented using other servers, such as an image processing server performing the visual algorithm processing and providing the processing results to the server where the strategy analysis module resides.
[0172] The embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).
[0173] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.
[0174] And an electronic device, comprising:
[0175] One or more processors; and
[0176] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.
[0177] in, Figure 6An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.
[0178] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.
[0179] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a risk identification device 625, etc. The aforementioned risk identification device 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.
[0180] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.
[0181] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0182] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.
[0183] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.
[0184] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0185] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0186] The methods, apparatus, and systems provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. Risk identification methods for self-checkout include: The system acquires video information of customers in the self-checkout area, identifies product information in the video information, identifies abnormal behavior in the user's scanning behavior of products in the video information, and determines first risk information based on the abnormal behavior identification results; wherein, the product information identified from the video information includes: product trajectory information and product feature information; If payment success information is obtained, based on the product information identified from the video information and the scanned product information, the unscanned product information is determined, and the similarity between the product trajectory information and product feature information of the scanned and unscanned products is calculated. If the similarity between an unscanned product and a scanned product is greater than a preset similarity threshold, it is confirmed that the unscanned product and the scanned product are the same product but were mistakenly identified as different products. After filtering the unscanned product out of the unscanned product set, the second risk information is determined based on whether there are still unscanned products in the unscanned product set. The risk identification process is divided into multiple detection states, including: not in use and no customer in front of the device, not in use and customer in front of the device, in use, in payment, and payment completed and customer not leaving. Different detection states correspond to different switching conditions and different identification processes; wherein: If a customer is detected entering the self-checkout area in the video information, the detection status is set to "not in use and there is a customer in front of the device", and the attributes of each product identified from the video information are initialized to "not scanned". If scanning start information is obtained when the device is not in use and there is a customer in front of it, the detection status is set to "in use". In the "in use" status, the step of identifying abnormal behavior of the user scanning the product in the video information is executed, and the attribute of the product that has been successfully scanned is set to "scanned". If, during the payment process, the payment success information is obtained and the customer is detected from the video information as not having left the self-service checkout area, then the detection status is set to "payment completed and customer not left". In the "payment completed and customer not left" status, the step of determining the unscanned product information based on the product information identified from the video information and the scanned product information is executed. Based on the first risk information and the second risk information, output the risk identification result.
2. The method according to claim 1, further comprising: Before detecting a customer entering the self-checkout area in the video information, the detection status is set to "not in use and no customer in front of the device"; If a product is detected at the self-checkout device from the video information when the device is not in use and there are no customers in front of it, an alert message will be output.
3. The method according to claim 1, further comprising: If, after payment is completed and the customer has not left, a new customer is detected entering the self-checkout area from the video information, the detection status is set to "not in use and there is a customer in front of the device"; and / or, If, after payment is completed and the customer has not left, the video information detects that the customer has left the self-checkout area and no new customer has entered the self-checkout area, then the detection status is set to "not in use and no customer in front of the device".
4. The method according to claim 1, wherein, The step of identifying abnormal behavior in the user's scanning behavior of goods in the video information, and determining the first risk information based on the identification result, includes: The product location information and / or human posture information in the video information are identified, and at least one of the following is performed based on the identification result: If no successful scanning message is received within the set time period after the product's trajectory passes through the area in front of the scanning device, it is determined that there is a risk that the product has passed through the scanning area without being scanned. If a successful scan is obtained, the product features corresponding to the product label information contained in the successful scan are compared with the product features identified from the video information. If the similarity is lower than the preset similarity threshold, it is determined that there is a risk of product label swapping during scanning. If the product's trajectory is in the area in front of the scanning device and the customer does not receive a successful scan notification within the set time period for multiple scans of the product, then there is a risk of multiple failed scans of the product. If the product's trajectory does not pass through the area in front of the scanning device and an abnormal movement trajectory is observed, and no information indicating that the product was successfully scanned is received, then there is a risk of abnormal movement of the product that has not been scanned. If a product is successfully scanned and then removed from the purchase list, or if the product count is 0, then there is a risk that the product will be removed after scanning. If no payment success information is received after scanning the code and the video information detects that the customer has left the self-service checkout area, then it is determined that there is a risk of leaving without paying.
5. The method according to any one of claims 1 to 4, wherein, Based on the first risk information and the second risk information, the risk identification results output include: Based on the level of the first risk information and the level of the second risk information, a recording, reminder, or warning method appropriate to the corresponding level is adopted.
6. Risk identification devices for self-checkout, including: The video acquisition unit is configured to acquire video information of customers in the self-checkout area and identify product information in the video information, wherein the product information identified from the video information includes: product trajectory information and product feature information; The first risk analysis unit is configured to identify abnormal behavior of users scanning goods in the video information and determine the first risk information based on the identification results. The second risk analysis unit is configured to, upon receiving payment success information, determine unscanned product information based on product information identified from the video information and scanned product information, and perform similarity calculation on the product trajectory information and product feature information of scanned and unscanned products. If the similarity between an unscanned product and a scanned product is greater than a preset similarity threshold, it is confirmed that the unscanned product and the scanned product are the same product but were mistakenly identified as different products. After filtering the unscanned product out of the unscanned product set, the second risk information is determined based on whether there are still unscanned products in the unscanned product set. The risk identification process is divided into multiple detection states, including: not in use and no customer in front of the device, not in use and customer in front of the device, in use, in payment, and payment completed and customer not leaving. Different detection states correspond to different switching conditions and different identification processes; wherein: If a customer is detected entering the self-checkout area in the video information, the detection status is set to "not in use and there is a customer in front of the device", and the attributes of each product identified from the video information are initialized to "not scanned". If scanning start information is obtained when the device is not in use and there is a customer in front of it, the detection status is set to "in use". In the "in use" status, the step of identifying abnormal behavior of the user scanning the product in the video information is executed, and the attribute of the product that has been successfully scanned is set to "scanned". If, during the payment process, the payment success information is obtained and the customer is detected from the video information as not having left the self-service checkout area, then the detection status is set to "payment completed and customer not left". In the "payment completed and customer not left" status, the step of determining the unscanned product information based on the product information identified from the video information and the scanned product information is executed. The risk result output unit is configured to output a risk identification result based on the first risk information and the second risk information.
7. A risk identification system for self-service checkout, comprising video capture equipment, self-service checkout equipment, surveillance equipment, and the risk identification device as described in claim 6: The video capture device is configured to capture video information of customers in the self-checkout area and provide the video information to the risk identification device. The self-service checkout equipment includes a barcode scanner, a shopping counter, and a payment device; The scanning device is configured to scan the product under the operation of the customer and provide the scanned product information to the risk identification device. The shopping counter is used to store goods; The payment device is configured to provide an interactive payment interface to the customer and send payment-related information to the risk identification device based on the customer's payment operation on the interface. The monitoring equipment is configured to acquire and display the risk identification results output by the risk identification device.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 5.
9. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 5.