Self-service checkout method, system, device and suite combining sensor and artificial intelligence

Abstract

The invention relates to a self-checkout method, which comprises the following steps executed by a control module: driving a first image sensor to obtain a first image containing a hand; driving a second image sensor to acquire a second image containing the article; executing a gesture recognition independent model to generate a first recognition result based on the first image; executing an object recognition independent model to generate a third recognition result based on the second image; and executing and inputting the first identification result and the third identification result into a checkout auxiliary decision independent model so as to judge whether a swiping omission event occurs according to the first identification result and the third identification result.

Description

Technical Field

[0001] The present invention relates to a self-service checkout method, system, device, and kit that combine sensors and artificial intelligence, and more particularly to a self-service checkout method, system, device, and kit that combine sensors and artificial intelligence to determine whether a product barcode has been missed. Background Art

[0002] With rising labor costs and the impact of the novel coronavirus, the retail industry has been rapidly introducing unmanned stores and self-checkout systems. Self-checkout technology has become a significant innovation trend. Its goal is to leverage technology combined with electronic payments to automatically identify and settle items, thereby reducing the need for labor. Existing mainstream self-checkout technologies rely on product weight sensing, RFID (radio frequency identification) tag sensing, eTag (electronic tag) tag sensing, pure image (visual) recognition, and barcode recognition.

[0003] The weight-sensing method identifies the type and quantity of goods by measuring the weight of the items being checked out. However, the disadvantage of this technology is that it may cause recognition errors for different items of similar weight, and its ability to recognize multiple items is poor. For example, when a customer places multiple items on the scale together, it will be difficult for the system to correctly identify each item, and it is easy to miss items. At the same time, the weight of the goods is easily affected by the packaging and environment, increasing the risk of errors.

[0004] RFID tag sensing technology identifies products by reading RFID tags attached to them. While this technology can accurately identify products, it is relatively expensive. Small or inexpensive products may not be able to afford the high cost of using RFID tag technology. In addition, RFID tags are at risk of damage, and when RFID tags are not properly attached to the product, the reader cannot correctly read the product information, which can easily lead to missed scans.

[0005] eTag tag sensing technology is similar to RFID, but uses electronic tags to store product information. Similarly, the use of eTag sensing technology also requires investment and the establishment of specific equipment and infrastructure, so there are still issues of cost and practicality. eTag tags also have the risk of damage, and if they are not correctly set on the product, it is easy for the product to be missed.

[0006] Pure image recognition technology relies on image analysis algorithms to identify products. However, pure image recognition technology is expensive to implement and has a high technical threshold. It is sensitive to ambient light and the angle at which products are placed. It may be difficult to distinguish between products with similar appearances. When there are many types of products, insufficient image resolution, and unexpected blind spots and blind spots appear, recognition errors are also prone to occur, resulting in problems such as missed products or inability to check out.

[0007] Barcode recognition technology, which identifies products and facilitates checkout by reading barcodes attached to them, is a relatively popular self-checkout technology. However, consumers are often unfamiliar with barcode scanning or make careless mistakes, resulting in inaccurate scanning, missed items, blocked barcodes, incorrect placement of the barcode in the scanning area, or misalignment of the barcode with the scanner. These situations often lead to missed items and unsuccessful checkout. During peak hours, consumers may be distracted and forget to scan certain items. Alternatively, when purchasing large items or a large number of items at once, it is easy for the barcode to be misaligned with the scanner, resulting in missed items. All of these reasons can lead to a certain number of items being missed.

[0008] Overall, the various existing self-service checkout technologies are still unable to effectively identify missed scans of product barcodes, nor can they completely prevent customers from intentionally or unintentionally missing certain products. In addition to directly causing economic losses to the owner, missed scans will also cause incorrect inventory records, which may subsequently cause chain errors and bring management difficulties. Frequent missed scans will also make the owner exhausted.

[0009] In addition, there are already a large number of traditional checkout and cashier management devices on the market. These traditional devices do not have self-service checkout functions, but a complete self-service checkout system must have part of its hardware composed of these traditional devices. Therefore, after appropriate adjustments and modifications, these traditional devices can be upgraded to self-service checkout systems on the spot.

[0010] The software and hardware costs of self-checkout systems currently on the market remain high, placing a significant burden on many small and medium-sized enterprise (SME) retailers. Retrofitting existing checkout equipment to a self-checkout system would significantly reduce the cost of implementing a system. However, no one currently offers such upgrade services or retrofit technology.

[0011] Therefore, in view of the shortcomings of the existing technology, the inventors, after careful experimentation and research, and with a spirit of perseverance, finally conceived the present invention, "Self-service checkout method, system, device and kit combining sensors and artificial intelligence," which can overcome the above-mentioned shortcomings. The following is a brief description of the present invention. Summary of the Invention

[0012] The present invention relates to a self-service checkout method, system, device, and kit that combine sensors and artificial intelligence, and more particularly, to a self-service checkout method, system, device, and kit that combine sensors and artificial intelligence to determine whether a product barcode has been missed.

[0013] Based on this, the present invention proposes a self-service checkout method, which includes: executing the following steps through a control module: driving a first image sensor to obtain a first image including a hand; driving a second image sensor to obtain a second image including an object; executing an independent gesture recognition model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; and executing and inputting the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0014] The present invention further proposes a self-service checkout system, comprising: a first image sensor, configured to acquire a first image including a hand; a second image sensor, configured to acquire a second image including an object; and a control module, configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0015] The present invention further proposes a self-service checkout device, comprising: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including an object; and a control module configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0016] The present invention further proposes a self-service checkout kit, which is attached to a checkout management device and includes: a first image sensor, which is configured to capture a first image including a hand; a second image sensor, which is configured to capture a second image including an object; and a control module, which is integrated into the checkout management device or separated from the checkout management device but maintains a communication link. The control module is configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0017] The above invention summary is intended to provide a simplified summary of the present disclosure so that readers can have a basic understanding of the present disclosure. This invention summary is not a complete description of the present invention and is not intended to point out the important / critical elements of the embodiments of the present invention or to define the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the system architecture of the self-service checkout system included in the present invention is disclosed;

[0019] Figure 2 A schematic diagram showing the hardware architecture of the control module included in the present invention;

[0020] Figure 3 A block diagram illustrating the collaboration between the self-checkout software and hardware components included in the present invention;

[0021] Figure 4 A block diagram illustrating a deep learning model included in the present invention is disclosed;

[0022] Figure 5 A schematic diagram of the system architecture of the self-service checkout separation kit included in the present invention is disclosed;

[0023] Figure 6 A first flow chart illustrating the implementation steps of a first self-service checkout method included in the present invention;

[0024] Figure 7 A second flow chart illustrating the implementation steps of the first self-service checkout method included in the present invention; and

[0025] Figure 8 A flow chart of the implementation steps of the second self-service checkout method included in the present invention is disclosed.

[0026] Description of Figure Numbers:

[0027] 100 Self-checkout system

[0028] 11 Host

[0029] 12 Touch screen

[0030] 13. Barcode Scanner

[0031] 14 Credit card machine

[0032] 20 Control Module

[0033] 21 Basic Input / Output System (BIOS) firmware

[0034] 22 Microcontroller

[0035] 23 processors

[0036] 25 Information Display

[0037] 31 First proximity sensor

[0038] 32 Second proximity sensor

[0039] 33 Third proximity sensor

[0040] 41 First Intelligent Image Sensor

[0041] 42 Second smart image sensor

[0042] 43 High-position intelligent image sensor

[0043] 200 Self-Checkout Software

[0044] 311 Checkout Management Equipment

[0045] 312 touch screen

[0046] 313 barcode scanner

[0047] 314 Credit Card Machine

[0048] 400 Self-Checkout Separation Kit

[0049] 420 Control Module

[0050] 425 Information Display

[0051] 431 First proximity sensor

[0052] 432 Second proximity sensor

[0053] 433 Third proximity sensor

[0054] 441 First Intelligent Image Sensor

[0055] 442 Second Intelligent Image Sensor

[0056] 450 high-bit intelligent image sensor

[0057] 451 First Bracket

[0058] 452 Second bracket

[0059] 453 Communication Interface

[0060] 500 First Self-Checkout Method

[0061] 501-514 Implementation Steps

[0062] 600 Second self-checkout method

[0063] 601-613 Implementation Steps

[0064] H Hand

[0065] C Information Code

[0066] P Items

[0067] PSA item sensing area

[0068] SCO self-checkout area

[0069] S1 First Image

[0070] S2 Second image

[0071] S3 third image

[0072] D1 First sensing output

[0073] D2 Second sensing output

[0074] D3 third sensing output

[0075] D4 Fourth sensor output

[0076] M1 gesture recognition standalone model

[0077] M2 Information Code Recognition Model

[0078] M3 Object Recognition Standalone Model

[0079] M4 Checkout Decision Support Independent Model

[0080] R1 First recognition result

[0081] R2 Second recognition result

[0082] R3 Third recognition result

[0083] R4 Fourth recognition result DETAILED DESCRIPTION

[0084] The present invention will be fully understood by the following examples, so that those skilled in the art can complete the invention accordingly. However, the implementation of the present invention is not limited to the following examples. The drawings of the present invention do not include any limitations on size, dimensions, and scale. When the present invention is actually implemented, its size, dimensions, and scale are not limited by the drawings of the present invention.

[0085] The term "preferably" used herein is non-exclusive and should be understood as "preferably but not limited to". Any steps described or recorded in any specification or claims may be performed in any order and are not limited to the order described in the claims. The scope of the present invention should be determined only by the appended claims and their equivalents, and should not be determined by the examples of the implementation examples. The term "including" and its variations when appearing in the specification and claims herein are open-ended terms without restrictive meanings and do not exclude other features or steps.

[0086] Figure 1 A schematic diagram of the system architecture of a self-checkout system included in the present invention is disclosed. In one embodiment, the self-checkout system (self-checkout device) 100 proposed by the present invention preferably includes a host 11, a touch screen 12, a barcode scanner 13, a card reader 14, a first proximity sensor 31, a second proximity sensor 32, a third proximity sensor 33, a first smart image sensor 41, a second smart image sensor 42, a high-position smart image sensor 43, a control module 20, and an information display 25. The control module 20 has a built-in self-checkout software 200.

[0087] The host 11 is preferably an electronic device with point-of-sale (POS) host functions, and is equipped with a touch screen 12, a barcode scanner 13, a card reader 14, and a control module 20 to implement the basic manual checkout functions that a POS host should have. The control module 20 is preferably configured inside the host 11, or is externally connected to the control circuit inside the host 11 to communicate with it.

[0088] An item sensor group is disposed beneath the self-checkout system 100 or the host 11. The item sensor group includes a first proximity sensor 31, a second proximity sensor 32, a third proximity sensor 33, a first smart image sensor 41, a second smart image sensor 42, and a barcode scanner 13. The item sensor group is preferably disposed beneath the self-checkout system 100 or the host 11, forming a virtual, three-dimensional item sensing area (PSA) in front of the host 11.

[0089] The first proximity sensor 31 , the second proximity sensor 32 , and the third proximity sensor 33 are preferably distance sensors, time-of-flight (TOF) sensors, multi-object tracking (MOT) sensors, gesture recognition sensors, optical motion trackers, motion sensors, proximity sensors, active infrared (AIR) sensors, passive infrared (PIR) sensors, or consumer infrared (CIR) sensors.

[0090] Figure 2 A schematic diagram of the hardware architecture of the control module included in the present invention is provided. The self-checkout method proposed in the present invention is preferably implemented in the form of self-checkout software 200, i.e., a self-checkout computer program product, installed on a control module 20 included in a self-checkout system 100. The control module 20 includes at least components such as basic input and output system (BIOS) firmware 21, a microcontroller (MCU) 22, and a processor 23. The self-checkout software 200 is loaded onto the processor 23 and executed jointly by the processor 23 and the microcontroller 22 to implement the self-checkout method through the control module 20 and perform various controls on the self-checkout system 100.

[0091] In one embodiment, the microcontroller 22 is primarily responsible for lower-level, basic hardware control tasks, such as controlling peripheral hardware devices, including but not limited to the barcode scanner 13, the card reader 14, the first proximity sensor 31, the second proximity sensor 32, the third proximity sensor 33, the first intelligent image sensor 41, the second intelligent image sensor 42, and the high-order intelligent image sensor 43. This includes controlling basic input and output operations of these peripheral hardware devices, as well as collecting and processing digital data generated by these peripheral hardware devices. In one embodiment, the microcontroller 22 also participates in energy consumption control of various hardware components, such as controlling sensors to enter an idle state to reduce power consumption and other energy-saving controls.

[0092] The control module 20 defines a certain range around the item sensing area PSA as a virtual self-checkout area SCO, and then detects whether a user enters the self-checkout area SCO based on the image captured by the high-position intelligent image sensor 43, so as to initiate the self-checkout process accordingly.

[0093] When the self-checkout process is not being executed, the control module 20 controls the information display 25 and the touch screen 12 to play preset advertising content, and controls the sensors included in the item sensor group to enter a low-power standby state to reduce the overall energy consumption of the self-checkout system 100.

[0094] When the control module 20 determines, based on the image captured by the high-position intelligent image sensor 43, that a user has entered the self-checkout area SCO, the self-checkout process will be initiated. Alternatively, when the control module 20 detects that any sensor included in the item sensor group is triggered, the control module 20 will also initiate the self-checkout process accordingly.

[0095] The information display 25 will begin to display the guidance content about the self-checkout standard operating procedures, showing and explaining the self-checkout operation steps to the user, and the touch screen 12 will begin to display the self-checkout operation interface and wait for the user to further perform the self-checkout operation.

[0096] Figure 3 A block diagram illustrating the collaboration between the self-checkout software and various hardware components included in the present invention is provided. The self-checkout software 200 of the present invention is executed by a control module 20. The self-checkout software 200 includes four independent models: a gesture recognition independent model M1, an information code recognition model M2, an object recognition independent model M3, and a checkout decision support independent model M4.

[0097] In one embodiment, the first proximity sensor 31, the second proximity sensor 32, and the third proximity sensor 33 are preferably time-of-flight sensors (TOF sensors), wherein the first proximity sensor 31 is preferably configured to sense first distance information about the hand H, the information code C marked on the object P, or the object P in the object sensing area PSA and generate a first sensing output D1. The second proximity sensor 32 is preferably configured to sense second distance information about the hand H, the information code C marked on the object P, or the object P in the object sensing area PSA and generate a second sensing output D2. The third proximity sensor 33 is preferably configured to sense third distance information about the hand H, the information code C marked on the object P, or the object P in the object sensing area PSA and generate a third sensing output D3. The barcode scanner 13 is preferably configured to read the information code C marked on the object P in the object sensing area PSA and generate a fourth sensing output D4.

[0098] The first intelligent image sensor 41 is preferably configured to obtain a first image S1 including a hand H and an information code C appearing on the object P within the object sensing area PSA, and the second intelligent image sensor 42 is preferably configured to obtain a second image S2 including the object P within the object sensing area PSA, wherein the object P is preferably attached with an information code C that can be read by the scanner 13, and the form of the information code includes but is not limited to: a one-dimensional barcode (barcode) and a two-dimensional code (QRcode), etc. The information code can store various information, including but not limited to the name of the item, production location, manufacturer, production date, item price, etc.

[0099] The high-position intelligent image sensor 43 is preferably located near the top of the self-service checkout system 100, for example, integrated into the upper frame of the information display 25, or mounted near the top of the information display 25 via a bracket. The high-position intelligent image sensor 43 located at a higher position can achieve a larger and wider field of view (FOV), allowing it to capture a panoramic view of the user's entire self-service checkout process from a commanding vantage point and generate a third image S3.

[0100] After the self-checkout program is started, the control module 20 is preferably configured to input the first image S1 acquired by the first intelligent image sensor 41 into the gesture recognition independent model M1 and the information code recognition model M2, so as to identify whether the first image S1 contains a hand and a gesture skeleton through the gesture recognition independent model M1, and identify whether the gesture skeleton contains a checkout action containing the meaning of checkout, and output a first recognition result R1, and then identify the information code contained in the first image S1 through the information code recognition model M2, and output a second recognition result R2.

[0101] The control module 20 is preferably configured to input the second image S2 captured by the second intelligent image sensor 42 into the object recognition independent model M3 to identify whether the second image S2 contains an object and the name of the object through the object recognition independent model M3 and output a third recognition result R3.

[0102] Next, the control module 20 is preferably configured to input the first recognition result R1 , the second recognition result R2 , and the third recognition result R3 into the checkout auxiliary decision independent model M4 , and output a fourth recognition result R4 .

[0103] The checkout assist decision-making independent model M4 is preferably a rule-based machine learning model. For example, the checkout assist decision-making independent model M4 includes but is not limited to the following rules:

[0104] Rule (1): When the third proximity sensor 33 senses that the distance between the object P and the barcode scanner 13 is greater than the effective sensing distance of the barcode scanner 13, and the second recognition result R2 indicates that the information code C carried by the object P is successfully obtained, and the user's gesture skeleton includes a checkout action, the control module 20 determines that the barcode distance is too far beyond the reading range of the barcode scanner 13, and a barcode out-of-distance event occurs. The control module 20 will control the touch screen 12 to display a prompt message, the content of which includes prompting the user to move the object closer to the barcode scanner.

[0105] Rule (2): When the third recognition result R3 indicates that the item is successfully detected, and the second recognition result R2 indicates that the user's gesture skeleton includes a checkout action, but the barcode scanner 13 fails to read the information code, the control module 20 determines that a missed scan event has occurred. The control module 20 controls the touch screen 12 to display a prompt message indicating that the item scanning has failed. The content of the message includes prompting the user to check whether the item barcode is aligned with the barcode scanner 13.

[0106] Rule (3): When the third recognition result R3 indicates that the number of items detected is N1, but the number of items successfully scanned by the scanner 13, N2, is not equal to N1, and when N1 is greater than N2, it is determined that a missed scan event has occurred. At this time, the control module 20 will control the touch screen 12 to display a missed scan prompt message, which includes prompting the user to confirm whether the scan code of an item is missed. If N1 continues to be greater than N2, the control module 20 will lock the touch screen 12 screen and disable the touch screen 12 to suspend the self-service checkout process, and notify the back-end management staff at the same time.

[0107] Rule (4): When the third recognition result R3 indicates that the number of items detected is N1, but the number of items scanned by the barcode scanner 13 is N2 but not equal to N1, and N2 is greater than N1, it is determined that a duplicate checkout event has occurred. The control module 20 controls the touch screen 12 to display a duplicate checkout prompt message, which includes prompting the user to confirm whether to check out repeatedly.

[0108] Rule (5): When the third recognition result R3 indicates that the detected item P disappears, it is determined that an abnormal checkout event has occurred. The control module 20 will lock the touch screen 12 screen and disable the touch screen 12 to suspend the self-service checkout process, and notify the back-end management staff at the same time.

[0109] The checkout auxiliary decision independent model M4 is a self-service checkout auxiliary decision model, such as Figure 3 As disclosed, the independent checkout decision-making support model M4 receives all sensor outputs D1-D4, all images S1-S3, and all recognition results R1-R3 generated by the self-checkout system 100. It then identifies the consumer's spending behavior based on built-in preset rules, determines whether any items were missed during the self-checkout process, and outputs a fourth recognition result R4. By increasing the number of built-in preset rules within the independent checkout decision-making support model M4 and optimizing them, the comprehensive capabilities of the independent checkout decision-making support model M4 can be significantly enhanced.

[0110] Figure 4A block diagram of the deep learning model included in the present invention is shown. This invention utilizes multiple independent small models, each performing different recognition tasks, replacing the integrated large model. Large models not only take up storage space but also consume significant computing power and memory. By splitting the models into small, independent models, not only does the storage space occupied by all the small models decrease, but the computing power and memory requirements are also significantly reduced. Furthermore, these small models can directly apply pre-trained models, further reducing model training costs, or use open source models to further reduce model construction costs.

[0111] Generally speaking, large supervised deep learning models require extensive training data. More labeled training data translates to higher development costs. Furthermore, larger supervised deep learning models converge more easily if they are broken down into several smaller models and trained separately before being integrated, rather than using end-to-end training.

[0112] Based on the above two principles, this invention proposes a training method and decision-making system that combines multiple models to detect consumer checkout behavior. By combining multiple common smaller models in series instead of a single large point-to-point model, the difficulty and cost of training the model are reduced.

[0113] The present invention conducts model training in two stages. In the first stage of model training, each model can be trained separately, or the existing model and weights can be directly used for transfer learning. In the second stage of model training, the trained small models are connected in series to complete the end-to-end large model training.

[0114] In one embodiment, the gesture recognition independent model M1 is preferably a convolutional neural network (CNN), a 3D convolutional neural network (3DCNN), a regional convolutional neural network (R-CNN), a graph convolutional network (GCN), a long short-term memory network (LSTM), a bidirectional long short-term memory network (Bi-LSTM), a temporal convolutional network (TCN), an OpenPose model, a GestureNet model, a BlazePalm model, a Landmark model, a MediaPipe model, a Rekognition model, a support vector machine (SVM), or a combination thereof.

[0115] In a certain embodiment, the information code recognition model M2 is preferably a ZBar model, an OpenCV model, a PyZbar model, a Google Vision model or a combination thereof, and the object recognition independent model M3 is preferably a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, a Faster R-CNN model, a Mask R-CNN model, a ResNet model, an EfficientDet model, an EfficientNet model, a VGG model, a VGGNet model, a COCO model, a CNN model, a MobileNet model, an AlexNet model, a GoogLeNet model or a combination thereof.

[0116] Figure 5 A schematic diagram of the system architecture of the self-checkout separation kit included in the present invention is disclosed. In one embodiment, the self-checkout method proposed in the present invention is further implemented in the form of a self-checkout separation kit 400.

[0117] Existing checkout management devices 311, such as POS machines and cash registers, generally do not support self-service checkout. Checkout requires a dedicated operator to operate the checkout management device 311, including scanning item barcodes and operating the checkout interface. However, the existing checkout management device 311 already includes basic hardware components, including a basic control circuit, a touch screen 312, a barcode scanner 313, and a card reader 314. It only requires the addition of an item sensor group, an information panel, and self-service checkout software 200 to upgrade it to a self-service checkout device with self-service checkout functionality.

[0118] Therefore, the present invention further provides a self-checkout detachable kit 400. This kit preferably provides the checkout management device 311 with the necessary components for upgrading to a self-checkout device: a first proximity sensor 431, a second proximity sensor 432, a third proximity sensor 433, a first intelligent image sensor 441, a second intelligent image sensor 442, a high-position intelligent image sensor 443, a control module 420, and an information display 425. The control module 420 already has built-in self-checkout software 200. By installing the self-checkout detachable kit 400, an existing checkout management device 311 can be easily converted into a self-checkout device with self-checkout functionality.

[0119] In one embodiment, the item sensor group, including the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the first intelligent image sensor 441, and the second intelligent image sensor 442, is preferably integrated into an item sensor assembly 450 and attached to the bottom of the checkout management device 311. The information display 425 is preferably configured above the checkout management device 311 via a first bracket 451, or placed on the side of the checkout management device 311. The high-position intelligent image sensor 443 is also preferably disposed above the checkout management device 311 via a second bracket 452.

[0120] In one embodiment, the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the first intelligent image sensor 441, the second intelligent image sensor 442, the high-position intelligent image sensor 443, the control module 420, and the information display 425 are ultimately connected to the checkout management device 311 through an appropriate communication interface 453, such as a USB communication interface, an HDMI communication interface, etc. In one embodiment, the control module 420 is preferably attached to the checkout management device 311 via a USB OTG.

[0121] Figure 6 A first flow chart illustrating the implementation steps of a first self-service checkout method included in the present invention; Figure 7 A second flow chart illustrating the implementation steps of a first self-checkout method included in the present invention is provided. Taking the aforementioned self-checkout system 100 as an example but not limited to the self-checkout system 100, the first self-checkout method 500 included in the present invention preferably includes, but is not limited to, the following steps:

[0122] Step 501: System standby

[0123] (a) The information display 25 plays an advertisement; (b) The high-position intelligent image sensor 43 performs face detection; and (c) The first to third proximity sensors 31-33 continue to detect whether there is a person approaching, YES, go to step 502.

[0124] Step 502: Checkout

[0125] (a) The high-position intelligent image sensor 43 detects a human face or the first to third proximity sensors 31 - 33 sense that a consumer wants to check out. If YES, go to step 503; if NO, go back to step 501.

[0126] Step 503: Start the checkout process

[0127] (a) The self-checkout software 200 / control module 20 starts the checkout process; (b) The host 11 prompts the consumer with the checkout steps through voice; (c) The information display 25 plays the self-checkout SOP; (d) The touch screen 12 displays the self-checkout operation interface; (e) The second intelligent image sensor 42 is aroused to start the object recognition independent model M3; and (f) The first to third proximity sensors 31-33 continuously grasp the distance between the hand H, the information code C, and the object P

[0128] Step 504: Checkout Process

[0129] (a) The host 11 plays voice to assist the self-service checkout process; (b) The information display 25 plays the self-service checkout SOP video according to the customer's steps; and (c) The code scanner 13 is called to execute the information code reading execution sequence.

[0130] Step 505: Determine the item

[0131] If (a) the second smart image sensor 42 detects that the guest is holding an item P; (b) the second smart image sensor 42 detects that the number of items P exceeds one; and (c) the third proximity sensor 33 detects that an item P is present in the item sensing area PSA through distance sensing, YES, then proceed to step 506.

[0132] Step 506: Determine the scanner

[0133] (a) The self-checkout software 200 / control module 20 determines whether the barcode scanner 13 has read the information code C of the item P within 5 seconds. If yes, the process goes to step 507 .

[0134] Step 507: Confirm the transaction

[0135] Step 508: Determine the item

[0136] (a) The second intelligent image sensor 42 detects whether the item P in the guest's hand disappears. If NO, go to step 509; if YES, go to step 514.

[0137] Step 509: Determine the information code

[0138] (a) Instruct the first intelligent image sensor 41 to capture the position of the information code C and determine whether the information code C is in the wrong direction. If yes, proceed to step 510.

[0139] Step 510: Handling missed scan events

[0140] (a) The touch screen 12 prompts that the information code position is incorrect, and the host 11 reminds the user with a voice message and continues to lock the self-checkout operation interface and disable the touch screen 12 until the same item P completes the card swiping action.

[0141] Step 511: Confirm that the abnormality is resolved

[0142] Step 512: Checkout completed? If yes, go to step 513; if no, go back to step 504.

[0143] Step 513: Confirm unchecked items

[0144] (a) Confirm whether the item P read by the second intelligent image sensor 42 is successfully checked out. If YES, end; if NO, return to step 504.

[0145] Step 514: Abnormal Transaction

[0146] (a) The touch screen 12 displays a prompt and the host 11 plays a voice prompt to remind the user; (b) the self-checkout software 200 / control module 20 locks the self-checkout interface and disables the touch screen 12 until the same item P is swiped; (c) the indicator light on the host 11 continues to flash; and (d) the self-checkout software 200 / control module 20 activates the remote video option for the user to select.

[0147] Figure 8 A flowchart of the implementation steps of the second self-service checkout method included in the present invention is disclosed; the second self-service checkout method 600 included in the present invention preferably includes but is not limited to the following steps: the following steps are executed by the control module (step 601): driving the first image sensor to obtain a first image including a hand (step 602); driving the first image sensor to obtain the first image including an information code (step 603); driving the second image sensor to obtain a second image including an item (step 604); driving the high-position image sensor to obtain a third image including one of the consumer, the hand, the item, and the information code (step 605).

[0148] Drive the first proximity sensor to detect first distance information about the hand, the object or the information code, and obtain a first sensing output (step 606); drive the second proximity sensor to detect second distance information about the hand, the object or the information code, and obtain a second sensing output (step 607); drive the third proximity sensor to detect third distance information about the hand, the object or the information code, and obtain a third sensing output (step 608); drive the barcode scanner to read the information code, and obtain a fourth sensing output (step 609); execute the gesture recognition independent model to generate a first recognition result based on the first image (step 610); execute the object recognition independent model to generate a third recognition result based on the second image (step 611); execute the information code recognition model to generate a second recognition result based on the first image (step 612); and execute and input the first to the third recognition results, the first to the third images and the first to the fourth sensing outputs into the checkout auxiliary decision independent model to thereby identify the consumption behavior and determine whether a missed swipe event occurs (step 613).

[0149] The above embodiments of the present invention may be arbitrarily combined or replaced with each other to derive more implementation methods, but all of them are within the scope of protection of the present invention. Further embodiments of the present invention are provided as follows:

[0150] Embodiment 1: A self-service checkout method comprises: executing the following steps by a control module: driving a first image sensor to acquire a first image including a hand; driving a second image sensor to acquire a second image including an object; executing an independent gesture recognition model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; and executing and inputting the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event occurs.

[0151] Example 2: The self-service checkout method as described in Example 2 further includes one of the following: performing the following steps through the control module: driving the first image sensor to obtain the first image including the information code; driving the high-position image sensor to obtain a third image including one of the consumer, the hand, the object and the information code; driving the first proximity sensor to detect first distance information about the hand, the object or the information code, and obtaining a first sensing output; driving the second proximity sensor to detect second distance information about the hand, the object or the information code, and obtaining a second sensing output; driving the third proximity sensor to detect third distance information about the hand, the object or the information code, and obtaining a third sensing output; driving the barcode scanner to read the information code, and obtaining a fourth sensing output; executing the information code recognition model to generate a second recognition result based on the first image; and executing and inputting the first to the third recognition results, the first to the third images and the first to the fourth sensing outputs into the checkout auxiliary decision-making independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

[0152] Example 3: The self-service checkout method as described in Example 2, wherein the checkout auxiliary decision independent model is configured to execute one of the following self-service checkout judgment rules: when the information code distance is greater than the effective reading distance of the scanner, it is judged that a barcode out-of-range event has occurred; when the third recognition result indicates that the item is successfully detected, but the scanner does not successfully read the information code, it is judged that a missed scan event has occurred; when the third recognition result indicates that the item is successfully detected, and the second recognition result indicates that the information code is successfully obtained, but the scanner does not successfully read the information code, it is judged that a missed scan event has occurred; when the third recognition result indicates that the number of detected items is greater than the number of items successfully scanned by the scanner, it is judged that a missed scan event has occurred; when the third recognition result indicates that the number of detected items is less than the number of items successfully scanned by the scanner, it is judged that a duplicate checkout event has occurred; and when the third recognition result indicates that the successfully detected item disappears, it is judged that an abnormal checkout event has occurred.

[0153] Embodiment 4: A self-service checkout system comprises: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including an object; and a control module configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event occurs.

[0154] Embodiment 5: The self-checkout system as described in embodiment 4 further comprises one of the following: the first image sensor, which is configured to obtain the first image including the information code; the high-position image sensor, which is configured to obtain a third image including one of the consumer, the hand, the item, and the information code; the first proximity sensor, which is configured to detect first distance information about the hand, the item, or the information code, and obtain a first sensing output; the second proximity sensor, which is configured to detect second distance information about the hand, the item, or the information code, and obtain a second sensing output. output; a third proximity sensor, which is configured to detect third distance information about the hand, the object or the information code, and obtain a third sensing output; a barcode scanner, which is configured to read the information code and obtain a fourth sensing output; and the control module, which is configured to: execute the information code recognition model to generate a second recognition result based on the first image; and execute and input the first to the third recognition results, the first to the third images and the first to the fourth sensing output into the checkout auxiliary decision-making independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

[0155] Example 6: A self-service checkout device, comprising: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including an object; and a control module configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0156] Embodiment 7: The self-checkout device as described in embodiment 6 further comprises one of the following: the first image sensor, which is configured to obtain the first image including the information code; the high-position image sensor, which is configured to obtain a third image including the consumer, the hand, the item, and the information code; the first proximity sensor, which is configured to detect first distance information about the hand, the item, or the information code, and obtain a first sensing output; the second proximity sensor, which is configured to detect second distance information about the hand, the item, or the information code, and obtain a second sensing output. output; a third proximity sensor, which is configured to detect third distance information about the hand, the object or the information code, and obtain a third sensing output; a barcode scanner, which is configured to read the information code and obtain a fourth sensing output; and the control module, which is configured to: execute the information code recognition model to generate a second recognition result based on the first image; and execute and input the first to the third recognition results, the first to the third images and the first to the fourth sensing output into the checkout auxiliary decision-making independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

[0157] Example 8: A self-service checkout kit, which is attached to a checkout management device, comprises: a first image sensor, which is configured to acquire a first image containing a hand; a second image sensor, which is configured to acquire a second image containing an object; and a control module, which is integrated into the checkout management device, or is separated from the checkout management device but maintains a communication link, the control module being configured to: execute an independent gesture recognition model to generate a first recognition result based on the first image; execute an independent object recognition model to generate a third recognition result based on the second image; and execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed swipe event has occurred.

[0158] Embodiment 9: The self-checkout kit according to embodiment 8 further comprises one of the following: a first image sensor configured to acquire the first image including the information code; a high-position image sensor configured to acquire a third image including one of the consumer, the hand, the item, and the information code; a first proximity sensor configured to detect first distance information about the hand, the item, or the information code and obtain a first sensing output; a second proximity sensor configured to detect second distance information about the hand, the item, or the information code and obtain a second sensing output. output; a third proximity sensor, which is configured to detect third distance information about the hand, the object or the information code, and obtain a third sensing output; a barcode scanner, which is configured to read the information code and obtain a fourth sensing output; and the control module, which is configured to: execute the information code recognition model to generate a second recognition result based on the first image; and execute and input the first to the third recognition results, the first to the third images and the first to the fourth sensing output into the checkout auxiliary decision-making independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

[0159] Embodiment 10: The self-service checkout kit as described in embodiment 8, wherein the checkout management device is a point of sale (POS) machine or a cash register.

[0160] The various embodiments of the present invention may be arbitrarily combined or replaced with each other to derive more implementation methods, but all of these are within the scope of protection of the present invention. The definition of the scope of protection of the present invention shall be based on the claims of the present application.

Claims

1. A self-service checkout method, comprising: Perform the following steps via the control module: driving the first image sensor to acquire a first image including a hand; driving a second image sensor to acquire a second image including the object; executing a gesture recognition independent model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; as well as Execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed check event occurs.

2. The self-service checkout method according to claim 1, further comprising one of the following: The control module performs the following steps: driving the first image sensor to acquire the first image including the information code; driving a high-position image sensor to acquire a third image including one of a consumer, the hand, the object, and the information code; driving a first proximity sensor to detect first distance information about the hand, the object, or the information code, and obtaining a first sensing output; driving a second proximity sensor to detect second distance information about the hand, the object, or the information code, and obtaining a second sensing output; driving a third proximity sensor to detect third distance information about the hand, the object, or the information code, and obtaining a third sensing output; driving a code scanner to read the information code and obtain a fourth sensing output; executing an information code recognition model to generate a second recognition result based on the first image; as well as Execute and input the first recognition result to the third recognition result, the first image to the third image, and the first sensing output to the fourth sensing output into the checkout auxiliary decision independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

3. The self-service checkout method according to claim 2, wherein the checkout auxiliary decision-making independent model is configured to execute one of the following self-service checkout judgment rules: When the information code distance is greater than the effective reading distance of the barcode scanner, it is determined that a barcode out-of-distance event occurs; When the third recognition result indicates that the item is successfully detected, but the barcode scanner fails to read the information code, it is determined that the missed scan event occurs; When the third recognition result indicates that the object is successfully detected, and the second recognition result indicates that the information code is successfully obtained, but the barcode scanner fails to read the information code, it is determined that the missed scan event occurs; When the third recognition result indicates that the number of detected objects is greater than the number of objects successfully scanned by the scanner, it is determined that the missed scan event occurs; When the third recognition result indicates that the number of items detected is less than the number of items successfully scanned by the barcode scanner, determining that a duplicate checkout event has occurred; as well as When the third recognition result indicates that the successfully detected item disappears, it is determined that an abnormal checkout event occurs.

4. A self-checkout system comprising: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including the article; as well as A control module configured to: executing a gesture recognition independent model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; as well as Execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed check event occurs.

5. The self-service checkout system according to claim 4, further comprising one of the following: The first image sensor is configured to acquire the first image including the information code; a high-position image sensor configured to acquire a third image including one of the consumer, the hand, the item, and the information code; a first proximity sensor configured to detect first distance information about the hand, the object, or the information code and obtain a first sensing output; a second proximity sensor configured to detect second distance information about the hand, the object, or the information code and obtain a second sensing output; a third proximity sensor configured to detect third distance information about the hand, the object, or the information code and obtain a third sensing output; a code scanner configured to read the information code and obtain a fourth sensing output; as well as The control module is configured to: executing an information code recognition model to generate a second recognition result based on the first image; as well as Execute and input the first recognition result to the third recognition result, the first image to the third image, and the first sensing output to the fourth sensing output into the checkout auxiliary decision independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

6. A self-service checkout device, comprising: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including the article; as well as A control module configured to: executing a gesture recognition independent model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; as well as Execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed check event occurs.

7. The self-service checkout device according to claim 6, further comprising one of the following: The first image sensor is configured to acquire the first image including the information code; a high-position image sensor configured to acquire a third image including one of the consumer, the hand, the item, and the information code; a first proximity sensor configured to detect first distance information about the hand, the object, or the information code and obtain a first sensing output; a second proximity sensor configured to detect second distance information about the hand, the object, or the information code and obtain a second sensing output; a third proximity sensor configured to detect third distance information about the hand, the object, or the information code and obtain a third sensing output; a code scanner configured to read the information code and obtain a fourth sensing output; as well as The control module is configured to: executing an information code recognition model to generate a second recognition result based on the first image; as well as Execute and input the first recognition result to the third recognition result, the first image to the third image, and the first sensing output to the fourth sensing output into the checkout auxiliary decision independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

8. A self-checkout kit, attached to a checkout management device, comprising: a first image sensor configured to acquire a first image including a hand; a second image sensor configured to acquire a second image including the article; as well as A control module, which is integrated within the checkout management device or is separate from the checkout management device but maintains a communication link with the checkout management device, is configured to: executing a gesture recognition independent model to generate a first recognition result based on the first image; executing an independent object recognition model to generate a third recognition result based on the second image; as well as Execute and input the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed check event occurs.

9. The self-checkout kit according to claim 8, further comprising one of the following: The first image sensor is configured to acquire the first image including the information code; a high-position image sensor configured to acquire a third image including one of the consumer, the hand, the item, and the information code; a first proximity sensor configured to detect first distance information about the hand, the object, or the information code and obtain a first sensing output; a second proximity sensor configured to detect second distance information about the hand, the object, or the information code and obtain a second sensing output; a third proximity sensor configured to detect third distance information about the hand, the object, or the information code and obtain a third sensing output; a code scanner configured to read the information code and obtain a fourth sensing output; as well as The control module is configured to: executing an information code recognition model to generate a second recognition result based on the first image; as well as Execute and input the first recognition result to the third recognition result, the first image to the third image, and the first sensing output to the fourth sensing output into the checkout auxiliary decision independent model to thereby identify the consumer's consumption behavior and determine whether the missed swipe event occurs.

10. The self-service checkout kit according to claim 8, wherein the checkout management device is a point of sale machine or a cash register.

Images