Self-checkout method, system, device and suite

Abstract

The invention provides a self-service checkout method, system, device and suite, and the method comprises the steps: carrying out the following steps at a self-service terminal through a control module: continuously detecting the occurrence of a checkout interruption event; driving a plurality of sensors to obtain a plurality of sensing information, executing a plurality of independent models to identify the sensing information, generating a plurality of identification results according to the sensing information, and judging whether the checkout interruption event occurs according to the identification results; when the checkout interruption event occurs: disabling a first display device providing checkout operation, and forwarding the sensing information and the identification results to a remote service station; and establishing a transmission connection line with the remote service station, so that the user can carry out instant messaging with the remote service station through the second display device.

Description

Technical Field

[0001] The present invention relates to a self-service checkout method, system, device and kit, and more particularly to a self-service checkout method, system, device and kit capable of automatically providing remote assistance in response to a checkout interruption event. Background Art

[0002] The core advantage of SCO technology is that customers can complete their shopping and checkout quickly, especially during periods of high customer traffic, which helps reduce waiting times. Furthermore, since SCO technology does not require cashier operation and a single staff member can simultaneously oversee multiple SCO terminals, it helps reduce the number of employees and corresponding labor costs. This is especially true in the traditional cashier checkout model, where cashier costs have always been one of the main expenses. Introducing SCO technology allows stores to operate more checkout stations with less manpower.

[0003] However, this new form of self-checkout has also created many new issues that need to be addressed. Common problems include customers unfamiliar with self-checkout procedures, not aligning the product barcode with the scanner without additional assistance, placing the product too far from the scanner, and making duplicate purchases. While most simple issues can be resolved directly by the self-checkout terminal's built-in artificial intelligence without the need for back-end customer service personnel, other issues remain beyond the reach of AI and require the intervention of back-end customer service personnel.

[0004] According to the current development of self-checkout technology, when a problem occurs at a checkout site, the SCO terminal will not provide detailed information about the problem to customer service personnel. Customer service personnel need to rely on customers' descriptions to understand the problem. However, because customers are not familiar with self-checkout technology, these descriptions usually do not provide enough information and cannot enable customer service to quickly resolve the problem.

[0005] This causes customer service to spend more time understanding and confirming the problem, causing them to be stuck dealing with a single issue for a long time and unable to assist with checkout issues encountered by other customers. Instead, it causes more trouble and inconvenience to consumers and prevents customer service from supervising other self-service checkout stations.

[0006] Therefore, in view of the shortcomings and deficiencies of existing self-checkout technology, the inventors of this case, after careful experimentation and research, and with a spirit of perseverance, finally conceived the present "self-checkout method, system, device and kit", which can overcome many of the shortcomings of the existing technology. The following is a brief description of this invention. Summary of the Invention

[0007] The present invention relates to a self-service checkout method, system, device and kit, and more particularly to a self-service checkout method, system, device and kit that can automatically provide remote assistance in response to a checkout interruption event.

[0008] Accordingly, the present invention provides a self-service checkout method, comprising: at a self-service terminal, executing, via a control module, the following steps: continuously detecting the occurrence of a checkout interruption event; driving multiple sensors to acquire multiple sensing information and executing multiple independent models to identify the sensing information and generate multiple recognition results based on the sensing information, and determining whether the checkout interruption event has occurred based on the identification results; when the checkout interruption event occurs: disabling a first display device providing checkout operations and forwarding the sensing information and the recognition results to a remote service station; and establishing a transmission connection with the remote service station to facilitate real-time communication between the user and the remote service station via a second display device.

[0009] The present invention further provides a self-service checkout system, comprising: a self-service terminal including a control module configured to: continuously detect the occurrence of a checkout interruption event; drive a plurality of sensors to acquire a plurality of sensing information and execute a plurality of independent models to identify the sensing information and generate a plurality of recognition results based on the sensing information, and thereby determine whether the checkout interruption event has occurred; when the checkout interruption event occurs: disable a first display device that provides checkout operations and forward the sensing information and the recognition results to a remote service station; and establish a transmission connection with the remote service station to facilitate real-time communication between the user and the remote service station via a second display device.

[0010] The present invention further provides a self-service checkout device, comprising: a control module configured to execute the following steps: continuously detecting the occurrence of a checkout interruption event; driving multiple sensors to obtain multiple sensing information and executing multiple independent models to identify the sensing information and generate multiple recognition results based on the sensing information, and determining whether the checkout interruption event has occurred based on the sensing information; when the checkout interruption event occurs: disabling a first display device that provides checkout operations and forwarding the sensing information and the recognition results to a remote service station; and establishing a transmission connection with the remote service station to facilitate real-time communication between the user and the remote service station via a second display device.

[0011] The present invention further provides a self-checkout kit that is attached to a checkout management device and includes a control module that is integrated within the checkout management device or separately disposed from the checkout management device but maintains a communication link. The control module is configured to: continuously detect the occurrence of a checkout interruption event; drive multiple sensors to obtain multiple sensing information and execute multiple independent models to identify the sensing information and generate multiple recognition results based on the sensing information, and thereby determine whether the checkout interruption event has occurred; when the checkout interruption event occurs: disable a first display device that provides checkout operations and forward the sensing information and the recognition results to a remote service station; and establish a transmission connection with the remote service station to facilitate real-time communication between the user and the remote service station via a second display device.

[0012] The above invention summary is intended to provide a simplified summary of the present disclosure so that readers can have a basic understanding of the present disclosure. This invention summary is not a complete description of the present invention and is not intended to point out the important / critical elements of the embodiments of the present invention or to define the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram illustrating the system architecture of a self-service checkout system included in the present invention;

[0014] Figure 2 is a schematic diagram illustrating the hardware architecture of the control module included in the present invention;

[0015] Figure 3 is a block diagram illustrating the cooperation between the self-checkout software and various hardware components included in the present invention;

[0016] Figure 4 is a schematic diagram illustrating a checkout interruption mode included in the present invention;

[0017] Figure 5 is a block diagram illustrating a deep learning model included in the present invention;

[0018] Figure 6 is a schematic diagram of the system architecture of a first embodiment of a self-service checkout separation kit included in the present invention;

[0019] Figure 7 is a schematic diagram of the system architecture of a second embodiment of a self-service checkout separation kit included in the present invention;

[0020] Figure 8 A flowchart illustrating the implementation steps of the self-service checkout method included in the present invention; and

[0021] Figure 9 It is a flow chart showing the implementation steps of the checkout interruption event determination method included in the present invention.

[0022] Description of Reference Numerals

[0023] 100 Self-checkout system

[0024] 110 Self-service terminal

[0025] 170 Image Server

[0026] 150 Remote Service Station

[0027] 151 Remote Assistant

[0028] 12. First display device

[0029] 13 Information code sensor

[0030] 14 Credit card machine

[0031] 20 Control Module

[0032] 21 Basic Input / Output System (BIOS) firmware

[0033] 22 Embedded Controller

[0034] 23 processors

[0035] 24 physical remote assistance call buttons

[0036] 25 Touch remote assistance call button

[0037] 26 Second display device

[0038] 27 Microphone

[0039] 28 speakers

[0040] 29MCU

[0041] 31 First proximity sensor

[0042] 32 Second proximity sensor

[0043] 33 Third proximity sensor

[0044] 41 First Image Sensor

[0045] 42 Second image sensor

[0046] 43 High-position image sensor

[0047] 200 Self-Checkout Software

[0048] 210 Unified Endpoint Management Platform Software

[0049] 311 Checkout Management Equipment

[0050] 312 touch screen

[0051] 313 barcode scanner

[0052] 314 Credit Card Machine

[0053] 400 Self-Checkout Separation Kit

[0054] 405 housing

[0055] 410 External Control Box

[0056] 420 Control Module

[0057] 424 Physical remote assistance call button

[0058] 425 Touch remote assistance call button

[0059] 426 Second Display Device

[0060] 427 Microphone

[0061] 428 Speaker

[0062] 431 First proximity sensor

[0063] 432 Second proximity sensor

[0064] 433 Third proximity sensor

[0065] 441 First Image Sensor

[0066] 442 second image sensor

[0067] 443 High-position image sensor

[0068] 450 Hardware component assembly

[0069] 451 First Bracket

[0070] 452 Second bracket

[0071] 453 transmission interface

[0072] 461 First Network Connector

[0073] 462 DisplayPort connector

[0074] 463 First USB Connector

[0075] 464 Universal Input / Output Connector

[0076] 465 Communication power supply connector

[0077] 466 Integrated Communication Connector

[0078] 467 serial communication port

[0079] 468 Second network connector

[0080] 469 Second USB connector

[0081] 471 Network

[0082] 472 barcode scanner

[0083] 473 Second Display

[0084] 474 indicator

[0085] H Hand

[0086] C Information Code

[0087] P product

[0088] PSA product sensing area

[0089] SCO self-checkout area

[0090] S1 First image sensing information

[0091] S2 Second image sensing information

[0092] S3 Third image sensing information

[0093] D1 First distance sensing information

[0094] D2 Second distance sensing information

[0095] D3 third distance sensing information

[0096] D4 Fourth information code sensing information

[0097] M1 gesture recognition standalone model

[0098] M2 Information Code Recognition Independent Model

[0099] M3 Object Recognition Standalone Model

[0100] M4 Checkout Decision Support Independent Model

[0101] R1 First recognition result

[0102] R2 Second recognition result

[0103] R3 Third recognition result

[0104] R4 Fourth recognition result

[0105] 500 First Self-Checkout Method

[0106] 501-514 Implementation Steps

[0107] 600 Second self-checkout method

[0108] 601-605 Implementation Steps DETAILED DESCRIPTION

[0109] The present invention will be fully understood through the following examples, allowing those skilled in the art to complete the invention accordingly. However, the implementation of the present invention is not limited to the following examples. The drawings of the present invention do not include any limitations on size, dimensions, or scale. The sizes, dimensions, and scales of the present invention in actual implementation are not limited by the drawings of the present invention.

[0110] The term "preferably" used herein is non-exclusive and should be understood as "preferably but not limited to". Any steps described or recorded in any specification or claims may be performed in any order and are not limited to the order described in the claims. The scope of the present invention should be determined only by the appended claims and their equivalents, and should not be determined by the examples of the implementation examples. The term "including" and its variations when appearing in the specification and claims herein are open-ended terms without restrictive meanings and do not exclude other features or steps.

[0111] Figure 1 1 is a schematic diagram illustrating the system architecture of a self-checkout system according to the present invention. In one embodiment, the self-checkout system (self-checkout device) 100 according to the present invention preferably includes a self-service terminal 110, a remote service station 150, and an image server 170. The self-service terminal 110 includes a first display device 12, an information code sensor 13, a card reader 14, a first proximity sensor 31, a second proximity sensor 32, a third proximity sensor 33, a first image sensor 41, a second image sensor 42, a high position image sensor 43, a control module 20, a physical remote assistance call button 24, a touch remote assistance call button 25, a second display device 26, a microphone 27, and a speaker 28. The control module 20 has a built-in self-checkout software 200.

[0112] The self-service terminal (SST) 110 is preferably an electronic device or host with point-of-sale (POS) functions, or directly a POS terminal, and is equipped with a first display device 12, an information code sensor 13, a card reader 14, and a control module 20 to implement the basic manual checkout functions of a POS terminal. The first display device 12 is preferably a touch screen, the second display device 26 is preferably a flat-panel display or a touch screen, and the control module 20 is preferably configured inside the self-service terminal 110, or is externally connected to the control circuit inside the self-service terminal 110 for communication.

[0113] A product sensor group is disposed below the self-checkout system 100 or self-service terminal 110. The product sensor group includes multiple sensors, such as, but not limited to, a first proximity sensor 31, a second proximity sensor 32, a third proximity sensor 33, a first image sensor 41, a second image sensor 42, a high-position image sensor 43, and an information code sensor 13. The product sensor group is preferably disposed below the self-service terminal 110, forming a virtual, three-dimensional product sensing area (PSA) in front of the self-service terminal 110.

[0114] In one embodiment, the first proximity sensor 31, the second proximity sensor 32, and the third proximity sensor 33 are preferably distance sensors, time-of-flight (TOF) sensors, multi-object tracking (MOT) sensors, gesture recognition sensors, optical motion trackers, motion sensors, proximity sensors, active infrared (AIR) sensors, passive infrared (PIR) sensors, or consumer infrared (CIR) sensors.

[0115] Figure 2 2 is a schematic diagram illustrating the hardware architecture of a control module included in the present invention. The self-checkout method proposed in the present invention is preferably implemented in the form of self-checkout software 200, i.e., a self-checkout computer program product, installed on a control module 20 included in a self-checkout system 100. The control module 20 includes at least components such as basic input / output system (BIOS) firmware 21, an embedded controller (EC) 22, and a processor 23. The self-checkout software 200 is loaded and executed by the processor 23 to implement the self-checkout method through the control module 20 and perform various controls on the self-checkout system 100.

[0116] The control module 20 defines a certain area around the product sensing area PSA as a virtual self-checkout region SCO. Then, based on the image captured by the high-position image sensor 43, it detects whether a user has entered the self-checkout region SCO, thereby initiating the self-checkout process accordingly.

[0117] When no customers enter the self-service checkout area SCO, the control module 20 switches the second display device 26 and the first display device 12 to the advertising mode to play preset advertising content, and controls the multiple sensors included in the product sensor group to enter a low-power standby state to reduce the overall device energy consumption of the self-service checkout system 100.

[0118] When the control module 20 determines, based on the image captured by the high-position image sensor 43, that a user has entered the self-checkout area SCO, the self-checkout process will be initiated. Alternatively, when the control module 20 detects that any sensor included in the product sensor group is triggered, the control module 20 will also correspondingly initiate the self-checkout process, including switching the second display device 26 and the first display device 12 to the checkout mode.

[0119] After entering the checkout mode, the first display device 12 will display the self-checkout operation interface to provide the user with self-checkout operations, while the second display device 26 will display checkout auxiliary information, such as but not limited to: guidance content on the self-checkout standard operating procedures, showing and explaining the self-checkout operation steps to the user.

[0120] Figure 3 This block diagram illustrates the interplay between the self-checkout software and hardware components included in the present invention. The self-checkout software 200 is executed by a control module 20 and includes four independent models: a gesture recognition model M1, an information code recognition model M2, an object recognition model M3, and a checkout decision support model M4. In one embodiment, these independent models M1-M4 are preferably executed by the control module 20 on the device or local side to implement edge computing.

[0121] In a certain embodiment, the first proximity sensor 31, the second proximity sensor 32 and the third proximity sensor 33 are preferably time-of-flight sensors, wherein the first proximity sensor 31 is preferably configured to sense first distance information about the hand H, the information code C marked on the product P, or the product P in the product sensing area PSA, and generate first distance sensing information D1. The second proximity sensor 32 is preferably configured to sense second distance information about the hand H, the information code C marked on the product P, or the product P in the product sensing area PSA, and generate second distance sensing information D2. The third proximity sensor 33 is preferably configured to sense third distance information about the hand H, the information code C marked on the product P, or the product P in the product sensing area PSA, and generate third distance sensing information D3. The information code sensor 13 is preferably configured to read the information code C marked on the product P in the product sensing area PSA, and generate fourth information code sensing information D4.

[0122] The first image sensor 41 is preferably configured to obtain first image sensing information S1 including the hand H and the information code C appearing on the product P within the product sensing area PSA, and the second image sensor 42 is preferably configured to obtain second image sensing information S2 including the product P within the sensing product sensing area PSA, wherein the product P is preferably attached with an information code C that can be read by the information code sensor 13. The form of the information code (info code) includes but is not limited to: a one-dimensional barcode (barcode) and a two-dimensional code (QR code), etc. The information code can store various information, including but not limited to the product name, production location, manufacturer, production date, product price, etc.

[0123] The high-position image sensor 43 is preferably located near the top of the self-service checkout system 100, for example, integrated into the upper frame of the second display device 26, or positioned near the top of the second display device 26 via a bracket or housing. Positioning the high-position image sensor 43 at a higher position allows for a larger and wider field of view (FOV), enabling a panoramic recording of the user's entire self-service checkout process from a commanding vantage point, and generating third image sensing information S3.

[0124] After the self-checkout process is initiated, the control module 20 is preferably configured to drive the first display device 12 to selectively generate a touch remote assistance call button 25 and continuously detect in the background whether the user presses the physical remote assistance call button 24 or touches the remote assistance call button 25 throughout the self-checkout process.

[0125] In a certain embodiment, the control module 20 is preferably configured to input the real-time picture of the first image sensing information S1 acquired by the first image sensor 41 into the gesture recognition independent model M1 and the information code recognition independent model M2 in real time, so as to identify whether the first image sensing information S1 contains a hand and a gesture skeleton through the gesture recognition independent model M1, and identify whether the gesture skeleton contains a checkout action containing the meaning of checkout, and output a first recognition result R1, and then identify the information code contained in the first image sensing information S1 through the information code recognition independent model M2, and output a second recognition result R2.

[0126] In one embodiment, the control module 20 is preferably configured to input the real-time image of the second image sensing information S2 captured by the second image sensor 42 into the object recognition independent model M3 in real time, so as to identify whether the second image sensing information S2 contains a product and a product name through the object recognition independent model M3, and output a third recognition result R3.

[0127] In one embodiment, the control module 20 is preferably configured to input the first recognition result R1 , the second recognition result R2 , and the third recognition result R3 into the checkout auxiliary decision independent model M4 , and output a fourth recognition result R4 .

[0128] In one embodiment, the checkout auxiliary decision-making independent model M4 is preferably a rule-based machine learning model and serves as a self-service checkout auxiliary decision-making model. Figure 3 As disclosed, the checkout auxiliary decision-making independent model M4 will receive all distance sensing information D1-D3, information code sensing information D4, all image sensing information S1-S3 and all recognition results R1-R3 generated by the self-service checkout system 100, and then identify the consumer's consumption behavior according to the built-in preset rules, and determine whether there is any missed product scan event during the self-service checkout process, and output it as the fourth recognition result R4.

[0129] By increasing the number of built-in preset rules in the checkout support decision-making independent model M4 and optimizing the preset rules, the comprehensive capabilities of the checkout support decision-making independent model M4 can be significantly improved. For example, the checkout support decision-making independent model M4 includes but is not limited to the following rules:

[0130] The first type of rules, after execution, can determine whether a non-checkout interruption event has occurred. The first type of rules include:

[0131] Rule 1.1: Determine whether a barcode over-distance event occurs.

[0132] When the third proximity sensor 33 senses that the distance between the product P and the information code sensor 13 is greater than the effective sensing distance of the information code sensor 13, and the second recognition result R2 indicates that the information code C carried by the product P is successfully obtained, and the user's gesture skeleton includes a checkout action, the control module 20 determines that the barcode distance is too far beyond the reading range of the information code sensor 13, and a barcode out-of-distance event occurs. The control module 20 will control the first display device 12 to display a prompt message, the content of which includes prompting the user to move the product closer to the information code sensor 13.

[0133] Rule 1.2: Determine whether a barcode misalignment event occurs.

[0134] When the third recognition result R3 indicates that the product is successfully detected, and the second recognition result R2 indicates that the user's gesture skeleton includes a checkout action, but the information code sensor 13 fails to read the information code successfully, the control module 20 determines that a barcode misalignment event has occurred. The control module 20 will control the first display device 12 to display a product scanning failure prompt message, and the information content includes prompting the user to check whether the product barcode is aligned with the information code sensor 13.

[0135] Rule 1.3: Determine whether a duplicate checkout event has occurred.

[0136] When the third recognition result R3 indicates that the number of detected products is N1, but the number of products scanned by the information code sensor 13 is N2 and is not equal to N1, when N2 is greater than N1, it is determined that a duplicate checkout event has occurred. The control module 20 will control the first display device 12 to display a duplicate checkout prompt message, which includes prompting the user to confirm whether to check out repeatedly.

[0137] In one embodiment, if a duplicate checkout event occurs, the first display device 12 will display a message "Do you want to continue checking out?" to the customer. If the customer chooses to continue, the first display device 12 will display "Please change the scanned item and continue" and continue the self-service checkout process. If the customer chooses not to continue, the self-service checkout process will end.

[0138] Rule 1.4: Determine whether a non-sales venue merchandise event has occurred.

[0139] When the third recognition result R3 indicates that the detected product is not found in the product inventory, it is determined that a non-sale location product event has occurred. The control module 20 controls the first display device 12 to display a non-sale location product prompt message, which includes prompting the user to confirm whether the checkout product is from a sales location.

[0140] In one embodiment, if a non-sale product event occurs, the first display device 12 will display a message "Do you want to continue with checkout?" to the customer. If the customer chooses to continue, the first display device 12 will display "Please change the scanned product and continue" and continue the self-checkout process. If the customer chooses not to continue, the self-checkout process will end.

[0141] The second type of rules, after being executed, can determine whether a checkout interruption event has occurred. When the checkout auxiliary decision-making independent model M4 determines that a checkout interruption event has occurred, the control module 20 will force the self-service checkout process to enter the checkout interruption mode. The second type of rules include:

[0142] Rule 2.1: Determine whether a missed scan event has occurred.

[0143] When the third recognition result R3 indicates that the number of detected products is N1, but the number of products successfully scanned by the information code sensor 13, N2, is not equal to N1, and N1 is greater than N2, it is determined that a missed scan event has occurred. At this time, the control module 20 will control the first display device 12 to display a missed scan prompt message, which includes prompting the user to confirm whether the scan code of any product has been missed. If N1 continues to be greater than N2, the control module 20 will lock the screen of the first display device 12 and disable the first display device 12 to suspend the self-checkout process, while notifying the remote assistant 151 located at the remote service station 150.

[0144] Rule 2.2: Determine whether an abnormal checkout event has occurred.

[0145] When the third recognition result R3 indicates that the detected product P suddenly disappears, it is determined that a checkout abnormality event has occurred. The control module 20 will lock the screen of the first display device 12 and disable the first display device 12 to suspend the self-service checkout process, and at the same time notify the remote assistant 151 located at the remote service station 150.

[0146] Rule 2.3: Determine whether a first-category remote assistance call event occurs.

[0147] When the control module 20 detects that the user presses the physical remote assistance call button 24 , it locks the screen of the first display device 12 and disables the first display device 12 to suspend the self-checkout process, while notifying the remote assistant 151 at the remote service station 150 .

[0148] Rule 2.4: Determine whether a second-type remote assistance call event occurs.

[0149] When the control module 20 detects that the user presses the touch remote assistance call button 25 , it locks the screen of the first display device 12 and disables the first display device 12 to suspend the self-checkout process, while notifying the remote assistant 151 at the remote service station 150 .

[0150] In one embodiment, during the entire self-checkout process, the control module 20 is preferably configured to drive the communication unit to establish a wireless or wired peer-to-peer connection (P2P) with a remote image server 170. The control module then drives the communication unit to execute a streaming protocol, such as, but not limited to, the Real Time Streaming Protocol (RTSP), to transmit and store the acquired checkout data set, including real-time images of the image sensing information S1-S3, all sensing information D1-D4, and all recognition results R1-R4, to the image server 170. If the system determines that a problem has occurred in the checkout process, or a checkout interruption has occurred, appropriate action can be taken based on the recorded images of the checkout process.

[0151] Figure 4 is a schematic diagram illustrating the checkout interruption mode included in the present invention. When the control module 20 determines that a checkout interruption event has occurred, the control module 20 will force the self-checkout process to enter the checkout interruption mode, including locking the screen of the first display device 12 and disabling the first display device 12 to suspend the self-checkout process. At the same time, the control module 20 will send a command to the cloud image server 170, instructing the cloud image server 170 to transmit the recorded and stored checkout data set to the remote service station 150.

[0152] Next, the control module 20 will drive the communication unit to establish a communication connection with the remote service station 150, such as but not limited to a peer-to-peer connection, a local area network (LAN) connection, a wireless network connection, or a USB OTG connection. After the communication connection is successfully established, the user can conduct real-time video and audio streaming communication with the remote assistant 151 of the remote service station 150 through devices such as the second display device 26, the microphone 27, and the speaker 28.

[0153] Upon receiving the current checkout data set from the cloud image server 170, the remote assistant 151 can immediately review the image data, recognition results, and sensor information contained therein to immediately identify and confirm any problems encountered during the checkout process. Once the problem is identified, the remote assistant 151 can immediately guide and assist the customer through real-time video streaming, quickly resolving the issue and returning to the normal self-checkout process. Furthermore, the remote assistant 151 can assist the customer with tasks such as, but not limited to, debugging, troubleshooting, system testing, remote assistance, analysis, and configuration.

[0154] In terms of problem handling, the present invention utilizes remote assistance to provide sales and customer problem handling services. The system provides a remote assistant call interface, which is not limited to physical buttons or touch methods and other interfaces that can be used to trigger problem handling functions. When customers, whether on the sales side or the client side, face any problems or situations in using the system, they can call the remote problem handling function through this interface. In terms of remote functions, the system can also be remotely connected to the SST to perform system operation, system testing, troubleshooting, etc., such as connecting to the LAN through a LAN socket, or performing system testing through the USBOTG function, etc., without limitation to any remote function interface.

[0155] Figure 5 This is a block diagram illustrating the deep learning model included in the present invention. The present invention utilizes multiple independent small-scale deep learning models, allowing each independent small-scale model to perform different recognition tasks, replacing the integrated large-scale model. Large-scale models not only take up storage space but also consume a lot of computing power and memory. By splitting the models into small-scale models with independent tasks, not only does the storage space occupied by all small-scale models decrease, but the complexity of each small-scale model is also reduced, significantly reducing the demand for computing power and memory. Furthermore, these small-scale models can directly apply to already trained models, further reducing model training costs, or use open source models to further reduce model construction costs.

[0156] Generally speaking, large supervised deep learning models require extensive training data. More labeled training data translates to higher development costs. Furthermore, larger supervised deep learning models converge more easily if they are broken down into several smaller models and trained separately before being integrated, rather than using end-to-end training.

[0157] Based on the above two principles, this invention proposes a training method and decision-making system that combines multiple models to detect consumer checkout behavior. By combining multiple common smaller models in series instead of a single large point-to-point model, the difficulty and cost of training the model are reduced.

[0158] The present invention conducts model training in two stages. In the first stage of model training, each model can be trained separately, or the existing model and weights can be directly used for transfer learning. In the second stage of model training, the trained small models are connected in series to complete the end-to-end large model training.

[0159] In one embodiment, the gesture recognition independent model M1 is preferably a convolutional neural network (CNN), a 3D convolutional neural network (3DCNN), a regional convolutional neural network (R-CNN), a graph convolutional network (GCN), a long short-term memory network (LSTM), a bidirectional long short-term memory network (Bi-LSTM), a temporal convolutional network (TCN), an OpenPose model, a GestureNet model, a BlazePalm model, a Landmark model, a MediaPipe model, a Rekognition model, a support vector machine (SVM), or a combination thereof.

[0160] In a certain embodiment, the information code recognition independent model M2 is preferably a ZBar model, an OpenCV model, a PyZbar model, a Google Vision model or a combination thereof, and the object recognition independent model M3 is preferably a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, a Faster R-CNN model, a Mask R-CNN model, a ResNet model, an EfficientDet model, an EfficientNet model, a VGG model, a VGGNet model, a COCO model, a CNN model, a MobileNet model, an AlexNet model, a GoogLeNet model or a combination thereof.

[0161] Figure 6 1 is a schematic diagram illustrating the system architecture of the self-checkout separation kit included in the present invention. In one embodiment, the self-checkout method proposed in the present invention is further implemented in the form of a self-checkout separation kit 400 .

[0162] Existing checkout management devices 311, such as POS machines and cash registers, generally do not support self-service checkout. Checkout requires a dedicated operator to operate the checkout management device 311, including scanning product barcodes and operating the checkout interface. However, the existing checkout management device 311 already includes basic hardware components, including a basic control circuit, a first display device 312, a barcode scanner 313, and a card reader 314. It can be upgraded to a self-service checkout device with self-service checkout functionality by simply connecting a product sensor group, an information panel, and the self-service checkout software 200.

[0163] Therefore, the present invention further provides a self-checkout detachable kit 400. This kit preferably provides the checkout management device 311 with the necessary components for upgrading to a self-checkout device: a first proximity sensor 431, a second proximity sensor 432, a third proximity sensor 433, a first image sensor 441, a second image sensor 442, a high-position image sensor 443, a control module 420, a physical remote assistance call button 424, a touch-sensitive remote assistance call button 425, a second display device 426, a microphone 427, and a speaker 428. The control module 420 already has built-in self-checkout software 200. By installing the self-checkout detachable kit 400 of the present invention, an existing checkout management device 311 can be easily converted into a self-checkout device with self-checkout functionality.

[0164] In one embodiment, hardware components such as the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the first image sensor 441, the second image sensor 442, and the physical remote assistance call button 424 are preferably integrated into a hardware component assembly 450 and attached to the bottom of the checkout management device 311. The second display device 426 is preferably configured above the checkout management device 311 via a first bracket 451, or is placed on the side of the checkout management device 311. The high-position image sensor 443 is also preferably disposed above the checkout management device 311 via a second bracket 452.

[0165] In one embodiment, the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the first image sensor 441, the second image sensor 442, the high-position image sensor 443, the control module 420, the physical remote assistance call button 424, the second display device 426, the microphone 427, and the speaker 428 are ultimately connected to the checkout management device 311 through an appropriate transmission interface 453, such as a USB transmission interface or an HDMI transmission interface. In one embodiment, the control module 420 is preferably attached to the checkout management device 311 via USB OTG.

[0166] Figure 7 1 is a schematic diagram illustrating the system architecture of a second embodiment of a self-checkout separation kit according to the present invention. In this embodiment, a control module 420 included in the self-checkout separation kit 400 is separated from the checkout management device 311 and disposed within a box-shaped housing 405 to form an independent external control box 410.

[0167] In one embodiment, the control module 420 is preferably configured with an x86 architecture processor 23, such as but not limited to the Alder Lake-N series processor, and a microcontroller (MCU) 29 as an embedded controller (EC). The control module 420 deploys the self-checkout software 200 and executes the self-checkout software 200 through the processor 23.

[0168] In one embodiment, the control module 420 further deploys a unified endpoint management platform software (UEM / inefi) 210 and executes the unified endpoint management platform software 210 via the processor 23, so that an administrator can remotely manage the checkout management device 311 and make the checkout management device 311 part of the Internet of Things (IoT).

[0169] The unified endpoint management platform software 210 is a cross-hardware device software based on hardware-agnostic principles. It is preferably used in an aggregation manner to uniformly manage and protect all authenticated endpoint devices in a specific organization. All endpoint devices include but are not limited to: service endpoint (POS) devices, sales endpoint (POS) devices, information station (Kiosk) devices, self-checkout (SCO) devices, self-service (self-service) devices, peripheral devices, tablet computers (panel PCs), workstations (workstations), mobile devices, laptops and desktop computers.

[0170] The unified endpoint management platform software 210 includes a remote management backend, allowing administrators to remotely configure, monitor, manage, and secure these devices, regardless of their hardware type or operating system. The unified endpoint management platform software 210 also records the operating status of the self-checkout software 200 and allows users to query the operating information of the self-checkout software 200 through the remote management backend.

[0171] The control module 420 is also equipped with multiple communication connectors, including but not limited to: a first network connector 461 such as an Ethernet connector, a display port connector 462, a first USB connector 463, a general-purpose input / output (GPIO) port connector 464, a communication power connector 465 such as a USB Type-C PD, USB OTG or PoE connector, an integrated communication connector (Fedp) 466, a serial communication port (COM port) connector 467, a second network connector 468 such as an RJ45 connector, a second USB connector 469, and the like.

[0172] The control module 420 preferably obtains the required power from the checkout management device 311 through the communication power connector 465, without the need for an additional electrical connection to another fixed power source such as a wall socket. This not only simplifies the installation process of the external control box 410, but also increases the convenience and flexibility of deploying the external control box 410, making it easier for users to deploy the external control box 410.

[0173] The control module 420 includes two sets of network connectors. The external control box 410 can access the network 471, including the Internet, intranet, and local area network (LAN), through the first network connector 461. The second network connector 468 then acts as a bridge to directly share the network connection with the checkout management device 311, making it easier for users to deploy the external control box 410.

[0174] The first USB connector 463 is preferably connected to the first image sensor 441 , the second image sensor 442 and the barcode scanner 472 , and the display port connector 462 is provided for connecting to an external second display 473 or a client display.

[0175] MCU29 is preferably used to control lower-level hardware controls such as peripheral devices and hardware, such as the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433 and the indicator 474. In this embodiment, the first proximity sensor 431, the second proximity sensor 432 and the third proximity sensor 433 are preferably time-of-flight (TOF) sensors, and the indicator 474 is preferably a light bar.

[0176] For example, the MCU 29 preferably controls the indicator 474 by executing, for example but not limited to, a pulse width modulation (PWM) algorithm, and controls the first proximity sensor 431, the second proximity sensor 432, and the third proximity sensor 433 by executing an I2C communication protocol. The MCU 29 is connected to the checkout management device 311 via the second USB connector 469.

[0177] MCU29 can also be further used for hardware simulation tasks, including but not limited to: simulation of input and output interfaces and simulation of peripheral devices. For example, since the barcode scanner 472 needs to be connected to the external control box 410, MCU29 will simulate the barcode scanner 472 and report it to the checkout management device 311.

[0178] When the checkout management device 311 needs to call and view the real-time image captured by the first image sensor 441 or the second image sensor 442 , the processor 23 executes RTSP between the checkout management device 311 and the image server 170 to report the real-time image to the checkout management device 311 .

[0179] Because the external control box 410 and the self-checkout software 200 are a separate and independent self-checkout system from the existing checkout management device 311, when the checkout management device 311 needs to know the status of the external control box 410, such as but not limited to the status of the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the indicator 474, the first image sensor 441, the second image sensor 442, the second display 473, and the self-checkout software 200, the external control box 410 provides the following access modes for the checkout management device 311 to query information.

[0180] The first access mode uses a microcontroller command set. The checkout management device 311 sends MCU commands to the MCU 29 via the serial communication port connector 467. The MCU responds to the commands and reports information to the checkout management device 311 via the second USB connector 469. The second access mode allows the checkout management device 311 to directly read each pin of the universal I / O port connector 464. Based on the pin definitions of the universal I / O port connector 464, the corresponding information can be obtained.

[0181] The third access mode involves querying the remote management backend of the unified endpoint management platform software 210 via the internet. The unified endpoint management platform software 210 records all operational information of the external control box 410, including, for example, but not limited to, the operating parameters and status of the first proximity sensor 431, the second proximity sensor 432, the third proximity sensor 433, the indicator 474, the first image sensor 441, the second image sensor 442, the second display 473, and the self-checkout software 200. Furthermore, the checkout management device 311 can also access the required operational information of the external control box 410 using the API or SDK provided by the unified endpoint management platform software 210.

[0182] Figure 8The present invention discloses a flowchart of the implementation steps of the self-service checkout method; the self-service checkout method 500 included in the present invention preferably includes but is not limited to the following steps: at the self-service terminal, the control module executes the following steps (step 501): continuously detecting the occurrence of a checkout interruption event (step 502); driving the first display device to provide the user with a checkout operation (step 503); driving the second display device to display checkout auxiliary information to the user (step 504); driving multiple sensors to obtain multiple sensing information and executing multiple independent models to identify the sensing information and based on the sensing information, the self-service terminal ... the first display device to provide the user with a checkout auxiliary information and based on the sensing information, the self-service terminal executes the following steps (step 504); driving the first display device to provide the user with a checkout auxiliary information and based on the sensing information, the self-service terminal executes the following steps (step 50 A plurality of recognition results are generated, and based on the results, it is determined whether the checkout interruption event has occurred (step 505); the sensing information and the recognition results are transmitted to a remote server (step 506); when the checkout interruption event occurs (step 507): the first display device is temporarily disabled to interrupt the checkout operation, and the remote server is instructed to transmit the sensing information and the recognition results to the remote service station (step 508); and the transmission connection is established with the remote service station to enable the user to conduct real-time video and audio streaming communication with the remote service station via the second display device (step 509).

[0183] Figure 9 6 is a flowchart illustrating the implementation steps of the checkout interruption event determination method included in the present invention; the present invention further includes a checkout interruption event determination method 600, which preferably includes but is not limited to the following steps: configuring a control module to execute an independent checkout assistance decision model, wherein the independent checkout assistance decision model is configured to execute one of the following rules to determine whether the checkout interruption event has occurred (step 601): detecting whether a physical remote assistance call button is pressed, and when the physical remote assistance call button is pressed, determining that a first type of remote assistance call event has occurred (step 602); detecting whether a touch remote assistance call button is pressed, and when the touch remote assistance call button is pressed, determining that a second type of remote assistance call event has occurred (step 603); when the third recognition result indicates that the number of detected products is greater than the number of products successfully scanned by the information code sensor, determining that a missed check event has occurred (step 604); and when the third recognition result indicates that the successfully detected product disappears, determining that a checkout abnormality event has occurred (step 605).

[0184] This invention aims to establish an independent and complete system to facilitate the checkout process at sales locations, providing a more convenient, automated, simple, accurate, and remotely assisted checkout system. The proposed system will provide comprehensive support procedures for all stages of the sales process, including before, during, and after checkout. By automatically sensing consumer behavior and switching the system's current operating mode, it offers consumers both self-service and automated checkout processes, while also providing comprehensive sales support at the sales location.

[0185] The complete system architecture of this invention integrates human body sensing, artificial intelligence image recognition, software services, and problem solving into an integrated service. The entire system is a standalone, one-stop service, requiring no human intervention or system equipment to complete comprehensive automated sales services.

[0186] The self-service checkout method proposed in the present invention can assist remote assistants or customer service personnel in quickly identifying problems and quickly intervene as soon as the problem occurs to assist customers in handling and resolving the problem, thereby helping to improve the fluency and efficiency of the entire self-service checkout process.

[0187] The above embodiments of the present invention may be arbitrarily combined or replaced with each other to derive more embodiments, but all of them are within the scope of protection of the present invention. More embodiments of the present invention are further provided as follows:

[0188] Embodiment 1: A self-service checkout method comprises: at a self-service terminal, executing, via a control module, the following steps: continuously detecting the occurrence of a checkout interruption event; driving multiple sensors to acquire multiple sensing information and executing multiple independent models to identify the sensing information and generate multiple recognition results accordingly, and determining whether the checkout interruption event has occurred accordingly; when the checkout interruption event occurs: disabling a first display device that provides a checkout operation and forwarding the sensing information and the recognition results to a remote service station; and establishing a transmission connection with the remote service station to facilitate instant communication between the user and the remote service station via a second display device.

[0189] Embodiment 2: The self-service checkout method described in Embodiment 1 further includes: at the self-service terminal, executing the following steps via the control module: driving the first display device to provide the user with the checkout operation; driving the second display device to display checkout assistance information to the user; driving the sensors to obtain the sensing information; when the checkout interruption event occurs: temporarily disabling the first display device to interrupt the checkout operation, and instructing the remote server to transmit the sensing information and the recognition results to the remote service station; and establishing the transmission connection with the remote service station to enable the user to conduct real-time video streaming communication with the remote service station via the second display device.

[0190] Example 3: The self-service checkout method as described in Example 2 further includes: at the self-service terminal, performing the following steps through the control module: driving the first image sensor to obtain first image sensing information including the user's hand or information code; driving the second image sensor to obtain second image sensing information including the commodity; executing the gesture recognition independent model to recognize the first image and generate a first recognition result; executing the object recognition independent model to recognize the second image sensing information and generate a third recognition result; executing and inputting the first recognition result and the third recognition result into the checkout auxiliary decision independent model to determine whether a missed swipe event has occurred; and when the missed swipe event occurs: determining that the checkout interruption event has occurred.

[0191] Example 4: The self-service checkout method as described in Example 3 further includes: at the self-service terminal, the control module performs the following steps: driving the high-position image sensor to obtain third image sensing information including the user, the hand, the product and one of the information code; driving the first proximity sensor to detect first distance information about the hand, the product or the information code, and obtain first sensing information; driving the second proximity sensor to detect second distance information about the hand, the product or the information code, and obtain second sensing information; driving the third proximity sensor to detect third distance information about the hand, the product or the information code, and obtain third sensing information; driving the information code sensor to read the information code and obtain fourth sensing information; executing the information code recognition independent model to generate a second recognition result based on the first image sensing information; executing and inputting the first to third recognition results, the first to third image sensing information and the first to fourth sensing information into the checkout auxiliary decision independent model to thereby identify the user's consumption behavior and determine whether the missed swipe event occurs; and when the missed swipe event occurs: determining that the checkout interruption event occurs.

[0192] Example 5: The self-service checkout method as described in Example 4 further includes: executing one of the following steps through the control module: driving the first display device to generate a touch remote assistance call button; and detecting whether the physical remote assistance call button or the touch remote assistance call button is pressed.

[0193] Example 6: The self-service checkout method as described in Example 5, wherein the checkout auxiliary decision-making independent model is configured to execute one of the following rules to determine whether the checkout interruption event occurs: when the physical remote assistance call button is pressed, it is determined that a first type of remote assistance call event has occurred; when the touch remote assistance call button is pressed, it is determined that a second type of remote assistance call event has occurred; when the third recognition result indicates that the number of detected products is greater than the number of products successfully scanned by the information code sensor, it is determined that a missed scan event has occurred; and when the third recognition result indicates that the successfully detected product disappears, it is determined that a checkout abnormality event has occurred.

[0194] Embodiment 7: The self-service checkout method as described in embodiment 6, wherein the checkout interruption event includes the first type of remote assistance call event, the second type of remote assistance call event, the missed checkout event, and the abnormal checkout event.

[0195] Embodiment 8: A self-service checkout system, comprising: a self-service terminal, comprising a control module configured to: continuously detect the occurrence of a checkout interruption event; drive a plurality of sensors to obtain a plurality of sensing information and execute a plurality of independent models to identify the sensing information and generate a plurality of recognition results accordingly, and thereby determine whether the checkout interruption event has occurred; when the checkout interruption event occurs: disable a first display device that provides a checkout operation and forward the sensing information and the recognition results to a remote service station; and establish a transmission connection with the remote service station to facilitate instant communication between a user and the remote service station via a second display device.

[0196] Embodiment 9: A self-service checkout device, comprising: a control module configured to execute the following steps: continuously detecting the occurrence of a checkout interruption event; driving multiple sensors to obtain multiple sensing information and executing multiple independent models to identify the sensing information and generate multiple recognition results accordingly, and determining whether the checkout interruption event has occurred accordingly; when the checkout interruption event occurs: disabling a first display device that provides a checkout operation and forwarding the sensing information and the recognition results to a remote service station; and establishing a transmission connection with the remote service station to facilitate instant communication between a user and the remote service station via a second display device.

[0197] Embodiment 10: A self-checkout kit is attached to a checkout management device, comprising: a control module, which is integrated into the checkout management device or separately disposed from the checkout management device but maintains a communication link, the control module being configured to: continuously detect the occurrence of a checkout interruption event; drive multiple sensors to obtain multiple sensing information and execute multiple independent models to identify the sensing information and generate multiple recognition results accordingly, and thereby determine whether the checkout interruption event has occurred; when the checkout interruption event occurs: disable a first display device that provides checkout operations and forward the sensing information and the recognition results to a remote service station; and establish a transmission link with the remote service station to facilitate real-time communication between the user and the remote service station via a second display device.

[0198] Embodiment 11: The self-service checkout kit as described in embodiment 10 further includes one of the following: a first image sensor, which is configured to obtain first image sensing information including the user's hand; a second image sensor, which is configured to obtain second image sensing information including the product; and the first image sensor, which is configured to obtain the first image sensing information including the information code; a high-position image sensor, which is configured to obtain third image sensing information including one of the user, the hand, the product and the information code; a first proximity sensor, which is configured to detect first distance information about the hand, the product or the information code, and obtain first sensing information; a second proximity sensor, which is configured to detect second distance information about the hand, the product or the information code, and obtain second sensing information; a third proximity sensor A sensor configured to detect third distance information about the hand, the product or the information code and obtain third sensing information; an information code sensor configured to read the information code and obtain fourth sensing information; and a control module configured to: execute an independent gesture recognition model to recognize the first image and generate a first recognition result; execute an independent information code recognition model to recognize the first image and generate a second recognition result; execute an independent object recognition model to recognize the second image sensing information and generate a third recognition result; execute and input the first to the third recognition results, the first to the third image sensing information and the first to the fourth sensing information into an independent checkout auxiliary decision model to thereby recognize the user's consumption behavior and determine whether a missed swipe event occurs; and when the missed swipe event occurs: determine that the checkout interruption event occurs.

[0199] Embodiment 12: The self-service checkout kit as described in embodiment 10, wherein the checkout management device is a point of sale (POS) machine or a cash register.

[0200] The various embodiments of the present invention may be arbitrarily combined or replaced with each other to derive more embodiments, but all of these are within the scope of protection of the present invention. The definition of the scope of protection of the present invention shall be based on the claims of the present invention.

Claims

1. A self-service checkout method, comprising: At the self-service terminal, perform the following steps through the control module: Continuously detect checkout interruption events; driving a plurality of sensors to acquire a plurality of sensing information and executing a plurality of independent models to recognize the plurality of sensing information and generate a plurality of recognition results accordingly, and determining whether the checkout interruption event occurs accordingly; When the checkout interruption event occurs: disabling a first display device providing a checkout operation, and forwarding the plurality of sensing information and the plurality of recognition results to a remote service station; as well as A transmission link is established with the remote service station so that the user can communicate with the remote service station in real time through the second display device.

2. The self-service checkout method according to claim 1, further comprising: At the self-service terminal, the control module performs the following steps: driving the first display device to provide the user with the checkout operation; driving the second display device to display checkout auxiliary information to the user; driving the plurality of sensors to obtain the plurality of sensing information; transmitting the plurality of sensing information and the plurality of recognition results to a remote server; as well as When the checkout interruption event occurs: Temporarily disabling the first display device to interrupt the checkout operation, and instructing the remote server to transmit the plurality of sensing information and the plurality of recognition results to the remote service station; as well as The transmission connection is established with the remote service station so that the user can perform real-time video and audio streaming communication with the remote service station through the second display device.

3. The self-service checkout method according to claim 2, further comprising: At the self-service terminal, the control module performs the following steps: driving a first image sensor to acquire first image sensing information including the user's hand or an information code; driving the second image sensor to acquire second image sensing information including the commodity; executing a gesture recognition independent model to recognize the first image and generate a first recognition result; executing an independent object recognition model to recognize the second image sensing information and generate a third recognition result; Executing and inputting the first recognition result and the third recognition result into an independent checkout auxiliary decision model to determine whether a missed check event occurs; as well as When the missed brushing event occurs: It is determined whether the checkout interruption event occurs.

4. The self-service checkout method according to claim 3, further comprising: At the self-service terminal, the control module performs the following steps: driving a high-position image sensor to acquire third image sensing information including the user, the hand, the product, and the information code; driving a first proximity sensor to detect first distance information about the hand, the commodity, or the information code, and obtaining first sensing information; driving a second proximity sensor to detect second distance information about the hand, the commodity, or the information code, and obtaining second sensing information; driving a third proximity sensor to detect third distance information about the hand, the commodity, or the information code, and obtaining third sensing information; driving the information code sensor to read the information code and obtain fourth sensing information; executing an information code recognition independent model to generate a second recognition result based on the first image sensing information; executing and inputting the first to third recognition results, the first to third image sensing information, and the first to fourth sensing information into the checkout auxiliary decision-making independent model to thereby identify the user's consumption behavior and determine whether the missed payment event occurs; as well as When the missed brushing event occurs: It is determined whether the checkout interruption event occurs.

5. The self-service checkout method according to claim 4, further comprising: Perform one of the following steps by the control module: selectively driving the first display device to generate a touch-sensitive remote assistance call button; and Detect whether the physical remote assistance call button or the touch remote assistance call button is pressed.

6. The self-service checkout method according to claim 5, wherein the checkout auxiliary decision-making independent model is configured to execute one of the following rules to determine whether the checkout interruption event occurs: When the physical remote assistance call button is pressed, determining that a first type of remote assistance call event occurs; When the touch remote assistance call button is pressed, determining that a second type of remote assistance call event occurs; When the third recognition result indicates that the number of commodities detected is greater than the number of commodities successfully scanned by the information code sensor, it is determined that the missed scan event occurs; and When the third recognition result indicates that the successfully detected commodity disappears, it is determined that an abnormal checkout event occurs.

7. The self-service checkout method according to claim 6, wherein the checkout interruption event comprises the first type of remote assistance call event, the second type of remote assistance call event, the missed checkout event, and the abnormal checkout event.

8. A self-service checkout system comprising: A self-service terminal comprising a control module for performing the following steps: Continuously detect checkout interruption events; driving a plurality of sensors to acquire a plurality of sensing information and executing a plurality of independent models to recognize the plurality of sensing information and generate a plurality of recognition results accordingly, and determining whether the checkout interruption event occurs accordingly; When the checkout interruption event occurs: disabling a first display device providing a checkout operation, and forwarding the plurality of sensing information and the plurality of recognition results to a remote service station; as well as A transmission link is established with the remote service station so that the user can communicate with the remote service station in real time through the second display device.

9. A self-service checkout device, comprising: The control module performs the following steps: Continuously detect checkout interruption events; driving a plurality of sensors to acquire a plurality of sensing information and executing a plurality of independent models to recognize the plurality of sensing information and generate a plurality of recognition results accordingly, and determining whether the checkout interruption event occurs accordingly; When the checkout interruption event occurs: disabling a first display device providing a checkout operation, and forwarding the plurality of sensing information and the plurality of recognition results to a remote service station; as well as A transmission link is established with the remote service station so that the user can communicate with the remote service station in real time through the second display device.

10. A self-checkout kit, attached to a checkout management device, comprising: A control module is integrated into the checkout management device, or is separate from the checkout management device but maintains a communication link with the checkout management device. The control module is configured to: Continuously detect checkout interruption events; driving a plurality of sensors to acquire a plurality of sensing information and executing a plurality of independent models to recognize the plurality of sensing information and generate a plurality of recognition results accordingly, and determining whether the checkout interruption event occurs accordingly; When the checkout interruption event occurs: disabling a first display device providing a checkout operation, and forwarding the plurality of sensing information and the plurality of recognition results to a remote service station; as well as A transmission link is established with the remote service station so that the user can communicate with the remote service station in real time through the second display device.

11. The self-checkout kit according to claim 10, further comprising one of the following: a first image sensor configured to acquire first image sensing information including a hand of the user; a second image sensor configured to acquire second image sensing information including the product; as well as The first image sensor is configured to acquire the first image sensing information including an information code; a high-position image sensor configured to obtain third image sensing information including one of the user, the hand, the product, and the information code; a first proximity sensor configured to detect first distance information about the hand, the commodity, or the information code, and obtain first sensing information; a second proximity sensor configured to detect second distance information about the hand, the commodity, or the information code, and obtain second sensing information; a third proximity sensor configured to detect third distance information about the hand, the commodity, or the information code, and obtain third sensing information; an information code sensor configured to read the information code and obtain fourth sensing information; as well as The control module is configured to: executing a gesture recognition independent model to recognize the first image and generate a first recognition result; executing an independent information code recognition model to recognize the first image and generate a second recognition result; executing an independent object recognition model to recognize the second image sensing information and generate a third recognition result; Executing and inputting the first recognition result to the third recognition result, the first image sensing information to the third image sensing information, and the first sensing information to the fourth sensing information into an independent checkout auxiliary decision model to thereby identify the user's consumption behavior and determine whether a missed payment event occurs; as well as When the missed brushing event occurs: It is determined whether the checkout interruption event occurs.

12. The self-service checkout kit according to claim 10, wherein the checkout management device is a point of sale machine or a cash register.

Images