Self-checkout system with readiness awareness

Abstract

We provide a self-checkout system that can detect and address shock events. [Solution] The self-checkout system includes a housing having a base support, a tower extending vertically upward from the base support, and a back panel extending vertically upward relative to the tower. Furthermore, the self-checkout system includes a display screen mounted on the back panel, input shelves and output shelves mounted on the tower, respectively, and a computing system configured to receive data indicating that an impact force has been applied to the self-checkout system, to determine, at least partially, whether the self-checkout system has experienced an impact event, and to perform control actions in response to the determination that the self-checkout system has experienced an impact event.

Description

Background Art

[0001]

[0001] Many retail stores offer purchasers the option to purchase items at a self-service kiosk. Self-service kiosks have become desirable for both purchasers and retailers. For purchasers, the kiosk reduces wait times compared to using a checkout lane. Retailers can benefit from improved checkout efficiency. During a checkout transaction, a purchaser can scan the product barcode of each product and place those products on a platform to be weighed and / or monitored during that transaction. A display screen can provide useful information to the purchaser, such as the cost of the scanned item, whether the item is on sale or discounted, the weight of the item, etc. In some cases, the kiosk may be affected by events that may prevent the kiosk from operating optimally.

Brief Description of the Drawings

[0002] [Figure 1]

[0002] Perspective view of a self-checkout system according to one or more aspects of the present disclosure. [Figure 2]

[0003] Enlarged perspective view of the self-checkout system of FIG. 1. [Figure 3]

[0004] Enlarged cross-sectional view of the self-checkout system of FIG. 1. [Figure 4]

[0005] Block diagram of the self-checkout system of FIG. 1. [Figure 5]

[0006] Flow diagram for implementing system preparation operations for detecting and handling impact events for a self-checkout system according to one or more aspects of the present disclosure. [Figure 6]

[0007] A flowchart for implementing a system readiness check operation for detecting, classifying, and addressing predefined events associated with a self-checkout system, according to one or more aspects of the present disclosure. [Modes for carrying out the invention]

[0003]

[0008] Self-checkout systems may experience impact events that could prevent them from functioning optimally. For example, a full shopping cart in a grocery store might collide with the tower or one of the shelves of a self-checkout system. Such impact events can lead to one or more of the product scanners or load cells of the scale losing accuracy, which can affect the weight measurements of scanned items. Furthermore, the field of view of one or more cameras in the self-checkout system may shift out of range, resulting in items not being scanned properly or creating blind spots, which can affect the self-checkout system's ability to perform damage prevention techniques.

[0004]

[0009] Self-checkout systems disclosed herein may include features for detecting and responding to such impact events. In one or more examples, the self-checkout system may have a housing including a base support, a tower extending vertically upward from the base support, and a back panel extending vertically upward from the tower. A display screen may be mounted on the back panel. Input and output shelves may each be mounted on the tower. In this regard, the self-checkout system may have a tower-based architecture without cabinets that could be susceptible to impact blows from shopping carts and other goods. The self-checkout system may include a computing system configured to perform a system preparation scheme for detecting and responding to impact events. In one example, the computing system is configured to receive data indicating that an impact force has been applied to the self-checkout system. For example, a shopping cart may hit the tower or one of the shelves, and at least the magnitude of this impact force can be captured. The computing system may be further configured to determine, at least in part, whether the self-checkout system has experienced an impact event. For example, the impact force may be compared to a force threshold to determine whether the impact force rises to the level of an impact event. The computing system may also be configured to take control actions in response to a determination that the self-checkout system has experienced a shock event. Illustrative control actions include, but are not limited to, shutting down lanes using lane blockers, communicating the event to an operator (e.g., a salesperson), changing the self-checkout system's mode from normal mode to standby mode or shutdown mode, and automatically performing a recalibration process for one or more load cells. Thus, such a self-checkout system can detect and address shock events.

[0005]

[0010] Furthermore, a self-checkout system may experience a variety of predefined events, such as spills on or near the self-checkout system, "lost and found" events where a user forgets an item in the self-checkout system, and damaged component events. The self-checkout systems disclosed herein may include functions for detecting, classifying, and addressing such predefined events. In one or more examples, the self-checkout system may include a computer vision system configured to compare one or more captured images with one or more baseline images, taking into account environmental conditions that may affect the captured images. In this regard, the accuracy of detecting and classifying predefined events can be improved.

[0006]

[0011] In a further embodiment, the self-checkout system disclosed herein may include a printer at least partially embedded within the counter of the self-checkout system. The printer engine of the printer may be embedded within the counter, while the cover may be located on top of the counter. Embedding the printer within the counter can reduce visual noise and reduce the possibility of the printer being bumped by goods, for example, being moved from the “purchase zone” to the bagging area. Furthermore, embedding at least the printer engine below the counter can reduce noise emission, while the cover remains easily accessible, for example, allowing paper to be added and / or the printer to be maintained without removing the printer.

[0007]

[0012] Referring here to the drawings, Figure 1 shows a perspective view of a self-checkout system 100 according to one or more embodiments of the present disclosure. The self-checkout system 100 may also be referred to as a self-service kiosk or checkout terminal. For reference, the self-checkout system 100 defines the X, Y, and Z directions, which are perpendicular to each other. In one or more examples, the X direction is longitudinal, the Y direction is transverse, and the Z direction is vertical.

[0008]

[0013] The self-checkout system 100 has a front side 102 and a rear side 104, a first side 106 and a second side 108, and an upper side 110 and a lower side 112. The self-checkout system 100 includes a housing 114 having a base support 116, a tower 118, and a back panel 120. The tower 118 is mounted on the base support 116 and extends vertically upward from the base support 116, for example, upward along the Z direction. The base support 116 may extend around the tower 118 or surround the tower 118 and be attached, for example, to the floor. The base support 116 is generally flat and extends, for example, in a plane perpendicular to the Z direction, or rather in the XY plane.

[0009]

[0014] The back panel 120 is coupled to the tower 118 and extends vertically upward, for example, along the Z direction. In one or more examples, the back panel 120 may be directly attached to the tower 118. In other examples, the back panel 120 may be indirectly attached to the tower 118, with one or more components, such as a counter 122 attached to the tower 118 and a platform 124 for a product scanner 126 attached to the counter 122, positioned between the back panel 120 and the tower 118. The back panel 120 has a front wall and a rear wall. The front wall has a vertically oriented plane and a curved bottom surface that transitions the front wall between a vertical and a horizontal orientation. The vertically oriented plane and the curved bottom surface can be continuous. In at least one example, the back panel 120 may be formed integrally with the tower 118 as a single monolithic component. In one or more other examples, the back panel 120 and the tower 118 may be separate components coupled together. The display screen 128 is mounted on the back panel 120. In at least one example, the display screen 128 is cantilevered from a vertically oriented plane of the front wall via a mounting bracket. In one or more examples, at least a portion of the display screen 128 is positioned vertically above the back panel 120. The curved bottom surface of the front wall can be curved directly below the display screen 128. In this way, the center of gravity of the self-checkout system 100 can be positioned more centrally. The display screen 128 can also be positioned vertically above the tower 118. In at least some examples, the front wall of the tower 118 may be positioned in front of the rear of the display screen 128, for example, along the X direction.

[0010]

[0015] The payment terminal 130 may be mounted on the side wall of the back panel 120. The payment terminal 130 may include a display, keypad, card reader, near-field communication (NFC) beacon, etc., to facilitate payment processing during a transaction. In addition, a lane status pole 132 may extend from the top wall of the back panel 120. The lane status pole 132 has a lane light 134 mounted on it. The lane light 134 may be controlled to indicate the status of the self-checkout system 100, for example, green if ready / open, red if closed, yellow if in use, etc. A camera 136 is mounted on the distal end of the lane status pole 132. In at least one example, the camera 136 may be pivotably coupled to the lane status pole 132 by a hinge, for example, as shown in Figure 1. The camera 136 can capture images of the self-checkout system 100, the user, etc.

[0011]

[0016] On the first side 106 of the self-checkout system 100, an input shelf 138 is attached to the tower 118. The input shelf 138 may be attached to the left front corner of the tower 118. The input shelf 138 can provide a place for the user to place items before registering or scanning them for purchase. On the second side 108 of the self-checkout system 100, an output shelf 140 is attached to the tower 118. The output shelf 140 may be attached to the right front corner of the tower 118. The output shelf 140 can provide a place for the user to bag items after purchase. The output shelf 140 has a horizontal section 142 that curves or transitions into a vertical section 144. In this respect, the output shelf 140 has a “waterfall” configuration. The vertical section 144 may be placed on the ground, while the horizontal section 142 may be attached to the right front corner of the tower 118. A bagging rack 146 may be attached to the horizontal section 142. The bagging rack 146 can hold bags that can be used to bag purchased goods. In at least one example, as shown in the example in Figure 1, the input shelf 138 and the output shelf 140 are mounted at opposing corners of the tower 118.

[0012]

[0017] As further shown in Figure 1, the counter 122 is mounted on the tower 118, for example, on the front side of the tower. A product scanner 126 is positioned on the counter 122. The product scanner 126 is communicatively coupled to one or more processors and, in conjunction with one or more processors, can visually identify products during scanning. For example, the product scanner 126 can detect encoded portions (e.g., Universal Product Code (UPC) or Quick Response (QR) codes) and / or identify the product type of the product by comparing an image of the product with a reference image. In at least some examples, the product scanner 126 may include one or more load cells 148 positioned with the platform 124 to measure the weight of the products. The product scanner 126 may include cameras 150, 152 that can be used to capture images. In at least some examples, such as the example in Figure 1 and the example shown in the enlarged cross-sectional view of Figure 3, the counter 122 may be mounted on the upper side of the tower 118, and the platform 124 of the product scanner 126 may be mounted on the upper side of the counter 122. Next, the back panel 120 may be mounted on the platform 124 of the product scanner 126. In other examples, the product scanner 126 may be omitted so that the back panel 120 is mounted on the counter 122. In further examples, the back panel 120 may be mounted directly on the top of the tower 118.

[0013]

[0018] The counter support 154 is positioned beneath the counter 122 to provide support to the counter. The counter support 154 is also attached to the tower 118. The vertical support member 156 is positioned to connect the input shelf 138 and the counter support 154. The vertical support member 156 extends between the top surface of the input shelf 138 and the bottom surface of the counter 122, connecting the top surface of the input shelf 138 to the bottom surface of the counter 122. The vertical support member 156 can provide support to the input shelf 138.

[0014]

[0019] Referring here to Figures 1, 2, and 3, the self-checkout system 100 includes a printer 158. The printer 158 has a printer housing 160, a printer engine 162 disposed within the printer housing 160, and a cover 164 that provides selective access to the printer engine 162 within the printer housing 160. The printer housing 160 and the printer engine 162 are at least partially embedded within the counter support 154 and the counter 122. Furthermore, the cover 164 is positioned above the upper surface of the counter 122. Embedding the printer 158 within the counter 122 reduces visual noise and the possibility of the printer 158 being bumped into goods, for example, by being moved from the “purchase zone” near the product scanner 126 to the bagging area. Furthermore, by embedding at least the printer engine 162 beneath the counter 122, noise emission can be reduced, while the cover 164 remains easily accessible. For example, this allows for paper to be added and / or maintenance of the printer 158 without removing it from the counter 122.

[0015]

[0020] In one or more examples, camera 166 may be positioned above the product scanner 126 and may be mounted on the front wall of the tower 118. Camera 166 may be positioned to capture images of products placed within the “purchase zone” or within the general area of ​​the product scanner 126. Camera 166 may also be used to capture images of users present in the self-checkout system 100, for example, to capture biometric data.

[0016]

[0021] Figure 4 is a schematic diagram of the self-checkout system 100. As illustrated in Figure 4, the self-checkout system 100 may include a computing system 168. The computing system 168 may include one or more computing devices, such as computing device 170. Computing device 170 may include one or more processors 171 and one or more memory devices 172 that store one or more programs 173, and when one or more programs 173 are executed by any combination of one or more processors 171, they cause one or more processors 171 to perform operations such as implementing system preparation operations or system preparation checks. One or more memory devices 172 may also store data 174. The data 174 may, among other things, include a library 175. The library 175 can associate baseline images 176 with a predefined set 177 of environmental conditions (i.e., the EC set 177 in Figure 4).

[0017]

[0022] In some examples, the library 175 may be stored locally on the computing device 170, for example, on one or more non-temporary memory devices 172 of the computing device 170. In other embodiments, the library 175 may be stored outside the self-checkout system 100, for example, on a data store 180 as shown in Figure 4. The computing device 170 can access the library 175 via a network 182, such as the Internet. The computing device 170 may include a communication interface 178 that enables communication with devices via the network 182 and also local communication with other devices of the self-checkout system 100 via a communication bus 179. The communication interface 178 may include a transmitter circuit configured to send communication signals and a receiver circuit configured to receive communication signals. In this regard, the communication interface 178 may include transmitters, receivers, transceivers, etc., for communicating via the network 182 and / or the communication bus 179. In yet another embodiment, the library 175 may be stored partially locally and partially remotely.

[0018]

[0023] The computing device 170 is communicatively coupled to other devices / components of the self-checkout system 100 via a communication bus 179, for example, by one or more wired and / or wireless communication links. As shown in Figure 4, the computing device 170 may be communicatively coupled to the product scanner 126, display screen 128, payment terminal 130, lane lights 134, cameras 136, 150, 152, 166, load cell 148, printer 158, accelerometer 184 (which may be embedded within the display screen 128), lane blocker 186, and environmental conditions generator 188 (i.e., ECG 188 in Figure 4). The computing device 170 can be located in any suitable position, such as behind the display screen 128. The computing device 170 may also be communicatively coupled to other devices, such as one or more speakers, user input devices, other light sources, and off-board devices (e.g., off-board cameras, computing devices, sensors, etc.).

[0019] System Preparation - Shock Event

[0024] In one or more examples, the self-checkout system 100 may be configured to implement a system preparation scheme in response to an impact force, such as when a shopping cart collides with the tower 118 or one of the shelves of the self-checkout system 100. In some cases, an impact force applied to the self-checkout system 100 may lead to the self-checkout system 100 experiencing an impact event, which may prevent the self-checkout system 100 from performing optimally. For example, an impact event may lead to one or more of the load cells 148 of the product scanner 126 losing accuracy, which may affect the weight measurement of scanned products. Furthermore, the field of view of one or more cameras may move out of range, resulting in products not being scanned properly or creating blind spots, which may affect the self-checkout system 100's ability to perform damage prevention techniques. The computing system 168 of the self-checkout system 100 may be used to implement a system preparation scheme. The computing system 168 may include one or more processors and one or more memory devices for storing a program, which, when executed, causes one or more processors to perform operations, for example, system preparation operations for detecting and dealing with a shock event, individually or collectively.

[0020]

[0025] Referring here to Figures 1, 2, 3, 4, and 5, the system preparation actions for detecting and responding to impact events may be performed by the self-checkout system 100 according to process 200 shown in the flowchart of Figure 5.

[0021]

[0026] In 202, when operating, one or more processors can receive data indicating that an impact force has been applied to the self-checkout system 100. As described above, the impact force on the self-checkout system 100 can be caused by various methods, such as a shopping cart hitting tower 118 or one of the shelves.

[0022]

[0027] In one or more examples, the display screen 128 has an accelerometer 184 so that, for example, it can determine whether the display screen 128 is in portrait mode or landscape mode, and accordingly an image can be displayed on the display screen 128. In such an example, the accelerometer 184 can also be used to detect the impact force applied to the self-checkout system 100, including the magnitude and direction of the impact force. Therefore, data indicating that an impact force has been applied to the self-checkout system 100 can be received from the accelerometer 184.

[0023]

[0028] In one or more other examples, data indicating that an impact force has been applied to the self-checkout system 100 can include one or more current images captured by one or more cameras of the self-checkout system 100, such as camera 136, cameras 150, 152, camera 166, or a combination thereof. For example, when the accelerometer 184 indicates that an impact force has been applied to the self-checkout system 100, when the field of view of one or more cameras goes out of range with respect to the baseline field of view, or based on some other condition, one or more cameras can capture images at a predetermined interval. Therefore, data indicating that an impact force has been applied to the self-checkout system 100 can be received from one or more cameras of the self-checkout system 100, or potentially even from an external camera, such as a camera in the retail store where the self-checkout system 100 is located.

[0024]

[0029] In 204, when performing an operation, one or more processors can determine, at least in part based on data, whether the self-checkout system has experienced an impact event.

[0025]

[0030] In one or more examples, data indicating that an impact force has been applied to the self-checkout system can include current images 190 captured by one or more cameras of the self-checkout system 100. In such examples, when determining, at least in part based on data, whether the self-checkout system 100 has experienced an impact event, one or more processors can compare one or more current images 190 with one or more baseline images 176. One or more processors can determine that the self-checkout system 100 has experienced an impact event if the current images 190 captured by one or more cameras do not match the baseline images 176 (e.g., do not match according to a predetermined percentage or do not substantially match).

[0026]

[0031] In one or more examples, when determining, at least in part based on data, whether the self-checkout system 100 has experienced an impact event, one or more processors can determine whether the current field of view of a camera 136 attached to a lane status pole 132 is outside the range of the baseline field of view of the camera 136. One or more processors can determine that the self-checkout system has experienced an impact event if the current field of view of the camera 136 is outside the range of the baseline field of view.

[0027]

[0032] In one or more examples, when determining, at least partially, whether the self-checkout system 100 has experienced an impact event, one or more processors can determine whether the impact force has reached an impact threshold. If the impact force reaches an impact threshold, for example, if the impact force reaches or exceeds the impact threshold, one or more processors can determine that the self-checkout system 100 has experienced an impact event. The impact threshold may be set to a predetermined force level that is greater than the force experienced by the self-checkout system 100 under normal use, for example.

[0028]

[0033] In one or more examples, after determining that the self-checkout system 100 has experienced an impact event when the impact force reaches an impact threshold, one or more processors can perform a visual confirmation process by comparing one or more current images 190 captured by one or more cameras of the self-checkout system 100 with one or more baseline images 176. The baseline images 176 can represent the self-checkout system 100 in a ready state, or rather, the self-checkout system 100 in a state where it is ready for use in normal mode without any damage to itself and the cameras and sensors are calibrated to produce accurate measurements. Furthermore, one or more processors can detect whether there is an anomaly in the self-checkout system 100, at least in part, based on the comparison of the current images 190 captured by one or more cameras with the baseline images 176. For example, if the impact event has damaged or otherwise damaged the input shelf 138, this anomaly can be detected by one or more processors by comparing the captured current images 190 with the baseline images 176, for example, by running a machine learning model (e.g., a convolutional neural network (CNN)). One or more processors can confirm an impact event when the self-checkout system 100 detects an anomaly. In at least one example, one or more processors can take a control action in response to confirmation of an impact event, but not before.

[0029]

[0034] In 206, when performing an operation, one or more processors may take control actions in response to a determination that the self-checkout system 100 has experienced an impact event.

[0030]

[0035] In one or more examples, the control action may include physically blocking access to the self-checkout system 100. In at least one example, the control action may include moving the lane blocker 186 to physically close the self-checkout system 100. The lane blocker 186 may be coupled, for example, to an input shelf 138 mounted on a tower 118. In at least one example, the lane blocker 186 may include a pole or bar coupled to an actuator (e.g., an electrically controlled actuator). The actuator may be controlled to selectively move the pole or bar, for example, outward along the X direction from the input shelf 138 and, for example, inward towards the input shelf 138 along the X direction. In this regard, the lane blocker 186, or its pole or bar, may be movable between a retracted position and an extended position in which the lane blocker 186 physically closes the self-checkout system 100. In the retracted position, the lane blocker 186 may allow physical access to the self-checkout system 100 and may be retracted to be hidden from view. For example, the pole or bar may be recessed into a recess defined by the input shelf 138, or it may be positioned below the input shelf 138, as shown in Figure 1, for example.

[0031]

[0036] In the event of an impact, one or more components of the self-checkout system 100 may be damaged, cracked, or otherwise affected. Therefore, the lane blocker 186 may be selectively deployed to prevent or enforce that users do not move near the self-checkout system 100. In other examples, the lane blocker 186 may be located at a distance from the tower 118 and shelves, for example, at the entrance to a lane or waiting area associated with the self-checkout system 100. In yet another example, in addition to or instead of a lane blocker coupled to an input shelf 138, the self-checkout system 100 may include a lane blocker coupled to an output shelf 140.

[0032]

[0037] In some cases, the lane blocker 186 may be deployed, for example, when it is confirmed that the user is not in the self-checkout system 100, so as not to hit the user with the lane blocker 186. In at least one example, the computing system 168 may "hold" the movement of the lane blocker 186 until the user has left the area. In at least one example, the speaker of the self-checkout system 100 may generate a sound to warn the user that the lane blocker 186 is being deployed or is about to be deployed, thereby enhancing safety. In at least one example, the lane blocker 186 may be deployed based on the magnitude of the impact force. For example, the lane blocker 186 may be deployed when it is determined that the impact force has reached an extreme impact threshold, the extreme impact threshold being a force level greater than the impact threshold that may be used to determine whether an impact event has occurred. The extreme impact threshold may correspond to a force level at which there is a high probability (e.g., a probability greater than 50%) that some components of the self-checkout system 100 are likely to be damaged or otherwise affected.

[0033]

[0038] In one or more examples, in addition to or instead of any of the control actions described above or below, a control action may include switching the self-checkout system 100 from normal mode to some other mode, such as shutdown mode or standby mode. In at least one example, performing a control action may include changing the self-checkout system from normal mode to standby mode, in which the functionality of the self-checkout system is reduced compared to normal mode, but it is still operational. For example, in standby mode, the self-checkout system 100 may allow the user to continue scanning items for purchase, but may withhold allowing the user to complete the payment in order to keep the user in the self-checkout system 100, for example, so that an operator can inspect the self-checkout system 100 before the user leaves. In at least one example, performing a control action may include changing the self-checkout system from normal mode to shutdown mode, in which the functionality of the self-checkout system is turned off.

[0034]

[0039] In at least one example, the self-checkout system 100 can be switched from normal mode to either standby mode or shutdown mode based on the magnitude of the impact force. For example, if the impact force is in a first range, the self-checkout system 100 can be switched from normal mode to standby mode, and if the impact force is in a second range, the self-checkout system 100 can be switched from normal mode to shutdown mode. The first range may be associated with a lower force value than the second range.

[0035]

[0040] In one or more examples, in addition to or instead of any of the control actions described above or below, the control action may include automatically initiating load cell recalibration for the load cells of the product scanner 126 of the self-checkout system 100. As described above, if the self-checkout system 100 is exposed to an impact event, the load cells of the product scanner 126 may lose accuracy, which may affect how the load cells measure the weight of the goods. Therefore, recalibration of one or more load cells of the product scanner 126 may be performed. In at least one example, during the recalibration process, the self-checkout system 100 may switch to an operating mode that prevents the user from scanning goods whose price is determined by weight. In such an example, if the recalibration process is successfully completed, the self-checkout system 100 may switch to an operating mode (e.g., normal mode) that allows the user to scan goods whose price is determined by weight.

[0036]

[0041] In one or more examples, in addition to or instead of any of the control actions described above, the control action may include automatically communicating to the operator that the self-checkout system has experienced an impact event. The communication provided to the operator may be an audible communication (e.g., output by a speaker of the self-checkout system 100), an optical signal (e.g., the lane lights 134 may flash in a specific color and / or at a predetermined frequency), a digital signal (e.g., transmitted to the operator's system), or a combination thereof.

[0037] System readiness check

[0042] In some cases, the self-checkout system 100 may experience, but is not limited to, predefined events such as spills onto the floor or onto the self-checkout system 100, damage to one or more components of the self-checkout system 100, a user forgetting one or more items (e.g., purchased goods, keys, telephone, etc.), or drift of the field of view of one or more cameras from alignment. In one or more examples, the self-checkout system 100 may be configured to implement system readiness checks to detect and address predefined events. In at least one example, the self-checkout system 100 may implement system readiness checks periodically according to a time interval, for example, every 10 minutes. In at least one other example, the self-checkout system 100 may perform system readiness checks based on the occurrence of one or more conditions, for example, after each user transaction. The computing system 168 of the self-checkout system 100 may be used to implement system readiness checks. The computing system 168 of the self-checkout system 100 may include one or more processors and one or more memory devices for storing a program, which, when executed, causes one or more processors to perform an operation, such as a system readiness check operation, individually or collectively.

[0038]

[0043] Referring here to Figures 1, 2, 3, 4, and 6, the system preparation actions for detecting and responding to impact events can be implemented by the self-checkout system 100 according to process 300 shown in the flowchart of Figure 6.

[0039]

[0044] In one or more examples, a baseline image 176 of the self-checkout system 100 may be captured for each of the predefined sets of environmental conditions 177. For example, as shown in Figure 4, a first baseline image B1 may be captured for a first set of environmental conditions EC1, a second baseline image B2 may be captured for a second set of environmental conditions EC2, a third baseline image B3 may be captured for a third set of environmental conditions EC3, and so on. The baseline image 176 can represent the self-checkout system 100 in a ready state, or rather, the self-checkout system 100 in a state where it is ready for use in normal mode without any damage, spillage, etc. affecting the self-checkout system 100 for a given predefined set of environmental conditions. Furthermore, each of the baseline images 176 may show at least one of the housing 114, display screen 128, input shelf 138 and output shelf 140, and / or counter 122. Baseline images 176 can be captured by one or more cameras 136, 150, 152, 166 within their respective predefined sets of environmental conditions 177. Thus, a library 175 can be constructed and stored in one or more memory devices, and the library 175 associates the baseline images 176 with the predefined sets of environmental conditions 177.

[0040]

[0045] In one or more examples, multiple baseline images 176 may be associated with a given predefined set 177 of environmental conditions. For example, multiple baseline images 176 associated with a given predefined set 177 of environmental conditions may include images of the housing 114, the display screen 128, the input shelf 138 and output shelf 140, and the counter 122. In at least one example, multiple images associated with a given predefined set of environmental conditions may be stitched together according to a predefined format.

[0041]

[0046] In at least one example, each of the predefined sets of environmental conditions 177 can define the environmental settings associated with the self-checkout system 100 at a given time and location. The environmental conditions in the predefined sets may include, but are not limited to, time of day, season, outdoor weather conditions (e.g., sunny, cloudy, precipitation, etc.), indoor lumen levels, or several combinations thereof. One or more of these environmental conditions may affect how one or more cameras 136, 150, 152, 166 capture images of the self-checkout system 100 and its surroundings. The self-checkout system 100 can thus advantageously account for these effects by capturing multiple baseline images for each of the predefined sets of environmental conditions. By comparing the baseline image 176 with the current image 190 captured under the same or similar environmental conditions, the accuracy of detecting predefined events can be improved, as will be fully explained below.

[0042]

[0047] In 302, when performing a system readiness check operation, one or more processors may receive a current set of environmental conditions. In at least one example, the current set of environmental conditions may include at least one of the following: time of day, season, outdoor weather conditions, or indoor lumen level. As described above, one or more of these environmental conditions may affect image acquisition of the self-checkout system 100 and its surroundings. The current set of environmental conditions may be generated or provided by an environmental condition generator 188, which may receive information about the time of day, season, outdoor weather conditions (e.g., sunny, cloudy, rainy, snowy, etc.) within the building where the self-checkout system 100 is located, and lumen levels or brightness provided by indoor lighting. The environmental condition generator 188 may include an internal clock, an interface for connecting to a weather forecaster, and / or one or more sensors for detecting lumen intensity, and / or an interface for connecting to a system configured to provide the current lumen level.

[0043]

[0048] In 304, when performing a system readiness check operation, one or more processors may select a baseline image from a plurality of baseline images 176. For example, a baseline image may be selected from a library 175. The selected baseline image may be associated with a predefined set of environmental conditions that matches or best matches the current set of environmental conditions. Thus, the current set of environmental conditions can be compared with a predefined set of environmental conditions in the library 175, and if the best match or a match is determined, a baseline image associated with a predefined set of environmental conditions may be selected. As described above, a plurality of predefined sets of environmental conditions are associated with each of the plurality of baseline images. The selected baseline image may represent at least one of the housing 114, display screen 128, input shelf 138 and output shelf 140, or counter 122. The baseline image may be captured by one or more cameras 136, 150, 152, 166 during a preceding time in a predefined set of environmental conditions.

[0044]

[0049] In 306, when performing a system readiness check operation, one or more processors may receive a current image 190 of the self-checkout system 100. The current image 190 may show at least one of the following: the housing 114, the display screen 128, the input shelf 138 and the output shelf 140, or the counter 122. The current image 190 may be captured by one or more cameras 136, 150, 152, 166 during the current set of environmental conditions at the current time.

[0045]

[0050] In 308, when performing a system readiness check operation, one or more processors can determine whether the self-checkout system has experienced a predefined event by comparing the current image 190 of the self-checkout system with a selected baseline image 176. For example, the current image 190 may be a stitched image formed from multiple images, and the stitched image may show the housing 114, display screen 128, input shelf 138 and output shelf 140, and counter 122 according to a predefined format. Similarly, the selected baseline image 176 may be a stitched image formed from multiple images, and the stitched baseline image may show the housing 114, display screen 128, input shelf 138 and output shelf 140, and counter 122 according to a predefined format.

[0046]

[0051] When performing image comparison, which can be implemented by running one or more computer vision algorithms or machine learning models (e.g., CNNs), one or more anomalies can be detected by one or more processors. As a first example, the current image 190 may show that milk or another liquid has spilled onto the counter 122, and the baseline image 176 does not show this spill. Therefore, the liquid spill on the counter 122 can be identified as an anomaly, and based on one or more image recognition techniques, the spill can be classified as a counter spill event, which can be one of a predefined set of events. As a second example, the current image 190 may show that a user has left the self-checkout system 100 after completing a transaction, but has left their key on the output shelf 140. The baseline image 176 does not show the key on the output shelf 140, and therefore, the forgotten key on the output shelf 140 can be identified as an anomaly, and based on one or more image recognition techniques, the forgotten key on the output shelf 140 can be classified as a "lost and found" event, which can be one of a predefined set of events. One or more processors may initiate control actions upon detecting an anomaly and / or classifying a predefined event.

[0047]

[0052] In 310, when performing a system readiness check operation, one or more processors may take control actions in response to a determination that the self-checkout system 100 has experienced a predefined event.

[0048]

[0053] Continuing with the first example described above, if the self-checkout system 100 is experiencing a counter spill event, control actions may be taken to address the spill. For example, if the spill is relatively large (e.g., liquid has spread across the entire product scanner 126 and is dripping onto the floor), the control action may include, for example, moving a lane blocker to physically close the self-checkout system 100 after the current user has finished their transaction and left the self-checkout system 100. This can prevent subsequent users from using the self-checkout system 100 until, for example, the spill has been addressed and thereby enhances user safety. As another example, if the spill is relatively small (e.g., a small amount of liquid is present on the counter 122), the control action may include automatically communicating to an operator (e.g., a salesperson) that the self-checkout system 100 has experienced a spill event and that assistance is needed. The control action may also include automatically communicating to an operator that the self-checkout system 100 has experienced a spill event and, if the spill is relatively large, that assistance is needed.

[0049]

[0054] Continuing with the second example described above, when the self-checkout system 100 experiences a “lost item” event, control actions may be taken to address the lost item. For example, the control action may include the self-checkout system 100 automatically communicating to an operator (e.g., a salesperson) that it has experienced a “lost item” event and that assistance is needed. This allows the operator to retrieve the lost item. In at least one example, the communication may include a captured image of the user who left the item on the output shelf 140. The captured image may be taken by one or more cameras 136, 150, 152, 166 of the self-checkout system 100. In this way, the operator can better match the user with the lost item. Furthermore, in at least one example, the control action may include communicating directly with the user who left the item, for example, by text, phone, email, or a combination thereof. The self-checkout system 100 can recognize the user, for example, by captured biometric data or payment information, and can use a recorded phone number or email address to communicate with the user. This communication can occur in real time or near real time, allowing the user to receive the message and return immediately if they forget something.

[0050]

[0055] In one or more other examples, the control action may include changing the self-checkout system from normal mode to a standby mode in which the functionality of the self-checkout system is reduced compared to normal mode (for example, if minor damage to the self-checkout system 100 is detected), or to a shutdown mode in which the functionality of the self-checkout system is stopped (for example, if significant damage to the self-checkout system 100 is detected).

[0051]

[0056] In one or more other examples, the control action may include automatically recalibrating the load cell 148 of the product scanner, for example, if a predefined event is a damage event associated with the counter 122 or the product scanner 126 located above the counter 122. In one or more further examples, any combination of the control actions described above may be performed in response to the detection and classification of predefined events.

[0052]

[0057] The descriptions of various embodiments are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best describe the principles of the embodiments, their practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0053]

[0058] References to embodiments presented in this disclosure are made below. However, the scope of this disclosure is not limited to the embodiments described. Rather, any combination of the following features and elements is intended to implement and practice the imagined embodiments, whether related to different embodiments or not. Furthermore, the embodiments disclosed herein may achieve advantages over other possible solutions or the prior art, but whether or not an advantage is achieved by a given embodiment does not limit the scope of this disclosure. Accordingly, the following aspects, features, embodiments and advantages are merely illustrative and should not be considered as elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to “this disclosure” should not be interpreted as a generalization of any inventive subject matter disclosed herein and should not be considered as elements or limitations of the appended claims unless expressly stated in the claims.

[0054]

[0059] The embodiments described may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware embodiments, which are generally referred to herein as “circuits,” “modules,” or “systems.”

[0055]

[0060] One or more of the embodiments described may be a system, method, and / or a computer program product. A computer program product may include one (or more) computer-readable storage media having computer-readable program instructions for causing a processor to execute an aspect of the embodiment.

[0056]

[0061] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of those described above. A non-exhaustive list of examples of computer-readable storage mediums includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves on which instructions are recorded, and any suitable combination of those described above. Computer-readable storage media as used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmitting media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0057]

[0062] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmitters, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers those computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0058]

[0063] The computer-readable program instructions for performing the operations of the described embodiments may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or remote server. In the latter scenario, the remote computer may connect to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may connect to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of computer-readable program instructions for personalizing the electronic circuit in order to perform the embodiments described.

[0059]

[0064] The aspects of the embodiments described herein are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0060]

[0065] These computer-readable program instructions are provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, so as to generate a machine such that instructions executed via the processor of the computer or other programmable data processing device create means for implementing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct computers, programmable data processing devices, and / or other devices to function in the described manner, thereby comprising a product in which the computer-readable storage medium storing the instructions includes instructions that implement modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0061]

[0066] Computer-readable program instructions are loaded into a computer, other programmable data processing device, or other device, and a series of operational steps are performed on the computer, other programmable device, or other device to generate a computer implementation process, thereby enabling the instructions executed on the computer, other programmable device, or other device to implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0062]

[0067] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction containing one or more executable instructions for implementing a specified(multiple) logical function. In some alternative implementations, the functions described in a block may occur in a different order than that shown in the figure. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in a block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0063]

[0068] The foregoing is directed toward one or more embodiments, but other embodiments and further embodiments may be conceivable without departing from their basic scope, the scope of which will be determined by the following claims.

Claims

1. It is a self-checkout system, A housing comprising a base support, a tower extending vertically from the base support, and a back panel connected to the tower and extending vertically relative to the tower, The display screen attached to the back panel, A computing system having one or more processors and one or more memory devices for storing programs, Equipped with, The program is provided to the processor: Data is received indicating that an impact force has been applied to the self-checkout system. Based on the data, it is determined that the self-checkout system has experienced an impact event. In response to the determination that the self-checkout system has experienced the impact event, it performs a control action. A self-checkout system that handles the process.

2. The program is provided to the processor: When the self-checkout system determines, based on the data, that it has experienced an impact event, it determines whether the impact force has reached an impact threshold. When the impact force reaches the impact threshold, the self-checkout system determines that the impact event has occurred. A self-checkout system according to claim 1, which allows processing to be performed.

3. The self-checkout system according to claim 2, wherein the display screen has an accelerometer, and the data indicating that the impact force has been applied to the self-checkout system is received from the accelerometer.

4. The program is provided to the processor: When the impact force reaches the impact threshold, the self-checkout system determines that it has experienced the impact event, and then performs a visual confirmation process by comparing the current image captured by one or more cameras of the self-checkout system with a baseline image of the self-checkout system. Based on a comparison of the current image and the baseline image, it is detected whether there is an abnormality in the self-checkout system. If the self-checkout system has the aforementioned abnormality, the impact event is confirmed, In response to confirmation of the aforementioned impact event, the control action is performed. A self-checkout system according to claim 2, which allows processing to be performed.

5. The data indicating that an impact force has been applied to the self-checkout system includes current images captured by one or more cameras of the self-checkout system. The program is provided to the processor: When determining whether the self-checkout system has experienced an impact event based on the data, the current image captured by the camera is compared with a baseline image. If the current image does not match the baseline image, the self-checkout system determines that the impact event has occurred. A self-checkout system according to claim 1, which allows processing to be performed.

6. The program is provided to the processor: When determining, based on the data, whether the self-checkout system has experienced an impact event, it is determined whether the current field of view of the camera mounted on the lane status pole is outside the range of the camera's baseline field of view. If the current field of view of the camera is outside the range of the baseline field of view, the self-checkout system determines that the impact event has occurred. A self-checkout system according to claim 1, which allows processing to be performed.

7. The self-checkout system according to claim 1, wherein performing the control action includes moving a lane blocker to physically close the self-checkout system.

8. The self-checkout system according to claim 7, wherein the lane blocker is coupled to an input shelf attached to the tower and is movable between a retracted position and an extended position in which the lane blocker physically closes the self-checkout system.

9. The self-checkout system according to claim 1, wherein performing the control action includes a process of changing to a standby mode in which the functionality of the self-checkout system is reduced compared to the normal mode, or to a shutdown mode in which the functionality of the self-checkout system is stopped.

10. The self-checkout system according to claim 9, wherein the self-checkout system is changed to the standby mode or the shutdown mode based on the magnitude of the impact force.

11. Further equipped with a product scanner that has a load cell, The self-checkout system according to claim 1, wherein performing the control action includes a process of automatically recalibrating the load cell of the product scanner.

12. The self-checkout system according to claim 11, wherein performing the control action includes changing to an operating mode that prevents the user from scanning items whose price is determined by weight while the load cell is being recalibrated.

13. The self-checkout system according to claim 1, wherein the control action includes a process that automatically communicates to the operator that the self-checkout system has experienced the impact event.

14. The tower further comprises input shelves and output shelves attached to opposing corners of the tower. The self-checkout system according to claim 1.

15. It is a self-checkout system, A housing comprising a base support, a tower extending vertically from the base support, and a back panel connected to the tower and extending vertically relative to the tower, The display screen attached to the back panel, The input shelf and output shelf attached to the tower, A counter attached to the aforementioned tower, One or more cameras, A computing system comprising one or more processors and one or more memory devices for storing programs, The program is provided to the processor: Receive the current set of environmental conditions, A baseline image is selected from multiple baseline images, the selected baseline image is associated with a predefined set of environmental conditions that matches or best matches the current set of environmental conditions, the predefined set of environmental conditions is one of multiple predefined sets of environmental conditions, each of which is associated with each of the multiple baseline images, the baseline image shows at least one of the housing, the display screen, the input shelf and the output shelf, or the counter, and the baseline image is captured by the camera during the predefined set of environmental conditions. The camera receives a current image of the self-checkout system, the current image showing at least one of the housing, the display screen, the input shelf and the output shelf, or the counter, and the current image is captured by the camera during the current set of environmental conditions. By comparing the current image with the selected baseline image, the self-checkout system determines that a predefined event has occurred. After the self-checkout system determines that a predefined event has occurred, it performs a control action. A self-checkout system that performs this as a system preparation check operation.

16. The current set of environmental conditions and the predefined set of environmental conditions include at least one of time of day, season, outdoor weather conditions, or indoor lumen level. The self-checkout system according to claim 15.

17. Performing the aforementioned control action means The self-checkout system is changed from normal mode to a standby mode in which the functionality of the self-checkout system is reduced compared to the normal mode, or to a shutdown mode in which the functionality of the self-checkout system is stopped. To physically close the self-checkout system, move the lane blocker, or Automatically recalibrate the load cell of the product scanner. Includes at least one of the following processes: The self-checkout system according to claim 15.

18. The self-checkout system according to claim 15, wherein the system readiness check operation is performed after each user transaction occurs.

19. The tower and A counter attached to the aforementioned tower, A counter support portion is attached to the tower, positioned below the counter, and provides support to the counter, A printer comprising a housing, a printer engine disposed within the housing, and a cover that provides selective access to the printer engine within the housing. Equipped with, The housing and the printer engine are embedded in the counter support and the counter, The cover is positioned above the upper surface of the counter. Self-checkout system.

20. An input shelf attached to the corner of the aforementioned tower, A vertical support member connecting the input shelf and the counter support section. The self-checkout system according to claim 19, further comprising:

Images