Self-checkout illumination system

Abstract

The present invention provides a self-checkout system and a technology for irradiating one or more items placed within a designated zone of the self-checkout system. [Solution] In one embodiment, the self-checkout system includes one or more cameras, a light source, and a computing device. The computing device is configured to perform an operation which includes receiving one or more images of one or more articles placed in a predetermined zone, the images being captured by a camera, determining identification information of one or more articles placed in the predetermined zone based on at least a portion of the captured images, and causing the light source to project an illumination indicator onto at least one of the one or more articles placed in the predetermined zone based on the identification information of at least one article.

Description

Technical Field

[0001] Embodiments of the present invention relate to an irradiation system for self-checkout.

Background Art

[0002]

[0001] Many retail stores offer purchasers the option of purchasing goods at self-service kiosk terminals. Self-service kiosk terminals have become desirable for both purchasers and retailers. For purchasers, kiosk terminals can reduce waiting times compared to using a checkout lane. Also, retailers benefit from reduced labor costs as one staff member can monitor several self-service counters. During a checkout transaction, a purchaser can scan the product barcode of each item and place them on a platform that is weighed and / or monitored during the transaction.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The problem to be solved by the present invention is to provide a technique for irradiating one or more articles placed within a self-checkout system and a predetermined zone of the self-checkout system.

Means for Solving the Problems

[0004] The self-checkout system of the embodiment includes a computing device having one or more cameras, a light source, one or more processors, and one or more memory devices for storing programs, and causing the one or more processors to perform an operation individually or collectively, wherein the operation includes: receiving one or more images captured by one or more cameras of one or more articles placed within a predetermined zone of the self-checkout system; determining identification information of the one or more articles placed within the predetermined zone based on at least a portion of the one or more images captured by the one or more cameras; and causing the light source to project an illumination indicator onto at least one of the one or more articles placed within the predetermined zone based on the identification information of the at least one article. [Brief explanation of the drawing]

[0005] [Figure 1] Figure 1 is a perspective view of an exemplary self-checkout system according to one or more embodiments of the present disclosure. [Figure 2] Figure 2 is a perspective view of another exemplary self-checkout system according to one or more embodiments of the present disclosure. [Figure 3] Figure 3 is a schematic diagram of the self-checkout system of Figure 1 according to one or more embodiments of the present disclosure. [Figure 4] Figure 4 is a perspective view of the self-checkout system shown in Figure 1, which illuminates products on the platform. [Figure 5] Figure 5 is a perspective view of the self-checkout system shown in Figure 1, which illuminates products on the platform. [Figure 6] Figure 6 is a perspective view of the self-checkout system shown in Figure 1, which illuminates products near the platform. [Figure 7]Figure 7 is a flowchart of a method according to one or more embodiments of the present disclosure. [Modes for carrying out the invention]

[0006]

[0009] Self-service kiosk terminals require a high level of customer involvement, including scanning the product barcode of each item and carefully placing such items on the platform, which in turn allows scanned items to be weighed and / or monitored during the checkout transaction. In some cases, customers may place non-purchasable items on the platform, potentially distorting the perceived weight of items on the platform or otherwise delaying the transaction. Furthermore, customers may place items near the platform but not on it, which can also affect the perceived weight of the items and the entire checkout transaction. Leaving these issues unresolved can lead to customer frustration and may require store staff to resolve the problems. Distortion of perceived weight can also lead to scale malfunctions and / or inaccurate scale calibration, as well as problems with the barcode scanner.

[0007]

[0010] Embodiments disclosed herein relate to smart, intuitive self-checkout systems that address one or more of the above-described problems. In at least one exemplary embodiment, the self-checkout system may include the ability to illuminate one or more items within a given zone. In another example, the self-checkout system includes the ability to illuminate items on or near the platform of the self-checkout system. The self-checkout system may include a light source, such as a projector or laser, which can be controlled to illuminate one or more items placed within a given zone based on their identification information. The computer vision system of the self-checkout system may include one or more cameras and computing devices. The cameras may capture images of items on or near the platform within a given zone, and these captured images can be used by the computing devices to identify the items. For example, captured images of items can be compared to images of known items stored in an image library. In at least one example, known items may include items that are available for purchase (i.e., known goods that can be purchased in the store) or items that are not available for purchase (i.e., known items that cannot be purchased in the store). Items on or near the platform within a given zone can be identified if the captured image of the item matches an image in the image library. If the image of an item does not match any image in the image library, the item can be identified as an unknown item.

[0008]

[0011] As described above, the light source can be controlled to illuminate one or more items placed within a predetermined zone based on the identification information of the items. For example, the light source can be controlled to project an illumination indicator onto a known non-purchasable item, for example, to suggest to the shopper and / or store staff that the item be removed from the platform so as not to distort the detected weight of the item on the platform. The illumination indicator, or the light directed at the item by the light source, can outline or otherwise highlight the item for the shopper or store staff. As another example, the light source can be controlled to project an illumination indicator onto an unknown item, for example, to suggest to the shopper and / or store staff that the item be removed from the platform and / or that an unknown item should be further investigated by a store staff member. As yet another example, the light source can be controlled to project an illumination indicator onto a known purchasable item that is located near the platform but not on the platform, for example, to suggest to the shopper and / or store staff that the item be moved onto the platform so as to be able to more accurately determine the detected weight of the item and / or as a result, the item can be monitored more easily. Therefore, illuminating specific items within a designated zone can, to its advantage, help create a smoother, more interactive shopping experience for shoppers and enable store staff to identify and resolve problems more quickly. Furthermore, among other benefits, it can eliminate or otherwise reduce issues such as scale malfunctions, inaccurate scale calibration, and barcode scanner problems.

[0009]

[0012] Figure 1 shows a perspective view of an exemplary self-checkout system 100 according to one or more embodiments of the present disclosure. In one example, the self-checkout system 100 may be a self-service checkout kiosk terminal. For reference, the self-checkout system 100 defines a vertical direction V.

[0010]

[0013] A shopper may select one or more items to purchase in a retail store and independently complete the checkout transaction using a self-checkout system 100. The self-checkout system 100 may include a base 110 and a back panel 112. The back panel 112 may be positioned perpendicular to the base 110, as shown in Figure 1. During the checkout transaction, the shopper may place one or more items intended for purchase on the platform 114 of the base 110. The shopper may then remove the items from the platform 114 to complete or cancel the transaction.

[0011]

[0014] One or more components and / or combinations of components for facilitating self-checkout transactions may be included in the housing of the base 110 and / or back panel 112. For example, the base 110 may include at least one load cell or weight sensor 116 (not shown in Figure 1; see Figure 3) located beneath the platform 114. The weight sensor 116 is configured to sense the weight of each and / or all items placed on the platform 114.

[0012]

[0015] The base 110 may include any combination of one or more user input devices 118, such as a touchscreen, keypad, card reader, and / or a near-field wireless receiver. The shopper may communicate with the self-checkout system 100 using any of the user input devices 118. For example, the user input devices 118 may be configured to provide a payment method. The back panel 112 may include a display 120 for presenting information to the shopper during the checkout process. For example, the display 120 may present the shopper with an updated checkout list, checkout instructions, and / or payment instructions during the checkout transaction.

[0013]

[0016] Furthermore, the self-checkout system 100 may include one or more cameras 122 to capture each viewpoint within and / or around a predetermined zone 115 or purchase zone. For example, a camera 122 may be configured to capture items placed on or near the platform 114 and / or shoppers standing close to the self-checkout system 100. Captured viewpoints of and / or around the predetermined zone 115 may include, for example, a left-side viewpoint, a right-side viewpoint, an overhead viewpoint (e.g., tilted downward toward the platform 114), and / or a forward viewpoint (e.g., tilted toward the shopper). One or more cameras 122 may be detachable from the housing of the self-checkout system 100.

[0014]

[0017] As an example, the self-checkout system 100 in Figure 1 includes a right arm 124 and a left arm 126 extending from a back panel 112 on either side of a platform 114. Each of the left and right arms 124, 126 may include a camera for capturing, for example, left and right viewpoint images of a predetermined zone 115, or articles placed on or near the platform 114, respectively. Articles located near the platform 114 but not on the platform 114 can be captured by camera 122 when the article is within the field of view of one or more of the cameras 122. The right arm 124 has a first camera 122A positioned to capture a right-side viewpoint image, and the left arm 126 has a second camera 122B positioned to capture a left-side viewpoint image. Furthermore, the back panel 112 may include a halo 128 having a third camera 122C positioned to capture a predetermined zone 115 located on or near the platform 114, or an overhead viewpoint image of an article. Furthermore, the halo 128 may include, for example, one or more forward-facing cameras 122D oriented in a forward-facing position to capture images of shoppers standing near the self-checkout system 100. Camera 122 may be configured to capture images based on an indication that one or more items have been detected within a predetermined zone 115, or an indication that a shopper has approached within a predetermined vicinity of the self-checkout system 100, or based on a weight sensor 116 that detects items on the platform 114, or a combination thereof, or any other trigger condition.

[0015]

[0018] The self-checkout system 100 may also include a computing device 130, which can be located, for example, on the back panel 112. The computing device 130 may also be located elsewhere, such as on the base 110 beneath the platform 114. The computing device 130 can be positioned as shown in Figure 3, which will be described in more detail below.

[0016]

[0019] In at least some exemplary embodiments, the computing device 130 and the camera 122 can form part of the computer vision system of the self-checkout system 100. Images captured by the camera 122 can be still images or video frames and can be sent to the computing device 130 for processing. The computing device 130 can use the captured images to identify one or more items placed on or near the platform 114. Feedback from the weight sensor 116 can also be supplied to the computing device 130 for processing, for example, to assist in identifying one or more items placed on or near the platform 114.

[0017]

[0020] The self-checkout system 100 may also include one or more light sources 140. In at least some exemplary embodiments, the halo 128 may include or support a light source 140A, or an overhead light source. The light source 140A may be a projector, a laser, or any other light-emitting device arranged to project light, such as an illumination indicator 142, onto one or more articles 102 placed within a predetermined zone 115, or rather on or near the platform 114. In some embodiments, the light source 140A may have an illumination field larger than the area of ​​the platform 114, for example, so that articles placed near the platform 114 but not on the platform 114 can be highlighted by the light source 140A. In this regard, the predetermined zone 115 may extend beyond the platform 114. Furthermore, by placing the light source 140A inside the halo 128 or attaching it to the halo, the light source 140A can be positioned directly above the platform 114 as shown in Figure 1, thereby allowing the light source 140A to project light onto one or more of the articles 102 from an overhead position.

[0018]

[0021] The light source 140A can be arranged to project light generally downward toward the platform 114 and onto one or more articles, based on their identification information, so as to be identified by the computing device 130 of the computer vision system. In FIG. 1, for example, based on the determined identification information of one of the articles 102 placed on the platform 114, the computing device 130 controls the light source 140A to project an illumination indicator 142 onto one of the articles placed on the platform 114, which in this example is the snack bag 102A. The snack bag 102A can be identified by the computing device 130 as a known non-purchasable article (i.e., an article that is not purchasable but is known to the computing device 130), namely a snack bag. Alternatively, the snack bag 102A can be identified as an unknown article (i.e., an article that is not purchasable and is unknown to the computing device 130). As a result, the computing device 130 can control the light source 140A to project the illumination indicator 142 onto the snack bag 102A and propose removing the snack bag 102A from the platform 114. This can help provide a smoother interactive shopping experience for the shopper.

[0019]

[0022] Other articles on the platform 114 in FIG. 1 (in this example, the bottled drinking water 102B) can also be identified by the computing device 130 based on the captured image from the camera 122. The bottled drinking water 102B can be identified by the computing device 130 as a known purchasable article (i.e., an article that is purchasable and is known to the computing device 130), namely bottled drinking water. Accordingly, the light source 140A can be controlled by the computing device 130 so as not to irradiate the bottled drinking water 102B with the illumination indicator.

[0020]

[0023] In at least some other embodiments, in addition to, or instead of, the light source 140A disposed on the hero 128, the self-checkout system 100 can include one or more light sources located elsewhere. An example is shown below.

[0021]

[0024] FIG. 2 shows an embodiment of the self-checkout system 100 having a light source 140B that can be disposed under the platform 114 or embedded in the platform. The platform 114 may be at least somewhat translucent, and the light emitted by the light-emitting element of the light source 140B can pass through the platform 114 or its upper translucent layer onto the articles placed on the platform 114. The light-emitting elements of the light source 140B can be arranged in an array, such as a grid, and the light-emitting elements are selectively controlled by the computing device 130 to irradiate one or more articles 102 placed on the platform 114, for example, based on the identification information of the article 102. In FIG. 2, while the bottled water 102B is not irradiated, the snack bag 102A is shown being irradiated by the irradiation indicator 142 projected by the light source 140B.

[0022]

[0025] Thus, in some exemplary embodiments, the self-checkout system 100 of FIGS. 1 and / or 2 can be an all-in-one, smooth self-service unit that uses computer vision, and in some embodiments, uses the weight of the article sensed for article identification, and can control one or more light sources 140 to highlight specific ones of the articles 102 based on the identification information of the articles 102 placed on or near the platform 114. By using computer vision and one or more light sources 140 to irradiate specific articles, an interactive and intuitive self-checkout experience can be provided to the shopper. Further, the irradiated articles can enable the store staff to more quickly identify problems and solve problems.

[0023]

[0026] Figure 3 is a schematic diagram of the self-checkout system 100. As shown in Figure 3, the computing device 130 may include one or more processors 131 and one or more memory devices 132 that store a program 133, which, when executed by any combination of the one or more processors 131, causes one or more processors 131 to perform actions such as facilitating an interactive checkout transaction as described above, or causing illumination of a specific item (Figure 1) placed on or near the platform 114. The one or more memory devices 132 may also store data 134. The data 134 may include, among other things, an image library 135 and item data 136. The image library 135 may include images of known items from one or more viewpoints. Known items may include known purchasable items (e.g., chips, bottled water, bananas, etc. that can be purchased at a retail store) and known non-purchasable items (keys, smartphones, wallets, etc. that cannot be purchased at a retail store but are known to the computing device 130). In some cases, known non-purchasable items may be inadvertently placed on platform 114 (Figure 1), which may affect the weight calculation of items placed on platform 114. As further described herein, it may be desirable to illuminate such items and suggest to shoppers that they remove them from platform 114. Item data 136 may include data on dimensions, weight, and packaging related to known items in image library 135.

[0024]

[0027] In some embodiments, the image library 135 and item data 136 can be stored locally on the computing device 130, for example, in one or more memory devices 132. In other embodiments, the image library 135 and / or item data 136 can be stored outside the self-checkout system 100, for example, in a data store 150 as shown in Figure 3. The computing device 130 can access the image library 135 and / or item data 136 via a network 160, such as the Internet. The computing device 130 may include a communication interface 137 that enables it to communicate locally with the device via the network 160 and with other devices in the self-checkout system 100 via a communication bus 138. The communication interface 137 may include a transmitter circuit that sends communication signals and a receiver circuit that receives communication signals. In this regard, the communication interface 137 may include transmitters, receivers, transceivers, etc., for communication via the network 160 and / or the communication bus 138. In yet another embodiment, the image library 135 and / or item data 136 can be stored partially locally and partially remotely.

[0025]

[0028] The computing device 130 is communicatively coupled to other devices / components of the self-checkout system 100 via a communication bus 138, for example, by one or more wired and / or wireless links. As shown in Figure 3, the computing device 130 can be communicatively coupled to a weight sensor 116, a user input device 118, a display 120, a camera 122, and a light source 140 (or more light sources). In at least some embodiments, the computing device 130 is communicatively coupled to one or more speakers 144 of the self-checkout system 100. In at least some embodiments, the display 120 and / or speakers 144 can be controlled to provide instructions or warnings to shoppers or store staff, in addition to the light source 140 which is controlled to illuminate one or more items (Figure 1) placed on or near the platform 114.

[0026]

[0029] Referring here to Figures 1 and 3, an exemplary method is provided in which the self-checkout system 100 can illuminate specific items on or near the platform 114 during a checkout transaction. In at least one embodiment, when one of the programs 133 is executed by one or more processors 131, one or more processors 131 can be made to perform operations individually or collectively, including the smart illumination operations described below.

[0027]

[0030] One or more processors 131 can activate a camera 122 to capture images of items 102 placed on or near the platform 114, or rather, items 102 within a predetermined zone 115. For example, once a shopper has finished collecting items to purchase in the store, they can approach the self-checkout system 100 to perform the checkout transaction. The camera 122 may be configured to capture images of the predetermined zone 115 based on one or more trigger conditions, such as an indication that one or more items 102 have been detected within the predetermined zone 115, or an indication that a shopper has approached within a predetermined vicinity of the self-checkout system 100, or a weight sensor 116 that detects items on the platform 114, or a combination thereof. The camera 122 can continuously capture images of the predetermined zone 115, for example, throughout the checkout transaction. In at least some embodiments, a first camera 122A, a second camera 122B, and a third camera 122C can capture images of the article 102 on or near the platform 114 from their respective viewpoints, namely, the right, left, and overhead viewpoints. Furthermore, in at least some embodiments, a weight sensor 116 can acquire the article weight 117 of the article 102 placed on the platform 114.

[0028]

[0031] One or more processors 131 can receive images, or rather captured images 123, of one or more articles 102 placed on or near the platform 114. The captured images 123 of the articles 102 can be sent to a computing device 130 for processing. In addition, the article weight 117 of the articles 102, detected by the weight sensor 116, can be provided to the computing device 130 for processing.

[0029]

[0032] One or more processors 131 can determine identification information of one or more articles 102 placed on or near the platform 114 based on one or more images captured by one or more cameras 122, or rather, on at least a portion of the captured images 123. For example, in at least some embodiments, a computing device 130 can use the captured images 123 to determine the dimensions of the articles 102 placed on or near the platform 114. Images of the articles 102 captured from multiple viewpoints can help determine the dimensions of the articles 102. The captured images 123 can also be used by the computing device 130 to determine the packaging of the articles 102. For example, the color scheme, brand name, logo, etc. of the packaging of the articles 102. Furthermore, in some embodiments, in addition to the captured images 123, the article weight 117 of the articles 102 may be received by the computing device 130 and used to determine the total weight or individual weight of the articles 102 placed on the platform 114. Any suitable technique can be used to determine the individual weight of the articles 102 placed on the platform 114.

[0030]

[0033] In at least some embodiments, when the dimensions and weight of the article 102 are determined, one or more processors 131 can access the image library 135 and the article data 136, and one or more subsets of the plurality of library images 139 can be selected for use in comparison purposes based on the determined dimensions and weight of the article 102. For example, based on the dimensions and weight determined for a first article (e.g., snack bag 102A) on the platform 114, a first subset of the library images 139 can be selected. The first subset of the library images 139 can include images of articles known to have dimensions and weights similar to the first article based on the article data 136. The images of the first subset can be utilized for comparison purposes with the captured image 123 of the first article. Similarly, based on the dimensions and weight determined for a second article (e.g., bottled drinking water 102B) on the platform 114, a second subset of the library images 139 can be selected. The second subset can be a different subset from the first subset. The second subset of the library images 139 can include images of articles known to have dimensions and weights similar to the second article based on the article data 136. The images of the second subset can be utilized for comparison purposes with the captured image 123 of the second article. By narrowing the library images 139 to those showing articles having dimensions and weights similar to a given article on the platform 114, the processing time for image comparison can be advantageously reduced.

[0031]

[0034] Once a subset of library images 139 is selected for each article 102 placed on or near the platform 114, the captured image 123 of the article can be compared to the library images 139 of the selected subset for that article. For example, for the first article on the platform 114 (e.g., a snack bag 102A), the captured image 123 of the first article can be compared to the library images 139 of the first subset. Similarly, for a second article on the platform 114 (e.g., a bottle of water 102B), the captured image 123 of the second article can be compared to the library images 139 of the second subset. In at least some embodiments, the packaging of the article 102 (e.g., color scheme, brand name, logo, etc.) can be used for image comparison purposes. One or more processors 131 can implement an image recognition program for comparing the captured image 123 of the article 102 with library images 139 of known articles. When a “match” occurs between an image 123 of an item and a library image 139, one or more processors 131 can identify the item as, for example, a known purchasable item or a known unpurchasable item. In other words, if a match occurs, the item can be labeled or classified as a known item that may be either purchasable or unpurchasable. To put it another way, if an image 123 of a given item matches one or more library images 139 of known purchasable items, the identification information of the given item can be determined to be a known purchasable item. If an image 123 of a given item matches one or more library images 139 of known unpurchasable items, the identification information of the given item can be determined to be a known unpurchasable item. If no “match” occurs between an image 123 of an item and a library image 139, one or more processors 131 can identify the item as an unknown item.

[0032]

[0035] In some other embodiments, one or more processors 131 can determine identification information for one or more articles 102 placed on or near the platform 114 by comparing an image 123 of a given article with a subset of library images 139 from an image library 135, rather than a subset of the article. In yet another embodiment, one or more processors 131 can determine identification information for one or more articles 102 placed on or near the platform 114 by comparing an image 123 of a given article with a subset of library images 139, the subset being determined based on the dimensions (but not the weight) of a given article, the packaging (but not the weight or dimensions) of a given article, or a combination of the dimensions and packaging (but not the weight) of a given article.

[0033]

[0036] As an example, one or more processors 131 can identify snack bag 102A as a known, unpurchasable item, or rather, a snack bag known to the computing device 130 but unpurchasable, based on comparing the captured image 123 of snack bag 102A with a library image 139. As another example, one or more processors 131 can identify bottled water 102B as a known, purchasable item, or rather, bottled water that can be purchased, based on comparing the captured image 123 of bottled water 102B with a library image 139.

[0034]

[0037] Based on the identification information of the determined article 102, one or more processors 131 can cause the light source 140A to project an illumination indicator 142 onto at least one of the articles 102 placed on or near the platform 114. Continuing the above example, if the snack bag 102A is identified as a known unpurchasable article (i.e., an article known to the computing device 130 but unpurchasable), one or more processors 131 can instruct the light source 140A to project the illumination indicator 142 onto the snack bag 102A, for example, as shown in Figure 1. Illuminating the snack bag 102A can, for example, warn a shopper or store employee that the snack bag 102A is unpurchasable and should be removed from the platform 114. In at least some embodiments, as shown in Figure 1, the illumination indicator 142 can outline the article to be illuminated, which in this example is the snack bag 102A.

[0035]

[0038] In other embodiments, the illumination indicator 142 can be projected onto an article having a specific shape, letter, number, or any combination thereof, based on at least a portion of the determined identification information of the article. For example, the illumination indicator 142 can be projected onto the article by the light source 140A as a question mark or "?" if the article is identified as an unknown article. In another example, the illumination indicator 142 can be projected onto the article by the light source 140A as the letter X or "X" if the article is identified as a known, unpurchasable article.

[0036]

[0039] For example, as shown in Figure 4, the illumination indicator 142 is projected by the light source 140A onto an item known as "X" that is not available for purchase. In Figure 4, the item not available for purchase is a smartphone 102C that has been mistakenly placed on the platform 114. The "X" projected onto the smartphone 102C can inform a shopper or store employee that the smartphone 102C should be removed from the platform 114.

[0037]

[0040] In some embodiments, referring again to Figures 1 and 3, the illumination indicator 142 projected by the light source 140A can be represented in a predetermined color based on at least some of the identification information of the article to be illuminated. For example, the illumination indicator 142 can be projected onto an article that has yellow light from the light source 140A, for example, when the article is identified as an unknown article. In another example, the illumination indicator 142 can be projected onto an article that has red light from the light source 140A, for example, when the article is identified as a known unpurchasable article. In yet another example, the illumination indicator 142 can be projected onto an article that has green light from the light source 140A, for example, when the article is identified as a known purchasable article for which a coupon or special item is available. Thus, by projecting the illumination indicator 142 in a predetermined color, shoppers and / or store staff can be more easily warned of the type or opportunity of a problem.

[0038]

[0041] In at least some embodiments, one or more processors 131 can generate other instructions or warnings to be presented to a shopper or store employee, in addition to causing the light source 140A to project an illumination indicator 142 onto at least one item. For example, one or more processors 131 can cause the display 120 to present instructions or warnings to the shopper and / or store employee regarding the illuminated item. The instructions or warnings may indicate a desired action to be taken by the shopper or store employee regarding the illuminated item, such as "remove item from platform," "move item to platform," or "unrecognized item." In this regard, the light source 140A can project the illumination indicator 142 onto an item in conjunction with the display 120 that presents instructions or warnings to the shopper and / or store employee.

[0039]

[0042] As another example, one or more processors 131 can cause a speaker 144 to present an audible instruction or warning related to the illuminated item to the shopper and / or store employee. The audible instruction or warning may, for example, indicate a desired action that the shopper or store employee should take regarding the illuminated item. In this way, the light source 140A can project an illumination indicator 142 onto the item in conjunction with the speaker 144 that presents an audible instruction or warning to the shopper and / or store employee.

[0040]

[0043] In at least some embodiments, the light source of a self-checkout system can be controlled to project an illumination indicator onto an item, and then project the illumination indicator away from the platform to suggest to the shopper and / or store staff that the item should be removed from the platform. That is, first, the light source can be controlled to project an illumination indicator onto an item on the platform. Then, the light source can be controlled to project the illumination indicator outside the platform to suggest that the item be moved away from the platform to a location where the illumination indicator is projected outside the platform. An example is shown below.

[0041]

[0044] Figure 5 shows the self-checkout system 100, where a light source 140A moves the illumination indicator 142 from its highlight on the smartphone 102C (which could be, for example, a known unpurchasable item or an unknown item) to a position outside the platform 114. Initially, the light source 140A can be controlled to project the illumination indicator 142 onto the smartphone 102C. This projection of the illumination indicator 142 onto the smartphone 102C can be considered the first illumination position (indicated by the circled "1" in Figure 5). Subsequently, the light source 140A can be controlled to project the illumination indicator 142 to a position outside the platform 114. This projection of the illumination indicator 142 outside the platform 114 can be considered the second illumination position (indicated by the circled "2" in Figure 5). The movement of the illumination indicator 142 from the first illumination position to the second illumination position can suggest to the shopper or store employee that the smartphone 102C should be removed from the platform 114.

[0042]

[0045] In some embodiments, the light source 140A can be controlled to repeatedly project the illumination indicator 142 between the first and second illumination positions, for example, until the smartphone 102C is removed from the platform 114, or after a predetermined number of repetitions, such as three iterations. In other embodiments, the light source 140A can be controlled to project the illumination indicator 142 onto the smartphone 102C and then flash at least twice at the two illumination positions. This process can also be repeated. In some further embodiments, the light source 140A can be controlled to "drag" or continuously project the illumination indicator 142 when moving from the first illumination position to the second illumination position. In an embodiment in which the light source 140A is controlled to repeatedly project the illumination indicator 142 between a first and a second illumination position, the light source 140A can be controlled not to "drag" or continuously project the illumination indicator 142 when moving from the second illumination position to the first illumination position. Rather, after the light source 140A has dragged or continuously projected the illumination indicator 142 when moving from the first illumination position to the second illumination position, the light source 140A can be controlled to stop projection while the light source 140A adjusts its light emitter to focus on the first illumination position, for example, so that the illumination indicator is pulled away from the platform 114 but not toward the platform 114.

[0043]

[0046] In some further embodiments, the light source 140A can be controlled to first project the illumination indicator 142 onto the smartphone 102C with a first indicator type (e.g., "X" as shown in Figure 4), and then, when projecting the illumination indicator 142 to a second illumination position, the light source 140A can be controlled to project the illumination indicator 142 outside the platform with a second indicator type (e.g., a checkmark). By changing the illumination indicator type from one illumination position to the other, it is advantageous that the suggestion to move the highlighted item from the platform 114 can be made clearer to the shopper or store employee. In some embodiments, the checkmark can be projected with green light while the "X" can be projected with red light, thereby making the suggestion to the shopper and / or store employee even clearer.

[0044]

[0047] In at least some embodiments, the light source of a self-checkout system can be controlled to project an illumination indicator onto an item that is located near the platform but not on the platform, and then project the illumination indicator toward or onto the platform to suggest to the shopper and / or store staff that the item be moved onto the platform. That is, first, the light source can be controlled to project an illumination indicator onto an item located outside the platform. Then, the light source can be controlled to project the illumination indicator toward or onto the platform to suggest that the item be moved onto the platform. An example is given below.

[0045]

[0048] Figure 6 shows the self-checkout system 100, illustrating how the light source 140A moves the illumination indicator 142 from a state where it highlights an item (in this example, a carrot 102D) located near the platform 114 but not on the platform, to a position on the platform 114. The carrot 102D can be identified, for example, as a known purchasable item to be placed on the platform 114. As shown in Figure 6, the light source 140A can first be controlled to project the illumination indicator 142 onto the carrot 102D located outside the platform 114. The projection of the illumination indicator 142 onto the carrot 102D can be considered the first illumination position (indicated by the circled "1" in Figure 6). The light source 140A can then be controlled to project the illumination indicator 142 onto a position on the platform 114. The projection of the illumination indicator 142 onto the platform 114 can be considered the second illumination position (indicated by the circled "2" in Figure 6). The movement of the illumination indicator 142 from the first illumination position to the second illumination position can suggest to the shopper or store employee that the carrot 102D should be moved onto the platform 114.

[0046]

[0049] In some embodiments, the light source 140A can be controlled to repeatedly project the illumination indicator 142 between the first and second illumination positions, for example, until the carrot 102D moves onto the platform 114, or after a predetermined number of repetitions, such as three iterations. In other embodiments, the light source 140A can be controlled to project the illumination indicator 142 onto the carrot 102D and then flash at least twice at the two illumination positions. This process can also be repeated. In some further embodiments, the light source 140A can be controlled to "drag" or continuously project the illumination indicator 142 as it moves from the first illumination position to the second illumination position. In an embodiment in which the light source 140A is controlled to repeatedly project the illumination indicator 142 between a first and a second illumination position, the light source 140A can be controlled not to "drag" or continuously project the illumination indicator 142 when moving from the second illumination position to the first illumination position. Rather, after the light source 140A has dragged or continuously projected the illumination indicator 142 when moving from the first illumination position to the second illumination position, the light source 140A can be controlled to stop projection while the light source 140A adjusts its light emitter to focus on the first illumination position, for example, to indicate that the illumination indicator is being dragged toward the platform 114 but not away from the platform 114.

[0047]

[0050] In some further embodiments, the light source 140A can be controlled to first project the illumination indicator 142 onto the carrot 102D having a first indicator type (e.g., a red outline of the carrot 102D), and then, when projecting the illumination indicator 142 to a second illumination position, the light source 140A can be controlled to project the illumination indicator 142 onto the platform 114 having a second indicator type (e.g., a green checkmark). By changing the type of illumination indicator and / or the color of the light from one illumination position to the other, the suggestion to move the highlighted item onto the platform 114 can be made clearer to the shopper or store employee.

[0048]

[0051] Figure 7 is a flowchart of a method 200 for illuminating one or more items within a predetermined zone of a self-checkout system, according to one or more embodiments of the present disclosure.

[0049]

[0052] In 202, method 200 may include using one or more cameras to capture images of one or more items placed on or near the platform of the self-checkout system within a predetermined zone of the self-checkout system. For example, the self-checkout system may include at least three cameras, at least one of which is at a different height from the others. The cameras can capture images of items on or near the platform.

[0050]

[0053] In 204, Method 200 may include determining identification information of one or more articles placed on or near a predetermined zone on the platform based on at least a portion of one or more images captured by one or more cameras. For example, cameras and a computing device can form a computer vision system for a self-checkout system. The computing device can be used to perform image analysis techniques to determine, for example, the shape, dimensions, weight, and / or location of one or more articles placed on and / or near the platform. The determined shape, dimensions, weight, and / or location of one or more articles can be used to identify one or more articles.

[0051]

[0054] In 206, Method 200 may include using a light source of a self-checkout system to project an illumination indicator onto at least one of one or more items placed on or near the platform in a predetermined zone, based on identification information of at least one item. For example, the self-checkout system may include a light source such as a projector or laser, which can be used to project an illumination indicator onto at least one item on or near the platform. As an example, an item may be identified as an unknown item. The light source may be controlled to project an illumination indicator onto the unknown item, for example, to inform a shopper or store employee that the item should be removed from the platform or otherwise removed from the purchase zone. As another example, an item may be identified as a known non-purchasable item. The light source may be controlled to project an illumination indicator onto the known non-purchasable item, for example, to inform a shopper or store employee that the item should be removed from the platform or otherwise removed from the purchase zone. As yet another example, an item may be identified as a known purchasable item that has been scanned but is not placed on the platform. The light source can be controlled to project illumination indicators onto known purchasable items, for example, to inform shoppers or store staff that an item should be moved onto the platform.

[0052]

[0055] In some implementations, Method 200 may include using a light source to project an illumination indicator onto an illumination position at a certain distance from at least one article, thereby suggesting that at least one article be moved to the illumination position. For example, if an article is placed on the platform and should not be, the light source can be controlled to project the illumination indicator onto a position outside the platform, as determined by the article's identification information. This projection of the illumination indicator onto the illumination position can suggest to the shopper or store employee that the article be moved off the platform. In another example, if an article is placed off the platform and should actually be placed on the platform, the light source can be controlled to project the illumination indicator onto a position on the platform, as determined by the article's identification information. This projection of the illumination indicator onto the illumination position can suggest to the shopper or store employee that the article be moved onto the platform.

[0053]

[0056] Therefore, illuminating specific items within a purchase zone can, among other advantages, help create a smoother, more interactive shopping experience for the buyer and enable store staff to identify and resolve problems more quickly.

[0054]

[0057] The descriptions of various embodiments are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Rather, any combination of the described features and elements is intended to carry out and implement the assumed embodiments, whether or not they relate to different embodiments. Furthermore, while the embodiments disclosed herein may achieve other possible solutions or advantages over the prior art, whether or not advantages are achieved by a given embodiment does not limit the scope of this disclosure. Accordingly, the aspects, features, embodiments and advantages disclosed herein are merely illustrative and shall not be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0055]

[0058] The embodiments described may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware embodiments which may be commonly referred to herein as “circuits,” “modules,” or “systems.”

[0056]

[0059] One or more of the embodiments described may be a system, method, and / or a computer program product. The computer program product may include a computer-readable storage medium(s) having computer-readable program instructions for causing a processor to execute an aspect of the embodiment.

[0057]

[0060] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of examples of computer-readable storage mediums includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved raised structures on which instructions are recorded, and any suitable combination thereof. As used herein, computer-readable storage mediums should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through optical fiber cables), or electrical signals transmitted through wires.

[0058]

[0061] The computer-readable program instructions described herein can be downloaded to each computing / processing device from a computer-readable storage medium or via an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0059]

[0062] The computer-readable program instructions for performing the operations of the embodiments described may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to perform aspects of the embodiments described.

[0060]

[0063] The aspects of the embodiments described herein are explained herein with reference to flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0061]

[0064] These computer-readable program instructions may be provided to a general-purpose computer, a dedicated computer, or a processor of another programmable data processing device for manufacturing a machine, such that instructions executed via the processor of a computer or other programmable data processing device create means for performing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in the manner described, and as a result, a computer-readable storage medium having stored instructions may include a product containing instructions that perform a manner of function / operation specified in one or more blocks of a flowchart and / or block diagram.

[0062]

[0065] Computer-readable program instructions may also be loaded onto a computer, another programmable device, or other device to generate a computer implementation process by causing the computer, another programmable device, or other device to execute a series of action steps on the computer, another programmable device, or other device so that the instructions executed on the computer, another programmable device, or other device perform a function / operation specified in one or more blocks of a flowchart and / or block diagram.

[0063]

[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions described in a block may be performed in a different order than shown in the figure. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the related functions. It should also be noted that each block in a block diagram and / or flowchart, as well as any combination of blocks in a block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0064]

[0067] While the above covers one or more embodiments, other and further embodiments can be devised without departing from their basic scope, and the scope is determined by the following claims. [Explanation of Symbols]

[0065] 100 Self-Checkout Systems 102 Goods 102A Snack Bag 102B Bottled drinking water 102C Smartphone 102D Carrot 110 base 112 Back Panel 114 platforms 115 Zones 116 Weight Sensor 117 Article weight 118 User Input Devices 120 displays 122 Cameras 122A First camera 122B Second Camera 122C Third Camera 122D Forward-facing camera 123 Acquired Images 124 Right Arm 126 Left Arm 128 Halo 130 Computing Devices 131 processors 132 memory devices 133 Programs 134 data 135 Image Library 136 Item Data 137 Communication Interface 138 Communications Bus 139 Library Images 140 light source 140A light source 140B light source 142 Irradiation Indicator 144 speakers 150 datastores 160 Networks

Claims

1. One or more cameras, Light source and A computing device having one or more processors and one or more memory devices for storing programs, which causes the one or more processors to perform operations individually or collectively, A self-checkout system equipped with, The aforementioned operation, The steps include receiving images of one or more items placed within a predetermined zone of the self-checkout system, which are images captured by one or more cameras, A step of determining identification information of one or more articles placed within a predetermined zone based on at least a portion of the one or more images captured by the one or more cameras, The steps of causing the light source to project an illumination indicator onto at least one of the one or more articles placed within the predetermined zone, based on the identification information of the at least one article; Includes a self-checkout system.

2. The self-checkout system according to claim 1, wherein the one or more cameras include at least three cameras, and at least one of the at least three cameras is at a different height from the other at least three cameras.

3. The self-checkout system according to claim 1, wherein the light source is supported by a halo and positioned above the platform.

4. The self-checkout system according to claim 1, wherein the light source is located beneath the platform or embedded in the platform.

5. The self-checkout system according to claim 1, wherein the identification information of each of the one or more articles can be determined to be either one of a plurality of known articles or an unknown article.

6. The aforementioned self-checkout system further comprises a platform, The aforementioned operation, A step of determining the dimensions and weight of each of the one or more articles placed on the platform within the predetermined zone, The step further includes comparing one or more images of the one or more articles placed on the platform with a subset of a plurality of library images of known articles, based on at least a portion of the dimensions and weight of one of the one or more articles, If one or more images of the given article match one or more library images of known articles in the subset associated with the given article, the identification information of the given article among the one or more articles is determined to be a known article. If one or more images of a given article do not match any of the multiple library images of known articles in the subset associated with the given article, the identification information of the given article is determined to be an unknown article. The self-checkout system according to claim 1.

7. The aforementioned operation, The step further includes comparing one or more images of one or more articles placed within the predetermined zone with a plurality of library images of known articles, If one or more images of the given article match one or more library images of known purchasable articles, the identification information of the given article among the one or more articles is determined to be a known purchasable article. If one or more images of the given article match one or more library images of known unpurchasable articles, the identification information of the given article among the one or more articles is determined to be a known unpurchasable article. The at least one article on which the irradiation indicator is projected is at least one of the known unpurchasable articles, while the known purchasable articles are not irradiated by the light source having the irradiation indicator. The self-checkout system according to claim 1.

8. The self-checkout system according to claim 1 further comprises a platform, The at least one article onto which the illumination indicator is projected is located near the platform within the predetermined zone but not on the platform, and when the light source projects the illumination indicator onto the at least one article, the light source causes the illumination indicator to illuminate the at least one article at a first illumination position outside the platform, and then to illuminate the illumination indicator at a second illumination position on the platform, or causes the illumination indicator to illuminate the second illumination position on the platform, and then to illuminate the at least one article at a first illumination position outside the platform. The self-checkout system according to claim 1.

9. The self-checkout system according to claim 1 further comprises a platform, The at least one article onto which the illumination indicator is projected is placed on the platform within the predetermined zone, and when the light source projects the illumination indicator onto the at least one article, the light source causes the illumination indicator to illuminate the at least one article at a first illumination position on the platform, and then to illuminate the illumination indicator at a second illumination position outside the platform, or causes the illumination indicator to illuminate the second illumination position outside the platform, and then to illuminate the at least one article at a first illumination position on the platform. The self-checkout system according to claim 1.

10. The self-checkout system according to claim 1, wherein when the light source projects the irradiation indicator onto the at least one article, the light source irradiates the irradiation indicator onto the at least one article at a first irradiation position, and then irradiates the irradiation indicator onto a second irradiation position, wherein the irradiation indicator is a first indicator type at the first irradiation position and a second indicator type at the second irradiation position, and the first indicator type is different from the second indicator type.

11. The self-checkout system according to claim 1, wherein when the light source projects the illumination indicator onto the at least one article, the light source illuminates the illumination indicator onto the at least one article at a first illumination position, and then illuminates the illumination indicator at a second illumination position, and the illumination indicator is a first color at the first illumination position and a second color at the second illumination position.

12. The self-checkout system according to claim 1, wherein when the light source projects the irradiation indicator onto the at least one article, the light source irradiates the at least one article with the irradiation indicator at a first irradiation position, and then irradiates the irradiation indicator at a second irradiation position, and the irradiation indicator is continuously projected and dragged from the first irradiation position to the second irradiation position.

13. The self-checkout system according to claim 1, wherein when the light source projects the irradiation indicator onto the at least one article, the light source irradiates the irradiation indicator with a color selected from a plurality of colors based on at least a portion of the identification information of the at least one article.

14. The self-checkout system according to claim 1, wherein when the light source projects the irradiation indicator onto the at least one article, the light source irradiates the at least one article with the irradiation indicator using an indicator type selected from a plurality of indicator types, based on at least a portion of the identification information of the at least one article.

15. A computer-readable storage medium in which computer-readable program code is embodied, wherein the computer-readable program code is executable by one or more computer processors to perform operations individually or collectively, and the operations are A step of receiving one or more images of one or more items placed within a predetermined zone of the self-checkout system, wherein the one or more images are captured by one or more cameras of the self-checkout system. A step of determining identification information of one or more articles placed within a predetermined zone based on at least a portion of the one or more images captured by the one or more cameras, The process includes the step of causing the light source of the self-checkout system to project an illumination indicator onto at least one of the one or more articles placed within the predetermined zone, based on the identification information of the at least one article. Computer-readable storage medium.

16. The computer-readable storage medium according to claim 15, wherein the at least one article onto which the illumination indicator is projected is located near the platform of the self-checkout system in the predetermined zone but is not located on the platform, and when the light source projects the illumination indicator onto the at least one article, the light source causes the illumination indicator to illuminate the at least one article at a first illumination position outside the platform, and then causes the illumination indicator to illuminate a second illumination position on the platform, or causes the illumination indicator to illuminate a second illumination position on the platform, and then causes the illumination indicator to illuminate the at least one article at a first illumination position outside the platform.

17. The computer-readable storage medium according to claim 15, wherein the at least one article on which the illumination indicator is projected is placed within the predetermined zone on the platform of the self-checkout system, and when the light source projects the illumination indicator onto the at least one article, the light source causes the illumination indicator to illuminate the at least one article at a first illumination position on the platform, and then to illuminate the illumination indicator at a second illumination position outside the platform, or causes the illumination indicator to illuminate the second illumination position outside the platform, and then to illuminate the illumination indicator at the first illumination position on the platform.

18. When the light source projects the irradiation indicator onto the at least one article, the light source Based on at least a portion of the identification information of the aforementioned at least one article, a color selected from a plurality of colors, or Based on at least a portion of the identification information of the at least one article, an indicator type selected from a plurality of indicator types, or Both, Then, the irradiation indicator is made to irradiate the at least one article. The computer-readable storage medium according to claim 15.

19. The steps include: using one or more cameras to capture images of one or more items placed on or near the platform of the self-checkout system within a predetermined zone of the self-checkout system; A step of determining identification information of one or more articles placed within a predetermined zone based on at least a portion of the one or more images captured by the one or more cameras, Using the light source of the self-checkout system, the steps include projecting an illumination indicator onto at least one of the one or more items placed within the predetermined zone, based on the identification information of the at least one item; Methods that include...

20. The light source further includes the step of projecting the illumination indicator onto an illumination position at a certain distance from the at least one article in order to suggest that the at least one article is moved to the illumination position, The method according to claim 19.

Images