Vision-based processing method and system for unrecognized items in a self-checkout system

Abstract

In vision-based self-checkout (SCO) systems, the handling of unrecognized items that cannot be individually distinguished is improved. By avoiding repeated recognition processes for the same item without requiring customers to remove unrecognized items from the tray, staff intervention is reduced, and the customer experience is enhanced. [Solution] A vision-based SCO terminal acquires image data of the items on the tray and determines that the item is an unrecognized item. After receiving identification information of the identified items associated with the unrecognized item, a visual exclusion zone is created corresponding to the location of the identified items, and subsequent detection of identified items within the visual exclusion zone is ignored. By comparing the aspect ratio and center point of the bounding box viewed from above, if a different item appears within the visual exclusion zone, the zone is cleared, enabling the recognition of new items.

Description

Technical Field

[0001] The present invention relates to a processing method and system for unrecognized products in a vision-based (sensor image-based) self-checkout (SCO) system that performs processing and determination.

Summary of the Invention

Problems to be Solved by the Invention

[0002] Vision checkout systems in retail environments face significant challenges when customers place products on trays that cannot be individually distinguished by the SCO system. Current solutions require the user to remove unrecognized products from the tray, which causes confusion and frustration, especially when multiple products are present. This leads to transaction delays, reduced accuracy, and overall deterioration of the customer experience, ultimately increasing staff intervention and reducing the efficiency of the SCO system.

[0003] In addition, accurately tracking products between video frames is difficult due to the discrete nature of video acquisition and the similar appearance of many retail products, and there is a lack of efficient methods to visually distinguish identified products from new products. Furthermore, due to the technical limitations of camera-based product tracking, it is difficult to maintain an accurate product count without forcing customers to completely remove products from the vision area.

[0004] The present invention has been made in view of the above problems, and aims to solve the technical problems related to the processing of unrecognized products, reduce staff intervention, and improve the customer experience.

Means for Solving the Problems

[0005] To solve the above problems, a method according to one aspect of the present invention includes: acquiring image data of at least one item on a tray using a vision-based self-checkout (SCO) terminal; determining that at least one item is an unrecognized item; receiving identification information of identified items associated with the unrecognized item; creating a visual exclusion zone corresponding to the location of the identified items; and ignoring subsequent detection of identified items within the visual exclusion zone.

[0006] A method according to another aspect of the present invention includes: using at least one camera of a vision-based SCO terminal to acquire image data from above one or more items on a checkout tray; identifying at least one unrecognized item from the one or more items; displaying a user interface for identifying at least one unrecognized item; receiving user input to identify at least one unrecognized item as identified item; generating a dead zone corresponding to the location of the identified item; and preventing further prompting for identification of the identified item while the identified item remains within the dead zone.

[0007] A system according to yet another aspect of the present invention comprises at least one camera configured to acquire image data of at least one item on a checkout tray, at least one processor, and at least one memory for storing instructions, wherein, when executed by the at least one processor, the system causes the system to analyze the image data to identify at least one unrecognized item on the checkout tray, to receive identification information of an identified item associated with the at least one unrecognized item, to create an exclusion region corresponding to the location of the identified item in the image data, to compare subsequent image data of the exclusion region with the stored characteristics of the identified item, and to maintain the exclusion region if the comparison indicates that the identified item remains within the exclusion region. [Effects of the Invention]

[0008] According to the present invention, it is possible to allow customers to leave identified items on the tray while preventing the system from repeatedly prompting them to identify the same item. This reduces the need for staff intervention, improves transaction speed and accuracy, and enhances the customer experience.

[0009] Furthermore, by using a camera view from above and bounding box information, computationally efficient processing is possible, and the problem of repeated product recognition in conventional vision-based checkout systems can be effectively solved. [Brief explanation of the drawing]

[0010] [Figure 1] A system diagram for vision-based processing of unrecognized self-checkout (SCO) items according to an exemplary embodiment of the present invention is shown. [Figure 2] A flowchart shows a method for vision-based processing of unrecognized SCO products according to an exemplary embodiment of the present invention. [Figure 3] This shows an exemplary user interface (UI) screen related to an unrecognized SCO product according to an exemplary embodiment of the present invention. [Figure 4] An exemplary UI screen related to a customer identifying an unrecognized SCO product according to an exemplary embodiment of the present invention is shown. [Figure 5] An exemplary UI screen relating to a visual exclusion zone associated with a customer-identified SCO product is shown, according to an exemplary embodiment of the present invention. [Figure 6] A flowchart shows a method for processing vision-based unrecognized SCO products according to an exemplary embodiment of the present invention. [Figure 7] A flowchart of another method for processing vision-based unrecognized SCO products, according to an exemplary embodiment of the present invention, is shown. [Modes for carrying out the invention]

[0011] Currently, a significant number of vision checkout transactions require employee intervention, creating a bottleneck in the self-checkout (SCO) process. The user interface flow for handling unrecognized items is confusing and cumbersome for customers, negatively impacting the adoption of vision checkout technology. This problem is exacerbated by the constant addition of new products to the store database and changes in product packaging, meaning that unrecognized items continuously appear in transactions. Certain products, such as salads, coffee, and sandwiches, are often indistinguishable from similar products from a vision perspective, creating a persistent recognition challenge.

[0012] The current approach, which requires customers to remove unrecognized items from the tray after identification, leads to several problems, including repeated recognition loops, double counting of items, or accidentally excluding items entirely from a transaction. Many stores lack the extra counter space for customers to move items aside, and tracking individual items with cameras is extremely resource-intensive and requires expensive equipment. Customers find it cumbersome to move some items aside while leaving the rest on the tray, resulting in a frustrating and confusing experience.

[0013] A key technical challenge in vision checkout systems is accurately tracking products across video frames, a problem exacerbated by the discrete nature of image acquisition and the similar appearance of many retail products. For example, if a different coffee appears on a tray in a similar position to a previously identified coffee, it is nearly impossible for the system to determine whether it is the same coffee or a new one. Current systems lack an efficient way to visually distinguish identified products from new products placed in similar positions, leading to redundant reminders, missales, and customer frustration. In addition, the technical limitations of camera-based product tracking make it difficult to maintain an accurate count without forcing customers to completely remove products from their vision area.

[0014] The technology disclosed herein provides a technical solution to at least the aforementioned technical challenges by excluding the area in which the goods were sold from image recognition until payment is initiated or certain conditions are met. This is achieved by comparing the aspect ratio and center point of the bounding box viewed from above to ensure that the position of the recognized goods has not moved significantly and can be ignored. In one embodiment, the bounding box may include a bounding rectangle. The method and system ignore the movement of goods in a zone after goods have been identified in that zone by creating a “dead zone” from the image viewed from above, and automatically clears this dead zone to indicate that previously detected goods have been removed if goods with different center points, areas, and aspect ratios appear in the area.

[0015] In one embodiment, the method and system prevent the unrecognized product screen from being displayed twice for the same product by excluding the area where the product is registered for sale from image recognition until payment is initiated. When a customer touches a candidate product on the screen without disturbing the position of existing products on the tray, the method and system creates a block area corresponding to the location of that product. Once the product is scanned, the customer will inevitably pick it up and move it from its original position. When the customer places the product in the bagging area, if the number of products in the vision has decreased by one, the method and system can skip the "bagging complete" confirmation. The block area is not kept blocked for the entire duration of the transaction under certain conditions. If the method and system determines from an overhead view that the area is empty, the rectangular block area is cleared, and product searching can be performed again in that area. Similarly, if another product with a sufficiently different aspect ratio and / or center point from the sold product is placed in the same position, the method and system determines that a new product exists and clears the block area. However, if an item with minimal variation in center point, size, and aspect ratio remains in that location, the method and system will assume it is the same item previously identified and will not prompt the user again.

[0016] The visual exclusion zone or dead zone approach provides a technique that allows customers to leave identified items on the tray while preventing the system from repeatedly prompting them to identify the same item. This significantly reduces the need for staff intervention, improves transaction speed and accuracy, and enhances the overall customer experience. By utilizing overhead camera views and bounding box information, the embodiments presented herein are computationally efficient and effectively solve the problem of repeated item recognition in conventional vision-based checkout systems.

[0017] FIG. 1 is a diagram of a system 100 for vision-based processing of unrecognized self-checkout (SCO) items according to an exemplary embodiment. It should be noted that each component is schematically shown in a simplified form, and only the components relevant to the understanding of the embodiments are illustrated.

[0018] Further, various components (identified in system 100) are illustrated, and the arrangement of components is presented for illustrative purposes only. In particular, other arrangements with more or fewer components are possible without departing from the teachings of vision-based processing of unrecognized SCO items presented herein and hereinafter.

[0019] The system 100 includes a cloud (or server) 110, one or more SCO terminals 120, and one or more retailer servers 130. The cloud 110 includes at least one processor 111 and a non-transitory computer-readable storage medium (hereinafter referred to as "medium") 112, and the medium 112 includes instructions for a vision-based item recognition unit 113, an unrecognized item manager 114, and an application programming interface (API) 115. When these instructions are executed by the processor 111, they cause the processor 111 to perform the processes or operations described herein and hereinafter with respect to 113 to 115.

[0020] Each SCO terminal 120 includes at least one processor 121 and a medium 122, and the medium 122 includes instructions for at least a transaction manager 123 and a transaction user interface (UI) 124. When these instructions are executed by the processor 121, they cause the processor 121 to perform operations related to 123 to 124.

[0021] Each SCO terminal 120 is also connected to at least one camera 125. In one embodiment, the camera 125 is an overhead camera having a downward field of view relative to the tray or countertop of the SCO terminal 120.

[0022] Each retailer server 130 includes at least one processor 131 and a medium 132, and the medium 132 includes instructions for the transaction system 133. When these instructions are executed by the processor 131, they cause the processor 131 to perform the processes or operations described herein and below with respect to 133.

[0023] The vision-based product recognition unit 113 is responsible for analyzing image data to identify products placed on the tray of the SCO terminal during an SCO transaction. When products cannot be distinguished individually, the unrecognized product manager 114 functions. The unrecognized product manager 114 implements a "dead zone" or "visual exclusion zone" by processing the image data input to the vision-based product recognition unit 113. This prevents the vision-based product recognition unit 113 from attempting to repeatedly recognize the product after the product has been once identified and counted in the dead zone or visual exclusion zone.

[0024] The unrecognized product manager 114 creates and manages visual exclusion zones for identified products that were not initially recognized. The unrecognized product manager 114 monitors these zones and determines the timing to delete the zones based on specific conditions such as the timing when a product with different visual characteristics appears in the zone. This directly addresses the technical problems associated with product tracking between video frames, which result from the discrete nature of video frames and the similar appearance of products.

[0025] The API 115 facilitates communication among the vision-based product recognition unit 113, the unrecognized product manager 114, and the transaction manager 123 on the SCO terminal 120. The API 115 enables seamless integration of the system 100 with vision-based transaction processing.

[0026] The transaction manager 123 controls the entire transaction flow, including payment processing and product registration. The transaction manager 123 communicates with the vision-based product recognition unit 113 and the unrecognized product manager 114 via API 115 to integrate vision-based product recognition into the checkout process.

[0027] The transaction UI 124 displays interface screens such as the exemplary screens shown in Figures 3 to 5, which will be described later, and enables the user to interact with the system 100 when an unrecognized product is detected. The transaction UI 124 displays a tray image containing recognized and unrecognized products, candidate products, and options for scanning or searching for products.

[0028] Camera 125 captures a view from above the checkout tray and provides image data to be analyzed by the vision-based product recognition unit 113. The view from above provided in the image data allows the unrecognized product manager 114 to determine whether the same product or a new product has appeared at a given position on the tray.

[0029] A retailer server 130, having a processor 131 and media 132, hosts a transaction system 133, which manages the entire retail transaction process. This transaction system 133 communicates with a transaction manager 123 on an SCO terminal 120 to process SCO transactions, process payments, update inventory, and maintain transaction records.

[0030] In actual implementation, system 100 operates as follows: A customer places multiple items, including a coffee cup that cannot be individually distinguished by the vision-based product recognition unit 113, onto the tray of the SCO terminal. Camera 125 captures a view from above the tray, and the vision-based product recognition unit 113 analyzes the image data. The vision-based product recognition unit 113 identifies the coffee cup as an unrecognized item and notifies the transaction manager 123 of this via API 115.

[0031] The transaction manager 123 instructs the transaction UI 124 to display the unrecognized product screen (shown in Figure 3), highlight the unrecognized product, and present candidate products. The customer selects the correct product from the candidates (shown in Figure 4). The unrecognized product manager 114 creates a visual exclusion zone in the subsequent image data of the SCO tray. This visual exclusion zone corresponds to the location, area, or region of the identified coffee cup. This prevents the vision-based product recognition unit 113 from continuing to attempt to sell the product during the SCO transaction unless a different product is included in the subsequent image data of the tray.

[0032] If a customer places another product with different visual characteristics (e.g., different size or shape) in the same position, the Unrecognized Product Manager 114 detects this change by comparing the aspect ratio and center point of the bounding box viewed from above, and eliminates exclusion zones in the image data, enabling the vision-based product recognition unit 113 to identify the new product. The SCO transaction continues without repeated prompting of the customer about the identified coffee cup, improving the checkout experience by reducing the need for staff intervention.

[0033] System 100 directly addresses the technical challenges associated with vision-based SCO checkout by reducing staff intervention and improving the currently cumbersome and confusing unrecognized item processing flow. Furthermore, System 100 prevents repeated reminders for the same unrecognized item while allowing customers to leave identified items on their trays.

[0034] Having described the system architecture and the relationships between its components in Figure 1, the system's operation will now be described as shown in Figures 2 to 5. Figure 2 is a flowchart of Method 200 for vision-based processing of unrecognized SCO items, and Figures 3 to 5 show the transitions in the transaction UI 124 as unrecognized items are detected, identified, and subsequently managed through the visual exclusion zone function. Collectively, these figures illustrate how the components described in Figure 1 work together to provide an improved SCO experience for customers using the vision-based self-checkout terminal 120.

[0035] Figure 2 is a flowchart of Method 200 for vision-based processing of unrecognized SCO products according to an exemplary embodiment. The unrecognized product manager 114 implements Method 200.

[0036] In step 201, the Unrecognized Product Manager 114 detects, based on image data captured by the camera 125, that a product has been placed on the tray of the SCO terminal during an SCO transaction. In step 202, the Unrecognized Product Manager 114 determines that the product is unrecognized based on candidate products already output for the product provided by the vision-based product recognition unit 113.

[0037] In step 203, the Unrecognized Item Manager 114 detects that the customer has scanned a barcode or selected a candidate item related to the unrecognized item, thereby allowing the Transaction Manager 123 to obtain product identification information for the unrecognized item. In step 204, the Unrecognized Item Manager 114 creates a ignore region in the image data in the top view for the location or area previously occupied by the item. In step 205, the Vision-Based Item Recognition Unit 113 is prevented from querying the customer or user about the unrecognized item during the SCO transaction.

[0038] In step 206, the Unrecognized Item Manager 114 detects an item placed in the ignore area. In step 207, the Unrecognized Item Manager 114 determines whether the item has a different shape, position, or size from the original unrecognized item. In step 208, if the item is different from the original unrecognized item, the Unrecognized Item Manager 114 clears the ignore area. In step 209, by clearing the ignore area from the image data, the Unrecognized Item Manager 114 allows the vision-based item recognition unit 113 to process the previous ignore area in the image data in order to attempt to visually identify a new item during an SCO transaction.

[0039] Figure 3 shows an exemplary user interface (UI) screen 300 related to an unrecognized SCO product according to an exemplary embodiment. The UI screen 300 has two main areas: area 301 which displays an image captured by camera 125 showing the checkout tray with recognized product 302 and unrecognized product 303, and area 304 which presents candidate options for the unrecognized product. Area 304 displays candidate products (305, 306) that the vision-based product recognition unit 113 has determined to be potential matches for the unrecognized product 303.

[0040] The transaction UI 124 provides the user with multiple options for identifying unrecognized items: selecting one of the candidate items, scanning the item using the scan button 307, or searching for the item using the search button 308. The user can select an unrecognized item 303, as indicated by the selection 309, which highlights the item in the tray image, helping the user understand which item needs to be identified. The transaction UI 124 facilitates interaction between the user and the system 100 to efficiently resolve the unrecognized item problem.

[0041] Figure 4 shows an exemplary UI screen 400 relating to a customer identifying an unrecognized SCO product, according to an exemplary embodiment. Figure 4 is similar to Figure 3, but shows UI screen 400 with the user selecting candidate products. UI screen 400 maintains the same layout, including an area 401 showing a tray image containing recognized products 402 and unrecognized products 403, and an area 404 showing candidate options. Candidate products (405, 406) are presented as potential matches for the unrecognized product, along with scan 407 and search 408 options.

[0042] In Figure 4, the user selects an unrecognized product as indicated by selection 409 and then selects "Candidate Product #2" as indicated by selection 410. This selection process allows the user to identify the unrecognized product without removing it from the tray, thereby improving the efficiency of the checkout process. After the selection is made, the system 100 proceeds to create an exclusion zone for the product, as shown in Figure 5.

[0043] Figure 5 is an exemplary UI screen 500 relating to a visual exclusion zone associated with customer-identified SCO products, according to an exemplary embodiment. Figure 5 depicts the UI screen 500 showing the state after the user has identified unrecognized products, as shown in Figure 4. The UI screen 500 maintains the same layout, having an area 501 that shows a tray image containing recognized products 502 and previously unrecognized products 503 that are now identified. Area 504 continues to display candidate options (505, 506) along with scan 507 and search 508 options.

[0044] The difference in Figure 5 is that the Unrecognized Product Manager 114 has created a dead zone or exclusion zone in the image data for the previously unrecognized product 503. This prevents the vision-based product recognition unit 113 from repeatedly attempting to recognize the product. Without this prevention, multiple reminders for the same product would occur. The Unrecognized Product Manager 114 continues to monitor this exclusion zone, and if a new product with different characteristics is placed in the area, it removes the exclusion zone from the image data input to the vision-based product recognition unit 113, allowing it to attempt to recognize the new product. This approach significantly improves the checkout experience by reducing unnecessary reminders while maintaining the system's ability to detect new products.

[0045] Figure 6 is a flowchart of Method 600 for processing vision-based unrecognized SCO products according to an exemplary embodiment. The software module implementing Method 600 is called the “Unrecognized Product SCO Assistant”. The Unrecognized Product SCO Assistant is implemented as executable instructions programmed and resident in memory and / or non-temporary computer-readable (processor-readable) storage medium and executed by one or more processors in one or more devices. The processor of the device running the Unrecognized Product SCO Assistant is configured and programmed specifically to process the Unrecognized Product SCO Assistant. The Unrecognized Product SCO Assistant may have access to one or more network connections during its processing. The network connections may be wired, wireless, or a combination of wired and wireless.

[0046] In one embodiment, the device running the Unrecognized Product SCO Assistant is the cloud 110 and / or the SCO terminal 120. In one embodiment, the Unrecognized Product SCO Assistant consists of a vision-based product recognition unit 113, an Unrecognized Product Manager 114, an API 115, a transaction manager 123, and / or a transaction UI 124.

[0047] In 610, the Unrecognized Product SCO Assistant acquires image data depicting at least one product on the SCO tray. In one embodiment, in 611, the Unrecognized Product SCO Assistant acquires image data using a camera view from above the tray.

[0048] In step 620, the Unrecognized Product SCO Assistant determines that the product is an unrecognized product. In one embodiment, in step 621, the Unrecognized Product SCO Assistant determines that the unrecognized product cannot be individually distinguished by the vision-based SCO terminal 120.

[0049] In 630, the Unrecognized Product SCO Assistant receives identification information of an identified product associated with the unrecognized product. In one embodiment, in 631, the Unrecognized Product SCO Assistant receives the selection of an identified product from a list of candidate products displayed on the transaction UI 124 of the vision-based SCO terminal 120. In one embodiment, in 632, the Unrecognized Product SCO Assistant receives scan data by scanning the barcode of the unrecognized product to obtain identification information of the identified product.

[0050] In 640, the Unrecognized Product SCO Assistant creates a visual exclusion zone corresponding to the location of identified products in the image data depicting the tray. In one embodiment, in 641, the Unrecognized Product SCO Assistant defines a bounding box around the unrecognized products within the image data based on the image data.

[0051] In 650, the Unrecognized Item SCO Assistant ignores subsequent detection of identified items within the visual exclusion zone of the image data. In one embodiment, in 651, the Unrecognized Item SCO Assistant maintains the visual exclusion zone until payment is initiated, as indicated by the transaction manager 123 of the SCO terminal 120.

[0052] In one embodiment, in 660, the Unrecognized Product SCO Assistant determines that the visual exclusion zone is empty based on the image data. The Unrecognized Product SCO Assistant maintains the visual exclusion zone in the image data because this determination indicates that the customer likely bagged the identified product after it had been identified.

[0053] In one embodiment, in 670, the Unrecognized Product SCO Assistant detects a new product within the visual exclusion zone from the image data. The Unrecognized Product SCO Assistant compares at least one of the new product's center point, size, and aspect ratio with the corresponding characteristics of the identified product. If the comparison indicates that the new product is different from the identified product, the Unrecognized Product SCO Assistant clears the visual exclusion zone.

[0054] In one embodiment, at 680, the Unrecognized Product SCO Assistant determines that an identified product has been removed from the tray based on a decrease in the number of products detected by the vision-based SCO terminal 120. The Unrecognized Product SCO Assistant then omits prompting the customer to confirm bagging, as it is highly likely that the customer has bagged the identified product.

[0055] In one embodiment, in 690, the Unrecognized Goods SCO Assistant maintains the visual exclusion zone if the center point, size, and aspect ratio of a particular good within the visual exclusion zone do not deviate significantly from the corresponding characteristics of the identified good. This indicates that the identified good remains within the visual exclusion zone during the SCO transaction.

[0056] Figure 7 is a flowchart of another method 700 for processing vision-based unrecognized SCO products, according to an exemplary embodiment. The software module implementing method 700 is called the “Vision-Based SCO Assistant”. The Vision-Based SCO Assistant is implemented as executable instructions programmed and resident in memory and / or non-temporary computer-readable (processor-readable) storage medium and executed by one or more processors in one or more devices. The processor running the Vision-Based SCO Assistant is configured and programmed specifically to process the Vision-Based SCO Assistant. In one embodiment, the Vision-Based SCO Assistant may have access to one or more network connections during its processing. The network connections may be wired, wireless, or a combination of wired and wireless.

[0057] In one embodiment, the device running the vision-based SCO assistant is the cloud 110 and / or the SCO terminal 120. In one embodiment, the vision-based SCO assistant is the vision-based product recognition unit 113, the unrecognized product manager 114, the API 115, the transaction manager 123, the transaction UI 124, and / or the method 600. The vision-based SCO assistant presents a different, and in some respects enhanced, perspective on processing for the method 600 in Figure 6 from those described above.

[0058] In 710, the vision-based SCO assistant uses at least one camera 125 of the vision-based SCO terminal 120 to acquire image data of one or more items on the checkout tray. In one embodiment, in 711, the vision-based SCO assistant generates bounding boxes around each of the one or more items on the checkout tray within the image data from above.

[0059] In 720, the vision-based SCO assistant identifies at least one unrecognized item from one or more items. In one embodiment, in 721, the vision-based SCO assistant determines that the unrecognized item cannot be visually distinguished from other items that are similar to it.

[0060] In 730, the vision-based SCO assistant displays a transaction UI 124 for identifying unrecognized products. In one embodiment, in 731, the vision-based SCO assistant displays at least one candidate product based on the visual characteristics of the unrecognized product.

[0061] In step 740, the vision-based SCO assistant receives user input to identify unrecognized items as identified items. In step 750, the vision-based SCO assistant generates a dead zone in the overhead image data corresponding to the location of the identified items on the checkout tray.

[0062] In 760, the vision-based SCO assistant prevents further prompting for identification of identified items while the identified items remain in the dead zone or are removed from the dead zone. In one embodiment, in 761, the vision-based SCO assistant maintains the item identification data associated with the dead zone until the transaction is completed on the vision-based SCO terminal 120.

[0063] In one embodiment, in 770, the vision-based SCO assistant detects changes in the dead zone within the image data, analyzes the changes, and determines when a new product with different visual characteristics from the identified product is placed in the dead zone. When the vision-based SCO assistant detects from the image data that a new product is in the dead zone, it automatically clears the dead zone.

[0064] In one embodiment, in 780, the vision-based SCO assistant tracks the total number of items on the checkout tray and, based on changes in the total number of items, detects the removal of identified items from the checkout tray. The vision-based SCO assistant automatically proceeds with the transaction without requiring confirmation of item removal.

[0065] When software is described in a specific form (such as components or modules), this is merely to aid understanding and is not intended to limit how the software implementing those functions may be architectured or structured. For example, modules are illustrated as separate modules, but they may be implemented as uniform code, as individual components, as some (but not all) of these modules combined, or as software structured in any other convenient form.

[0066] Furthermore, although the software module is illustrated as running on a single piece of hardware, the software may be distributed across multiple processors or in any other convenient form.

[0067] The above description is illustrative and not limiting. By examining the above description, many other embodiments will become apparent to those skilled in the art. Therefore, the scope of each embodiment should be determined by reference to the appended claims and the entire scope of equivalents to be given thereto.

[0068] In the above description of the embodiments, various features are grouped into a single embodiment for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting that the above embodiments have more features than are explicitly described in each claim. Rather, as reflected in the following claims, the subject matter of the invention lies in fewer features than all the features of the single disclosed embodiment combined. Accordingly, the following claims are incorporated herein into the description of the embodiments, and each claim stands as an independent exemplary embodiment.

Claims

1. A method for recognizing unrecognized items in a self-checkout system, The vision-based self-checkout (SCO) terminal retrieves image data of at least one item on the tray. If it is determined that at least one of the aforementioned products is an unrecognized product, The identification information of the identified product associated with the aforementioned unrecognized product is received, A visual exclusion zone is created corresponding to the location of the identified product. Ignore subsequent detection of the identified product within the aforementioned visual exclusion zone. A method that includes the act of doing so.

2. The method according to claim 1, Acquiring the aforementioned image data is a method that includes acquiring the image data using a camera view from above the tray.

3. The method according to claim 1, A method for determining that at least one of the aforementioned products is an unrecognized product, which includes determining that the unrecognized product cannot be individually distinguished by the vision-based SCO terminal.

4. The method according to claim 1, A method for receiving the identification information includes receiving the selection of the identified product from a list of candidate products displayed on the user interface.

5. The method according to claim 1, Receiving the identification information includes a method of receiving scan data by scanning the barcode of the unrecognized product in order to obtain the identification information of the identified product.

6. The method according to claim 1, Creating the aforementioned visual exclusion zone is a method that includes defining a bounding box around the unrecognized product based on the image data.

7. The method according to claim 1, Ignoring the subsequent detections includes maintaining the visual exclusion zone until payment is initiated.

8. The method according to claim 1, Based on the aforementioned image data, it is determined that the visual exclusion zone is empty. Maintain the aforementioned visual exclusion zone. A way to further include that.

9. The method according to claim 1, A new product is detected in the aforementioned visual exclusion zone. The center point, size, and at least one aspect ratio of the new product are compared with the corresponding characteristics of the identified product. If the comparison indicates that the new product is different from the previously identified product, the visual exclusion zone is removed. A way to further include that.

10. The method according to claim 1, Based on the decrease in the number of items detected by the vision-based SCO terminal, it is determined that the identified item has been removed from the tray. Omit the reminder to check the bagged items. A way to further include that.

11. The method according to claim 1, The visual exclusion zone is maintained if the center point, size, and aspect ratio of a specific product within the visual exclusion zone do not deviate significantly from the corresponding characteristics of the identified product. A way to further include that.

12. A method for recognizing unrecognized items in a self-checkout system, Using at least one camera of the vision-based SCO terminal, image data is acquired from above one or more items on the checkout tray. Identify at least one unrecognized product from the one or more products mentioned above, A user interface for identifying at least one unrecognized product is displayed. Upon receiving user input that identifies at least one of the aforementioned unrecognized products as an identified product, A dead zone corresponding to the location of the previously identified product is generated. While the identified product remains within the dead zone, the system prevents further requests for identification of the identified product. A method that includes the act of doing so.

13. The method according to claim 12, Acquiring the above-mentioned image data generates bounding boxes around each of the one or more items on the checkout tray within the above-mentioned image data. A method that includes doing so.

14. The method according to claim 12, Identifying the aforementioned at least one unrecognized product means determining that the aforementioned at least one unrecognized product cannot be visually distinguished from other products that are similar to the aforementioned at least one unrecognized product. A method that includes the act of doing so.

15. A method according to claim 12, Displaying the user interface means displaying at least one candidate product based on the visual characteristics of at least one unrecognized product. A method that includes the act of doing so.

16. The method according to claim 12, To prevent the aforementioned repeated reminders, maintain the product identification data associated with the dead zone until the transaction is completed. A method that includes the act of doing so.

17. The method according to claim 12, The change in the dead zone is detected, By analyzing the aforementioned changes, the timing at which a new product with different visual characteristics is placed in the dead zone is determined. When the aforementioned new product is detected, the dead zone is automatically cleared. A method that further includes the following.

18. The method according to claim 12, Track the total number of items on the aforementioned checkout tray, Based on the change in the total number of the aforementioned items, the removal of the specified items from the checkout tray is detected. The transaction proceeds automatically without requiring confirmation of the item's removal. A way to further include that.

19. A self-checkout system that recognizes unrecognized items, At least one camera configured to acquire image data of at least one item on the checkout tray, At least one processor, At least one memory to store instructions, Equipped with, The aforementioned instruction is, When executed by the at least one processor, the system a) Analyze the image data to identify at least one unrecognized item on the checkout tray, b) Receiving identification information of identified products associated with at least one unrecognized product, c) Create an exclusion area corresponding to the location of the identified product in the image data, d) Compare the subsequent image data of the excluded area with the stored characteristics of the identified product. e) If the comparison indicates that the identified product remains within the exclusion area, maintain the exclusion area. A system that executes each process.

20. A self-checkout system according to claim 19, When the instruction is executed by the at least one processor, the system f) Detect the placement of a new product in the exclusion area, g) Determine that the new product has at least one of the following characteristics compared to the previously identified product: a different center point, a different size, or a different aspect ratio. h) To enable the identification of the new product, the exclusion area is removed. A system that further executes each of these processes.

Images