Self-checkout equipment

Abstract

A self-checkout apparatus is disclosed. The automatic checkout device includes a checkout panel adapted to allow placement of a product thereon, and one or more cameras positioned to have a field of view surrounding the checkout panel, the one or more cameras configured to provide a video recording. The self-checkout device also includes a motion detection module configured to detect the presence of motion in the video recording; a sequence selection module configured to select a sequence of video frames within a time interval corresponding to detecting the presence of motion in the video recording; an appearance interpretation module configured to register one or more products present in the sequence of video frames; a billing module configured to acquire prices of the registered one or more products, generate a total bill based on the acquired prices, and process payment of the total bill; and a controller module that facilitates communication between the modules.

Description

Technical Field

[0001] The present disclosure generally relates to a self-checkout (SCO) device that allows a customer to easily check in one or more products by placing the products on the SCO device, where the products are individually identified, their individual prices are determined, and the total amount payable for the products is calculated. Background Art

[0002] In a traditional retail environment, a customer selects various items to purchase and brings those items to a store clerk for checkout. Over the past few decades, retail point-of-sale systems have been greatly automated to speed up the checkout process. Currently, computer-based point-of-sale systems are the standard in retail environments. However, at the end of the day, such point-of-sale systems are still operated by store clerks. The man-hours generated by servicing checkout counters contribute significantly to the overall expenses of a retail store. Reducing or eliminating the amount of time a store clerk spends handling and scanning items purchased by a customer can significantly reduce the labor required in a retail store, thereby alleviating the current growing problem.

[0003] To reduce operating expenses, some merchants have implemented self-service checkout systems that replace store clerks at each checkout terminal. A self-service checkout system is a terminal that is operated by the customer themselves without the direct assistance of a store clerk. A self-service checkout system typically includes a barcode (RFID or other identifier) ​​reader (also called a scanner); a weighing scale for weighing items such as fruits and vegetables; and an interactive screen for selecting products from a predefined list or entering product codes for products that do not have scannable identifiers (e.g., fresh produce such as fruits, vegetables, meats, bakery items, etc.). The self-service checkout system also typically includes a payment system that typically accepts cash and card transactions (or other contactless payment mechanisms).

[0004] In the case of a fixed self-checkout system, customers bring the products they wish to purchase to a fixed point in the store. The customer then presents the products to the self-checkout system and causes the self-checkout system to register, thereby forming a record of the products presented. Specifically, the customer presents each product individually to the self-checkout system by scanning each product individually with a self-checkout scanner (or the self-checkout system's scanning gun, if available), which is used to detect and interpret the identifier (i.e., barcode, RFID tag, etc.) present on the product. The self-checkout system then consolidates the details of the registered items, calculates the total cost, and facilitates the customer's payment process.

[0005] That is, existing self-checkout systems usually still require a high degree of intervention from store clerks. In addition, existing self-checkout systems have various problems, such as poor user interface, inability to handle multiple items at a time, inability to guide customers to place items on them, etc. For example, in a high-throughput sales environment, such as a fast lane or lunch area or grab-and-go area in a convenience store or grocery store, customers are usually in a hurry and need to quickly register and pay for their products. However, these same customers may present multiple products to be registered by the self-checkout system. The necessity of registering each of these products separately introduces delays in sales transactions. If a customer needs to register a product that does not have a scannable identifier, such as an item whose price depends on its weight, such as a bunch of bananas or a pile of lunch bowls, these delays are further extended. In order to register the products presented in this way, it may be necessary for the customer to manually search one or more product lists using the touch screen component of the self-checkout system to find and select matching products. Alternatively, the customer can use the touch screen component to manually enter the product code of the product presented in this way. In either case, the process of registering such a product may be very slow and cumbersome. These delays are a significant inconvenience and can be a deterrent to customers who are short on time and want to quickly pay for their purchases and leave.

[0006] In view of these considerations, the present disclosure has been made, and an object of the present disclosure is to provide a self-checkout device that creates a fast, simple, innovative experience for shoppers in the express lanes (e.g., lunch area or grab and go area) of a convenience store or grocery store by reducing the delays and inconveniences in high-throughput sales environments caused by the necessity to register each of multiple products in a sales transaction. Summary of the invention

[0007] In one aspect of the present disclosure, a self-service checkout device is disclosed. The self-service checkout device includes: a detection plate, the detection plate is suitable for allowing a product to be placed on the detection plate. The self-service checkout device also includes one or more cameras, the one or more cameras are positioned to have a field of view that at least surrounds the detection plate, and the one or more cameras are configured to provide video recording. The self-service checkout device also includes a motion detection module, the motion detection module is configured to detect the presence of motion in the video recording; a sequence selection module, the sequence selection module is configured to select a video frame sequence within a time interval corresponding to the detection of the presence of motion in the video recording; an appearance interpretation module, the appearance interpretation module is configured to register one or more products present in the video frame sequence; a billing module, the billing module is configured to obtain the price of the registered one or more products, generate a total bill based on the obtained price, and process the payment of the total bill; and a controller module, the controller module is operably connected to the one or more cameras and is communicatively coupled to the motion detection module, the sequence selection module, the appearance interpretation module, and the billing module to control the operation of these modules and facilitate communication between these modules.

[0008] In one or more embodiments, the appearance interpretation module includes: an object detection module configured to analyze a video frame sequence to detect one or more objects in the video frame sequence; a cropping module configured to isolate one or more objects in the detected video frame sequence and extract visual features of the detected one or more objects; an embedding module configured to convert the extracted visual features of the detected one or more objects into an embedding feature vector; and an expert system module configured to compare the embedded feature vector with a feature vector pre-stored in an embedding database, and identify the detected one or more objects based on the comparison. In this article, the identified one or more objects are registered as one or more products.

[0009] In one or more embodiments, the expert system module is further configured to determine whether any of the identified one or more objects is a weight-related bulk product item from the one or more products.

[0010] In one or more embodiments, the automatic checkout device further comprises a weighing module configured to activate a weighing scale unit to measure the weight of a weight-related bulk product item from one or more products placed on the detection plate. In this context, the billing module is configured to generate a total bill based on the measured weight of the weight-related bulk product item.

[0011] In one or more embodiments, the self-checkout device further comprises a barcode processing module configured to detect one or more barcodes in the selected video frame sequence and decode the detected barcodes corresponding to the one or more registered products. In this article, the billing module is configured to obtain the price of the one or more registered products based on the decoded barcodes.

[0012] In one or more embodiments, the self-service checkout device further comprises a guidance module operably connected to the design display unit. The guidance module is configured to activate the design display unit to display the design on the inspection board, thereby providing visual guidance to the user to optimally place the product on the inspection board.

[0013] In one or more embodiments, the automatic checkout device further comprises a concave fixing member arranged upright relative to the detection plate. In this context, one or more cameras are mounted on the concave fixing member.

[0014] In one or more embodiments, the concave fixing member accommodates a lighting device to illuminate the inspection plate.

[0015] In one or more embodiments, the one or more cameras include a first camera and a second camera, and the first camera and the second camera are oriented at different angles to capture video footage of the product from multiple perspectives.

[0016] In one or more embodiments, the billing module is further configured to generate a detailed list based on the registered one or more products.

[0017] In one or more embodiments, the self-checkout device further includes a display screen configured to display the itemized list and the total bill.

[0018] In one or more embodiments, the self-service checkout device further includes a management module configured to support updates to the configuration of the self-service checkout device, including updates to a product database of the self-service checkout device.

[0019] In one or more embodiments, the appearance interpretation module employs a machine learning model to facilitate the detection, cropping, embedding, and recognition processes.

[0020] In one or more embodiments, the self-service checkout device operates as a stand-alone device.

[0021] In another aspect, a method for real-time processing of a self-service checkout device is disclosed. The method includes: receiving a video recording of a detection plate of the self-service checkout device from one or more cameras. The method also includes detecting the presence of motion in the video recording by processing the video recording. The method also includes selecting a sequence of video frames within a time interval corresponding to the detection of the presence of motion in the video recording. The method also includes detecting and decoding one or more bar codes visible in the sequence of video frames. The method also includes calculating a total bill corresponding to the decoded one or more bar codes. The method also includes displaying the total bill on a display screen of the self-service checkout device.

[0022] In one or more embodiments, the method further includes detecting items visible in the sequence of video frames when one or more barcodes are not visible in the sequence of video frames. The method further includes distinguishing between sale items and non-sale items among the detected items. The method further includes issuing a first reminder when one or more non-sale items are detected, the first reminder including a message to remove the non-sale items placed on the detection plate of the self-service checkout device.

[0023] In one or more embodiments, the method further includes determining a distribution of the detected sales items on a detection plate of the self-service checkout device. The method further includes issuing a second reminder when it is detected that the determined distribution of the detected sales items is not appropriate.

[0024] In one or more embodiments, the method further includes cropping one or more regions substantially surrounding each detected sale item from each of the plurality of video frames. The method further includes generating an embedded representation of the sale item visible in the region based on each of the cropped one or more regions. The method further includes comparing the generated embedded representation with a record of embedded representations of products to find a record of embedded representations of matching products. The method further includes determining a price corresponding to the record of embedded representations of matching products. The method further includes calculating a total bill for all detected sale items as the sum of the determined prices corresponding to the records of embedded representations of matching products. The method further includes displaying the total bill on a display screen.

[0025] In one or more embodiments, the method further includes receiving payment of the master bill.

[0026] In another aspect, a computer program product is disclosed having machine-readable instructions stored therein, which when executed by one or more processing units cause the one or more processing units to perform the steps of the above method.

[0027] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] For a more complete understanding of example embodiments of the present disclosure, reference is made to the following description taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A schematic perspective view showing a self-service checkout device according to one or more exemplary embodiments of the present disclosure;

[0030] Figure 2 shows a schematic side plan view of a self-service checkout device according to one or more exemplary embodiments of the present disclosure;

[0031] Figure 3 A schematic top plan view of a self-service checkout device according to one or more exemplary embodiments of the present disclosure is shown;

[0032] Figure 4 An exploded view of a stand unit of a self-service checkout device according to one or more exemplary embodiments of the present disclosure is shown, the exploded view showing various components of the stand unit;

[0033] Figure 5 An exploded view of an interaction unit of a self-service checkout device according to one or more exemplary embodiments of the present disclosure is shown, the exploded view showing various components of the interaction unit;

[0034] Figure 6 A schematic block diagram of a self-service checkout device according to a first exemplary embodiment of the present disclosure is shown;

[0035] Figure 7 A schematic block diagram of a system having a plurality of self-service checkout devices according to a second exemplary embodiment of the present disclosure is shown;

[0036] Figure 8 A flow chart showing a method implemented by a self-service checkout device according to one or more exemplary embodiments of the present disclosure;

[0037] Fig. 9 A schematic perspective view showing a self-service checkout device according to an alternative embodiment of the present disclosure; and

[0038] Fig.10 An exemplary depiction of a self-checkout device implemented when a product is placed thereon is shown in accordance with one or more exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent to those skilled in the art that the present disclosure is not limited to these specific details.

[0040] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. In addition, the terms "one" and "an" herein do not indicate a limitation on quantity, but rather indicate the presence of at least one referenced item. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not of other embodiments.

[0041] Unless otherwise indicated in the following description, the terms "execution", "mathematical calculation", "computer-aided", "device calculation", "establishment", "generation", "configuration", "reconstruction" and the like preferably relate to operations and / or processes and / or processing steps which change and / or generate data and / or transform data into other data, wherein the data may in particular be represented or present in the form of physical variables, for example in the form of electrical impulses. The expression "computer" should in particular be interpreted as broadly as possible, so as to cover in particular all electronic devices with data processing properties. Thus, a computer may for example be a personal computer, a server, a programmable logic controller (PLC), a handheld computer system, a pocket computer device, a mobile radio device and other communication devices, processors and other electronic data processing devices which are capable of processing data in a computer-aided manner.

[0042] In addition, in particular, the (relevant) technical personnel in the field, knowing the method claim / multiple method claims, of course know all the conventional possibilities for realizing the product or the possibilities for implementation in the prior art, and therefore do not need to be specifically disclosed independently in the specification. In particular, these conventional implementation variants known to those skilled in the art can be implemented only by hardware (components) or only by software (components). Alternatively and / or additionally, those skilled in the art can choose any combination of hardware (components) and software (components) according to an embodiment of the present invention as much as possible within the scope of his / her professional ability, so as to realize the implementation variant according to an embodiment of the present invention.

[0043] The embodiments described herein may be discussed in the general context of computer-executable instructions residing on some form of computer-readable storage media, such as program modules, that are executed by one or more computers or other devices. By way of example and not limitation, computer-readable storage media may include non-transitory computer-readable storage media and communication media; non-transitory computer-readable media includes all computer-readable media except transient propagating signals. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of the program modules may be combined or distributed as desired.

[0044] Some parts of the following detailed description are presented and discussed in terms of a process or method. Although the steps and order of the method are disclosed in the drawings describing the operation of the method herein, these steps and order are exemplary. Embodiments are very suitable for performing various other steps or variations of the steps described in the flowcharts of the drawings herein, and for performing in a sequence different from that depicted and described herein. Some parts of the following detailed description are presented in terms of processes, logic blocks, processing, and other symbolic representations of data bit operations in computer memory. These descriptions and representations are the means by which technicians in the field of data processing use the most effective means to convey the essence of their work to other technicians in the field. In this application, processes, logic blocks, flows, etc. are considered to be a series of self-consistent steps or instructions that lead to a desired result. These steps are physical manipulations using physical quantities. Typically, although not necessarily, these physical quantities are presented in the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system. It turns out that it is convenient to sometimes refer to these signals as transactions, bits, values, elements, symbols, characters, samples, pixels, etc., mainly for general reasons.

[0045] In some embodiments, any suitable computer-available or computer-readable medium (or multiple media) can be utilized. Computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-available or computer-readable storage medium (including storage devices associated with computing devices) can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices, equipment or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable media can include the following: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, digital versatile disks (DVD), static random access memories (SRAM), memory sticks, floppy disks, etc., mechanical encoding devices (such as a punch card or a raised structure in a groove on which instructions are recorded), media (such as a medium supporting the Internet or an intranet), or magnetic storage devices. It is worth noting that a computer-usable or computer-readable medium may even be a suitable medium on which the program is stored, scanned, compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory. In the context of the present disclosure, a computer-usable or computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in association with an instruction execution system, apparatus or device.

[0046] In some embodiments, a computer readable signal medium may include a propagated data signal in which a computer readable program code is included, for example, in baseband or as part of a carrier wave. In some embodiments, such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic waves, light waves, or any suitable combination thereof. In some embodiments, the computer readable program code may be transmitted using any suitable medium, including but not limited to the Internet, wired, fiber optic cable, RF, etc. In some embodiments, a computer readable signal medium may be any computer readable medium that is not a computer readable storage medium, and may communicate, propagate, or transmit a program for use by or in association with an instruction execution system, device, or apparatus.

[0047] In some embodiments, the computer program code for performing the operations of the present disclosure may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Java, Smalltalk, C++, etc.). Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and / or its affiliates. However, the computer program code for performing the operations of the present disclosure may also be written in a conventional programming language, such as the "C" programming language, PASCAL or similar programming languages, and scripting languages ​​such as JavaScript, PERL or Python. In this embodiment, the language used for training may be one of Python, Tensorflow, Bazel, C, C++. In addition, the decoder in the user device (as will be discussed) may use C, C++ or any processor-specific ISA. In addition, the assembly code inside C / C++ may be used for specific operations. In addition, ASR (automatic speech recognition) and G2P decoders and the entire user system may run in embedded Linux (any distribution), Android, iOS, Windows, etc. without any restrictions. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider via the Internet). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), other hardware accelerators, microcontroller units (MCUs), or programmable logic arrays (PLAs) can be executed by utilizing the state information of computer-readable program instructions to perform computer-readable program instructions / codes, thereby personalizing the electronic circuits to perform various aspects of the present disclosure.

[0048] In some embodiments, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the device (system), method and computer program product according to various embodiments of the present disclosure. Each frame in the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can represent a module, a fragment or a part of the code, which includes one or more executable computer program instructions for implementing the specified (multiple) logical functions / (multiple) actions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the computer program instructions executed by the processor of a computer or other programmable data processing device can be created to implement the one or more frames of the flowchart and / or block diagram or the ability of one or more functions / actions specified in its combination. It should be noted that in some embodiments, the functions mentioned in the (multiple) frame may not occur in the order shown in the figure. For example, the two frames shown in succession can actually be executed substantially simultaneously, or the frame can sometimes be executed in reverse order, depending on the functions involved.

[0049] In some embodiments, these computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions / actions specified in one or more boxes of the flowchart and / or block diagram or a combination thereof.

[0050] In some embodiments, the computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps (not necessarily in a specific order) are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions / actions (not necessarily in a specific order) specified in one or more boxes of the flowchart and / or block diagram or a combination thereof.

[0051] In addition, in the following detailed description of the present disclosure, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced without these specific details. In other cases, well-known methods, processes, components and circuits are not described in detail to avoid unnecessarily obscuring various aspects of the present disclosure.

[0052] Common Reference Figure 1-3, showing different schematic diagrams of a self-service checkout device 10 according to one or more exemplary embodiments of the present disclosure. The self-service checkout device 10 is a device for a high-throughput sales environment, and is configured to provide an interface that allows customers to pay for services or goods without direct employee assistance (unless required). In the case of using the self-service checkout device 10, the customer assumes the responsibility of registering the products they wish to purchase and then paying for them. For the purposes of this disclosure, the process of registering one or more products using the self-service checkout device 10 refers to the process of presenting each product to the self-service checkout device 10 during a transaction so that the self-service checkout device 10 forms a record of the presented products and calculates the total bill thereof. For further clarity, the process can be decomposed into a connected sequence of events, which is referred to as a scene (episode) hereinafter. For example, the process of a customer registering one of their selected products at the self-service checkout device 10 will be referred to as a "product registration scene" hereinafter; and the process of a customer paying for all registered products will be referred to as a "payment scene" hereinafter.

[0053] In addition, for the purposes of this disclosure, products that do not have a scannable identifier will be referred to as "loose products" hereinafter. For further clarity, bulk product items whose price depends on their weight will be referred to as "weight-dependent bulk product items" hereinafter. For example, a weight-dependent bulk product item may include a bunch of bananas or a stack of lunch bowls. Those skilled in the art will understand that the examples of bulk product items and weight-dependent bulk product items being referred to are provided for illustrative purposes only. In particular, those skilled in the art will understand that the present disclosure is in no way limited to the above examples. On the contrary, the present disclosure is operable to register any product item that does not have a scannable identifier and whose price may or may not depend on the weight of the product item. For consistency, the process of registering a bulk product item will be referred to as the "bulk product registration scenario" hereinafter.

[0054] As shown in the figure, the self-service checkout device 10 includes a support unit 12 and an interactive unit 14. The interactive unit 14 includes a detection board 32 and a display screen 34 (such as Figure 5 ). Figure 1-3 As shown, the bracket unit 12 is coupled to the interactive unit 14 in the self-service checkout device 10 of the present disclosure. In particular, Figure 2As better shown in the figure, the bracket unit 12 includes a concave fixing (mounting) member 16, and its bottom end 18 is mounted on the base member 20. As shown in the figure, the concave fixing member 16 is uprightly arranged relative to the interactive unit 14 (specifically, the detection board 32 therein). The base member 20 may include a bracket matching structure (not shown) for allowing the bracket unit 12 to couple with the interactive unit 14, as described in more detail later. In this example, either or both of the concave fixing member 16 and the base member 20 are made of metal or plastic material. The concave fixing member 16 and the base member 20 both have non-reflective surfaces. Preferably, the surfaces of the concave fixing member 16 and the base member 20 are light absorbing. Even more preferably, the surfaces of the concave fixing member 16 and the base member 20 are black.

[0055] refer to Figure 4 , shows an exploded view of the bracket unit 12 according to one or more exemplary embodiments of the present disclosure, the exploded view showing the various components of the bracket unit 12. As shown, the inner surface 22 of the concave fixing member 16 provides a recessed portion, which is configured to accommodate a lighting device (not shown). In one example, the lighting device may include a plurality of light emitting diodes (LEDs). Preferably, the lighting device includes a plurality of RGB (red, green, blue) LEDs to enhance the illumination of nearby components (e.g., the interaction unit 14, more specifically, the detection board 32 (e.g., Figure 3The bracket unit 12 also includes a light diffusing shell member 24. As shown, the light diffusing shell member 24 is concave in shape. Here, the curvature of the light diffusing shell member 24 substantially matches the curvature of the concave fixed member 16. In one example, the light diffusing shell member 24 is made of any one of polycarbonate, acrylic or polymethyl methacrylate (PMMA), polystyrene or other suitable plastic materials. The light diffusing shell member 24 is preferably white and may be provided with an opal or translucent surface finish. Those skilled in the art will recognize that the above-mentioned materials and surface finishes provided for the light diffusing shell member 24 are for illustrative purposes only. In particular, those skilled in the art will recognize that the automatic checkout device 10 of the present disclosure is by no means limited to the use of these materials or surface finishes for the light diffusing shell member 24. Instead, the present disclosure may be implemented with any material or surface finish that allows for diffusion of light from the lighting device. In use, the light diffusing housing member 24 is mounted on the inner surface 22 of the concave fixing member 16 such that the light diffusing housing member 24 effectively forms a cover over the inner surface 22 of the concave fixing member 16. In this manner, the lighting device is sandwiched between the inner surface 22 of the concave fixing member 16 and the light diffusing housing member 24. Thus, the lighting device effectively forms a backlight member for the light diffusing housing member 24.

[0056] In addition, as shown in the figure, the bracket unit 12 includes one or more cameras. Specifically, the one or more cameras include a first camera 26 and a second camera 28 that are oriented at different angles to capture video footage of the product from multiple perspectives. Typically, the cameras 26, 28 are positioned to have a field of view that surrounds at least the detection plate 32 of the interactive unit 14. In this configuration, the first camera 26 is mounted on the inner surface 22 of the concave fixed member 16 and is located at the upper end 27 of the concave fixed member 16. Specifically, the first camera 26 is mounted on the concave fixed member 16 in a downward-facing orientation so that its field of view surrounds the area below it and a top-down view of one or more objects contained in the area. Preferably, the first camera 26 is an RGB-D camera. For example, the first camera 26 may include a time-of-flight (TOF) sensor, a structured light sensor, or a stereo sensor. Those skilled in the art will recognize that the above examples are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the self-service checkout device of the present disclosure is not limited to the above examples. On the contrary, the present disclosure can be implemented with any one or more sensors whose output signals provide a three-dimensional representation of the observed scene. In addition, preferably, the first camera 26 has a resolution of 4K or equivalent. Moreover, preferably, the first camera 26 includes an effective autofocus function. In addition, the second camera 28 is mounted on the inner surface 22 of the concave fixing member 16 and is located at a height of approximately middle (high-way) between the upper end 27 and the bottom end 18 of the concave fixing member 16. In this case, the light diffusion housing member 24 is provided with a hole 30, which is arranged at a position that substantially matches the position of the second camera 28 when the light diffusion housing member 24 is mounted on the concave fixing member 16. Here, the hole 30 ensures that the field of view of the second camera 28 is not blocked by the light diffusion housing member 24. Preferably, the second camera 28 includes an RGB camera. Further preferably, the second camera 28 has a resolution of 4K or equivalent. In addition, preferably, the second camera 28 has an effective autofocus function.

[0057] Common Reference Figure 1-4, the base member 20 of the bracket unit 12 is mechanically coupled to the interactive unit 14. Specifically, the base member 20 can be attached to either side of the interactive unit 14 by a mutual cooperation structure (not shown), which includes a bracket unit cooperation structure (not shown) and a corresponding interactive unit cooperation structure (not shown), and the bracket unit cooperation structure and the corresponding interactive unit cooperation structure are respectively formed in the base member 20 and the one or more sides of the interactive unit 14. The mutual cooperation structure may include one or more of the following: a flange and groove arrangement between the base member 20 and the corresponding side of the interactive unit 14; or a narrow groove or other recess in any one or more sides of the interactive unit 14, which is configured to accommodate at least a portion of the corresponding side of the base member 20. Those skilled in the art will understand that the above-mentioned coupling device is provided for illustrative purposes only. In particular, those skilled in the art will understand that the present disclosure is not limited to the above-mentioned coupling mechanism. Rather, the present disclosure may be implemented with any mechanism for coupling the base member 20 to the interactive unit 14 that is sufficiently robust to maintain the base member 20 in a fixed position relative to the interactive unit 14 over extended periods of time and while subject to impacts and knocks from users of the self-service checkout device 10 and objects located on the interactive unit 14. It will also be appreciated that the coupling of the base member 20 to the interactive unit 14 need not be achieved mechanically. Rather, the base member 20 may be, for example, magnetically coupled to the interactive unit 14. The coupling mechanism is also configured such that when coupled to the interactive unit 14, the orientation of the base member 20 causes the arc of the concave fixed member 16 to bend inwardly toward the interactive unit 14. Herein, the interactive unit mating structure includes an interactive unit contact point (not shown) configured so that the coupling of the interactive unit 14 with the base member 20 causes the interactive unit contact point to contact a stand unit contact point (not shown) mounted on the outer surface of the base member 20 and located at the stand unit mating structure to support electrical and communication coupling between the cameras 26, 28 of the stand unit 12 and the control circuit (described later) of the base member 20. In addition, the cameras 26, 28 and the lighting device are communicatively coupled and electrically coupled to the stand unit contact point, respectively.

[0058] In another embodiment, the concave fixing member 16 may include two spaced apart substantially matching arched members (not shown) mounted in parallel in an upright position relative to the interactive unit 14. The arched members are intermittently joined by a plurality of crossbars (not shown) to provide support and structural reinforcement and stability to the two arched members. In this embodiment, the cameras 26, 28 are supported at the upper and mid-height of the crossbars, respectively.

[0059] Reference now Fig. 9 , a depiction of a self-service checkout device 10 according to an alternative embodiment of the present disclosure is shown. Fig. 9As shown, the rear side of the self-service checkout device 10 (as shown in FIG. Figure 1-3 In contrast to the above (shown and described), the self-service checkout device 10 may have a bracket unit 12 mounted on a lateral side (herein, the left side) of the interaction unit 14. In yet another embodiment, two bracket units 12 (not shown) may be provided, which may be mechanically coupled to the interaction unit 14. Specifically, the base member 20 of each of the two bracket units 12 may be attached to any two of the one or more side surfaces of the interaction unit 14 by one or more mutually mating structures, the mutually mating structures including a bracket unit mating structure and a corresponding interaction unit mating structure of each bracket unit formed in any two of the one or more side surfaces of the base member 20 and the interaction unit 14, respectively. As described above, the interaction unit mating structure includes an interaction unit contact point, in which case the coupling of the interaction unit 14 with the two base members 20 causes the interaction unit contact point to contact each bracket unit contact point to support electrical coupling and communication coupling between the cameras of the two bracket units 12 and the control circuits (described later) of the two base members 20. This embodiment is particularly useful in difficult environmental conditions (lighting, glare, extraneous elements, etc.) because it doubles the number of cameras of the self-service checkout device 10 and increases the amount and variety of lighting for the self-service checkout device 10. This improves the reliability of detecting products placed on the self-service checkout device 10 by at least 10%.

[0060] Figure 5 FIG. 1 shows an exploded view of the interaction unit 14 according to one or more exemplary embodiments of the present disclosure, the exploded view showing various components of the interaction unit 14. Figure 5 , combined with Figure 1-4 As shown in the figure, the interactive unit 14 includes a detection board 32 and a display screen 34. In this article, the detection board 32 is suitable for allowing products to be placed thereon. In one example, the upper surface of the detection board 32 is configured to have a square shape with a size of 39cm×39cm. However, those skilled in the art will recognize that the above configuration of the upper surface of the detection board 32 is provided only for the purpose of illustration. In particular, those skilled in the art will recognize that the automatic checkout device 10 of the present disclosure is not limited to the above configuration of the upper surface of the detection board 32. On the contrary, the present disclosure can be implemented with any size or shape of the upper surface of the detection board 32, which is sufficient to accommodate the placement of multiple products (not shown) thereon without stacking them on top of each other or crowding them together so that the field of view of the cameras 26, 28 used to identify the characteristics of a product will not be blocked by other products. For example, without departing from the spirit and scope of the present disclosure, the detection board 32 can have a larger size, such as 50cm×50cm, to accommodate more products thereon.

[0061] In one embodiment, the detection plate 32 may be provided with a backlight. The backlight of the detection plate 32 is configured to eliminate reflections and shadows, which may otherwise cause or contribute to erroneous results from the self-service checkout device 10. In one example, the backlight of the detection plate 32 is implemented by a plurality of backlight elements (not shown). The backlight elements are spatially distributed on the horizontal plane of the detection plate 32, thereby providing lighting coverage on substantially the entire range of the upper surface of the detection plate 32. In addition, in some examples, the individual backlight elements may be individually activatable to allow configurable and variable lighting patterns to be provided to different areas of the detection plate 32. In particular, the area of ​​the upper surface of the detection plate 32 where the backlight element is not activated may appear darker in color than the remaining areas of the upper surface of the detection plate 32. Therefore, different designs can be effectively displayed on the upper surface of the detection plate 32 by the controllable activation of the individual backlight elements. These designs may be configured to provide guidance to the user on the position where the product is placed on the detection plate 32, so as to facilitate detection thereof by the self-service checkout device 10. In another example, the backlight of the detection board 32 may not be implemented by a plurality of individually activatable backlight elements; however, alternatively, the backlight of the detection board 32 may be implemented by one or more backlight elements, the operation of which is synchronized so that one or all of the backlight elements are activated or deactivated at the same time. In yet another example, the design may be displayed on the detection board 32 by a projection device (not shown) mounted on the bracket unit 12. For simplicity, the individually activatable backlight element and the projection unit will be referred to as a "design display unit" hereinafter.

[0062] In addition, as shown, the display screen 34 includes an upper surface 36 that is operable to display messages and information including a list of products being processed in a transaction to a user or a store clerk. The display screen 34 may be a touch screen that is configured to detect user touches on the upper surface 36 of the display screen 34 and the locations of those touches.

[0063] In addition, as shown, the interactive unit 14 includes a 1D barcode reader 38, a contactless card reader 40, and a multi-function button 42. The 1D barcode reader 38, the contactless card reader 40, and the multi-function button 42 are operable to provide the user with an additional mechanism for interacting with the self-service checkout device 10 of the present disclosure. Therefore, for the sake of brevity, the 1D barcode reader 38, the contactless card reader 40, and the multi-function button 42 will be collectively referred to as "additional user interface elements" hereinafter. The 1D barcode reader 38 can be configured to detect, read, and decode a barcode presented to the 1D barcode reader 38, and in response, output a detected barcode signal containing information related to the decoded barcode. The contactless card reader 40 can be configured to transmit a payment instruction to the presented payment card, receive payment details from the payment card, and output the payment details. The multi-function button 42 can be configured to output a signal corresponding to a detected press on it. For the sake of brevity, the signal output from each member of the additional user interface elements will be referred to hereinafter as an "additional user interface signal".

[0064] In one example, the multifunction button 42 can be omitted from the "additional user interface elements", and the user's interaction with the self-service checkout device 10 can be replaced by the detection of the user's finger movement relative to the display screen 34 via an IR device (not shown) or the camera 26, 28 of the bracket unit 12, thereby converting the display screen 34 into a basic touch screen device. Specifically, there is no need to press a button to advance the operation of the self-service checkout device 10 (for example, to remove a batch of products from the detection board 32 and load a batch of products to the detection board 32, or to advance to the payment process, or to access management features). Instead, well-defined gestures are used to switch back and forth between the process or change the stage or management features of the transaction. In this case, the display screen 34 can be configured to display the allowed gestures to the user. The gestures must be easy to distinguish, easy to detect, and in the case of errors, can be canceled by the user. By eliminating mechanical components such as the multifunction button 42, the durability of the self-service checkout device 10 will be increased and the maintenance requirements of the self-service checkout device 10 will be reduced. Those skilled in the art will recognize that the above-mentioned members of the additional user interface elements are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the self-service checkout device 10 of the present disclosure is not limited to the above-mentioned members of additional user interface elements. Rather, the present disclosure may be implemented with any other mechanism that allows a user to interact with the self-service checkout device 10. For example, a preferred embodiment may include a speaker and microphone system that is configured to play predefined messages to the user, and detect and receive speech from the user.

[0065] The interactive unit 14 also includes an open shell member 44. The shell member 44 is configured to accommodate and position the display screen 34 and the detection board 32 in a substantially side-by-side arrangement, so that the upper surface 36 of the display screen 34 and the detection board 32 are each exposed through the open end of the shell member 44. The shell member 44 is also configured to accommodate additional user interface elements in a manner that allows a user to access each of the additional user interface elements through the open end of the shell member 44. For example, the additional user interface elements can be arranged in a substantially side-by-side arrangement with the detection board 32 and the display screen 34, so that from the left hand side of the shell member 44 to its right hand side (such as Figure 3 ), the display screen 34 is effectively sandwiched between the detection board 32 and the additional user interface elements. Alternatively, the members of the additional user interface elements can be arranged around the periphery of the display screen 34 and / or the detection board 32 in other ways. In addition, those skilled in the art will recognize that the relative spatial order of the display screen 34 and the detection board 32 can be reversed, so that from the left-hand side of the housing member 44 to the right-hand side thereof, the detection board 32 is effectively sandwiched between the display screen 34 and the additional user interface elements.

[0066] In an alternative embodiment, in order to reduce the footprint of the interactive unit 14, in order to enhance its practicality in a narrow space, the additional user interface elements may be formed in the concave fixing member 16 of the bracket unit 12, rather than in the interactive unit 14. In addition, those portions of the interactive unit 14 that are not part of the detection plate 32 may be reduced by: moving the contactless card reader 40 closer to the display screen 34; omitting the 1D barcode reader 38 from the additional user interface elements and relying on the second camera 28 of the bracket unit 12 to read the barcode of the presented product. In doing so, the footprint of the self-service checkout device 10 may be reduced to further enhance its practicality in space-constrained environments. Similarly, the user's experience of operating the self-service checkout device 10 may be enhanced by providing a simplified user interface having fewer options for interacting with it.

[0067] The interactive unit 14 also includes a transparent protective plate member 46. The protective plate member 46 is configured to have a size that substantially matches the size of the open end of the housing member 44. The protective plate member 46 is also configured to be mounted on the open end of the housing member 44 to form a substantially waterproof seal with the housing member 44, so that the display screen 34, the detection board 32 and the additional user interface elements are effectively sandwiched between the protective plate member 46 and the housing member 44. In one embodiment, the protective plate member 46 is made of scratch-resistant glass or tempered glass. In another embodiment, the protective plate member 46 is made of a transparent impact-resistant plastic material, such as transparent polycarbonate, transparent acrylic, transparent polyethylene terephthalate glycol (PETG) or transparent polyvinyl chloride (PVC). In one example, the protective plate member 46 is coated with an opaque pigment or covered with an opaque adhesive foil. Preferably, either or both of the opaque pigment and the opaque adhesive foil are black. In this article, the opaque pigment or the opaque adhesive foil is not present in the observation area 48 of the protective plate member 46. As shown, the viewing area 48 is arranged to align with the display screen 34 and the detection plate 32 when the protection plate member 46 is installed on the open end of the shell member 44. In addition, the protection plate member 46 includes a plurality of cutout areas 50, which are arranged to align with the additional user interface elements when the protection plate member 46 is installed on the open end of the shell member 44. In this way, the user is presented with an unobstructed clear view of the display screen 34 and the detection plate 32, but the display screen 34 and the detection plate 32 are protected by the protection plate member 46 from impact, scratching and accidental spillage. In addition, the user is presented with unobstructed access to each member of the additional user interface elements.

[0068] In the present embodiment, the interactive unit 14 is configured to have height and width dimensions in a 16:9 ratio. Preferably, the interactive unit 14 is configured to have a 32-inch diagonal. However, those skilled in the art will recognize that the above dimensions and the relationships between them are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the self-service checkout device 10 of the present disclosure is by no means limited to these dimensions and the relationships between them. On the contrary, the present disclosure may be implemented with any physical dimensions and / or relationships between them that are sufficient to accommodate multiple products of different sizes and shapes simultaneously without crowding or stacking them; and allow for convenient integration and assembly into the pre-existing environment of a high-throughput sales environment while minimizing interference with customers, staff, and existing processes and systems operating in that environment.

[0069] In particular, in use, the mutual cooperation structure of the base member 20 and the interactive unit 14 is configured to arrange the concave fixed member 16 in an overhanging arrangement aligned with the detection plate 32. Specifically, the mutual cooperation structure is configured to arrange the first camera 26 of the concave fixed member 16 substantially directly above the detection plate 32, so that the first camera 26 has a top-down view of the detection plate 32 and the products placed thereon. In one example, the concave fixed member 16 is configured so that the first camera 26 is arranged at a height of 60cm from the detection plate 32. However, those skilled in the art will recognize that the above heights are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the self-service checkout device 10 of the present disclosure is not limited to the above heights. On the contrary, the concave fixed member 16 can be configured to have any height that is suitable for providing sufficient height for the first camera 26 so that the first camera 26 can have a clear and unobstructed top-down view of even the highest products that may be placed on the detection plate 32. In addition, the curvature of the concave fixing member 16 is configured to maximize its stability and minimize the obstruction of the concave fixing member 16 to the area in front of it, so as to allow the user to access the inspection plate 32 and allow the user to conveniently place the product on the inspection plate 32. The curvature of the concave fixing member 16 and the arrangement of the second camera 28 are also configured to provide the second camera 28 with a wide field of view, thereby forming a side view of the inspection plate 32 and the product placed thereon.

[0070] The interactive unit 14 also includes a control circuit (not shown). In this article, each member of the test board 32, the display screen 34 and the additional user interface element is communicatively coupled and electrically coupled to the control circuit. The control circuit is also communicatively coupled and electrically coupled to the interactive unit contact point. The housing member 44 is configured to accommodate the control circuit (not shown) in a manner arranged side by side with at least one of the display screen 34, the test board 32 and the additional user interface element. Alternatively, the housing member 44 can be configured with a plurality of vertically separated inner grooves, which are suitable for accommodating the control circuit (not shown) in a manner of sandwiching an arrangement between the bottom of the housing member 44 and the lower surface of either or both of the display screen 34 and the test board 32. In some examples, the housing member 44 is also provided with a plurality of vents to allow heat to escape from the display screen 34, thereby preventing it and / or the control circuit from overheating. In this example, the housing member 44 can be made of any sturdy, lightweight and waterproof plastic or rubber material suitable for withstanding daily wear and tear and preventing liquid from overflowing to the display screen 34. For sanitation, the housing member 44 should also be easily cleanable.

[0071] In one or more embodiments, the interaction unit 14 of the automatic checkout device 10 further includes a weighing scale unit (not shown) communicatively coupled to the control circuit. The weighing scale unit (not shown) can be configured to be accommodated in the housing member 44 in a sandwich arrangement between the bottom of the housing member 44 and the lower surface (not shown) of the detection plate 32 to allow the product placed on the detection plate 32 to be weighed.

[0072] The control circuit includes an LED receiver (not shown) that is communicatively coupled to the lighting device through the interactive unit contact points and the bracket unit contact points. The LED receiver can be configured to receive control signals from a controller (not shown), wherein the controller (not shown) can be located remotely from the LED receiver and can be wirelessly coupled to the LEDs of the lighting device through the LED receiver to control the color and intensity of the LEDs and turn one or more of the LEDs on or off as needed. The control circuit can also include a microprocessor (not shown) that is configured to receive video recordings from the cameras 26, 28 and additional user interface signals from additional user interface elements. The microprocessor can also be configured to receive a signal indicating the weight of the product placed on the detection plate 32 from a weighing scale unit (if present). The microprocessor can also be configured to process the received video recordings and additional user interface signals, as well as the signals from the weighing scale unit; and based on the processing, send control signals to at least one of the detection plate 32, the display screen 34, and the contactless card reader 40. The specific functions of the microprocessor in supporting the operation of the self-service checkout device 10 are discussed in more detail in the following paragraphs.

[0073] Reference now Figure 6 , shows a schematic block diagram of a self-service checkout device 10 according to a first exemplary embodiment of the present disclosure. In this document, in particular, Figure 6 The self-service checkout device 10 is shown as a stand-alone device. Figure 6 As shown, combined with the above paragraphs Figure 1-5, the self-service checkout device 10 includes a controller module 102 that is operably connected to a first camera 26 and a second camera 28 mounted on a concave fixed member 16 of the self-service checkout device 10 and is communicatively coupled to various modules / components in the self-service checkout device 10 (as discussed below). The self-service checkout device 10 also includes a motion detection module 104 that is configured to receive the video recording via the controller module 102 and detect the presence of motion in the video recording. The self-service checkout device 10 also includes a sequence selection module 106 that is configured to receive the video recording via the controller module 102 and select a sequence of video frames within a time interval corresponding to the detection of the presence of motion in the video recording. In other words, the sequence selection module 106 selects a sequence of video frames within a time interval that begins after the motion detection module 104 detects the motion and ends the detection of the motion (i.e., from the start of the detection of the presence of motion in the video recording to the end of the detection of the presence of motion in the video recording). The self-service checkout device 10 also includes a barcode processing module 108, which is configured to detect the presence of a barcode in the video frame sequence and decode the barcode. The self-service checkout device 10 also includes an appearance interpretation module 114, which is configured to detect, recognize and identify objects visible in the video frame sequence based on the appearance of the object, thereby registering one or more products present in the video frame sequence. The self-service checkout device 10 also includes a weighing module 126, which is configured to activate a weighing scale unit of the self-service checkout device 10 to measure the weight of the weight-related bulk product items. The self-service checkout device 10 also includes a billing module 128, which is configured to obtain the price of the one or more registered products, generate a detailed list based on the one or more registered products, and generate a total bill based on the obtained price of the one or more registered products, and process the payment of the total bill. The automatic checkout device 10 also includes a guide module 130, which is operably connected to the design display unit and displays the design on the detection board 32 to provide the user with visual guidance for optimal placement of the product on the detection board 32. In this article, the controller module 102 is communicatively coupled with the motion detection module 104, the sequence selection module 106, the barcode processing module 108, the appearance interpretation module 114, the weighing module 126, the billing module 128 and the guide module 130 to control their operations and facilitate communication between them. The self-service checkout device 10 also includes a management module 132 communicatively coupled with the controller module 102, which is used to support the update of the configuration of the software of the self-service checkout device 10 (including its product database 110), and reset the software of the self-service checkout device 10 as needed. Each of these modules and the relationship between them are described in more detail in the following paragraphs.

[0074] Motion detection module 104

[0075] In a high throughput sales environment, there may be significant movement of hands and products in the area near the self-service checkout device 10 and the detection plate 32 as products are placed on and removed from the detection plate 32. Such movement may make detection of the product being referred to more challenging. Therefore, to improve operational performance, monitoring of the detection plate 32 should occur only when a customer has completed placing a product on the detection plate 32.

[0076] In the present configuration, the motion detection module 104 is coupled to the first camera 26 located above the detection board 32 to obtain a bird's eye view of the detection board 32. The motion detection module 104 is also coupled to the second camera 28 located at a facade relative to the detection board 32 and the side of the detection board 32 to obtain a side view of the detection board 32. The motion detection module 104 is adapted to receive two video recording streams from each of the first camera 26 and the second camera 28, respectively. It is understood that the video recording from the video camera includes a plurality of continuously captured video frames, where p is the number of video frames in the captured video recording. Assuming that the video frames is captured by the video camera at a time point (also called sampling time) τ+iΔt, where τ is the time when the capture of the video recording begins, and Δt is the time interval between the capture of the first video frame and the capture of the next video frame (also called sampling interval). Using this notation, the video recording captured by the first camera 26 can be described as [Fr1(τ), Fr1(τ+Δt), Fr1(τ+2Δt)....Fr1(τ+pΔt)], where Fr1(τ+iΔt) is the video frame captured from the first camera 26 at the sampling time τ+iΔt. For simplicity, the video recording stream received from the first camera 26 will be referred to as the "first video stream" VID1 hereinafter. Similarly, the video recording captured by the second camera 28 can be described as Wherein Fr2(τ+iΔt) is the video frame captured from the second camera 28 at the sampling time τ+iΔt. For the sake of simplicity, the video recording stream received from the second camera 28 will be referred to as the "second video stream" VID2 hereinafter. In one embodiment, the video frames in the first video stream VID1 and the second video stream VID2 are encoded using the H.264 video compression standard. The H.264 video format uses motion vectors as a key element for compressing video recordings. The motion detection module 104 uses the motion vectors obtained from the decoding of the H.264 encoded video frames to detect motion in the first video stream VID1 and the second video stream VID2. In another embodiment, the first video stream VID1 and the second video stream VID2 are each sampled at a predefined interval. The sampling intervals of the first video stream VID1 and the second video stream VID2 are configured to have sufficient duration to avoid erroneously detecting small high-speed motions (such as finger movements) instead of larger motions corresponding to the placement or removal of the product on the inspection board 32.

[0077] Successive samples (Fr1(τ+qΔt), Fr1(τ+(q+1)Δt) of video frames from the first video stream VID1 are compared to detect differences therebetween. Similarly, consecutive samples (Fr2(τ+qΔt), Fr2(τ+(q+1)Δt)) of video frames from the second video stream VID2 are compared to detect differences therebetween. Differences exceeding a predefined threshold are considered to indicate motion occurring in the intervening period between consecutive samples. The threshold is configured to avoid instantaneous changes (such as light flicker) from being mistaken for motion. When the motion detection module 104 detects the start and completion of motion, the motion detection module 104 transmits a “motion trigger” signal to the controller module 102.

[0078] Sequence selection module 106

[0079] The sequence selection module 106 is communicatively coupled to the first camera 26 and the second camera 28 to receive the first video stream VID1 and the second video stream VID2 from the first camera 26 and the second camera 28. The sequence selection module 106 is also communicatively coupled to the controller module 102 to receive a motion trigger signal from the controller module 102. Upon receiving the motion trigger signal, the sequence selection module 106 is configured to extract a first video frame sequence and a second video frame sequence from the first video stream VID1 and the second video stream VID2, respectively. For the sake of brevity, the video frame sequence selected from the first video stream VID1 will be referred to as the first selected sequence VS1 hereinafter. Similarly, the video frame sequence selected from the second video stream VID2 will be referred to as the second selected sequence VS2 hereinafter. The first selected sequence VS1 and the second selected sequence VS2 include video frames that start when the motion trigger signal is issued and extend a predefined interval thereafter. For ease of understanding and consistent with the above symbols, the moment when the motion trigger signal is issued has been defined as t start =τ+q st Δt, where q st is the corresponding sampling interval at which the motion trigger signal is emitted. Similarly, the corresponding selected video sequence has been defined as end =τ+(q st +α)Δt ends, wherein the value of α is sufficient to allow the product placed on the detection plate 32 to be recognized.

[0080] Using this nomenclature, the first selected sequence VS1 can be described as VS1=(Fr1(τ+q st Δt),Fr1(τ+(q st +1)Δt)......Fr1(τ+(q st +α)Δt)). Similarly, the second selected sequence VS2 can be described as VS2=(Fr2(τ+q st Δt),Fr2(τ+(q st +1)Δt)……Fr2(τ+(q st In other words, the first selected sequence VS1 includes α consecutive sampled video frames captured from the first camera 26, and the second selected sequence VS2 includes α consecutive sampled video frames captured from the second camera 28. In addition, the start time (τ+q) of the first selected sequence VS1 and the second selected sequence VS2 is st Δt) and the end time (τ+(q st The sequence selection module 106 is communicatively coupled to the controller module 102 to transmit the first selected sequence VS1 and the second selected sequence VS2 to the controller module 102 .

[0081] Barcode processing module 108

[0082] The barcode processing module 108 is coupled to the controller module 102 to receive a first selected sequence VS1 and a second selected sequence VS2 from the controller module 102. The barcode processing module 108 includes a barcode analysis algorithm configured to receive video frames from either or both of the first selected sequence VS1 and the second selected sequence VS2. The barcode analysis algorithm is configured to detect the presence of a barcode in the received video frame; and decode any such detected barcode into a corresponding text expression, which, for the sake of brevity, is referred to as a "barcode ciphertext" hereinafter. In one embodiment, the barcode analysis algorithm includes a known "quick browser" model, such as described in T. Do and D. Kim, "Quick Browser: A Unified Model to Detect and Read Simple Object in Real-time", 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1-8. In another embodiment, the barcode analysis algorithm includes (a) a barcode detection, positioning, and rotation algorithm, such as described in Real-time barcode detection and Classification Using Deep Learning, DK Hansen, Nasrollahi K., Rasmussen CB, Moeslund TB, Proc 9 thInternational Joint Conference on Computational Intelligence, 2017; and (b) barcode decoding algorithms, such as the barcode decoding algorithms of the ZXing (“Zebra Crossing”) barcode scanning library; or deformable barcode digital models as described in O. Gallo and R Manduchi “Reading Challenging Barcodes with Cameras” Proceedings of the IEEE Workshop on Applications of Computer Vision, 2009 (7-8): 1-6. Those skilled in the art will understand that these algorithms are provided for illustrative purposes only. In particular, those skilled in the art will recognize that the present disclosure is not limited to the use of these barcode detection and decoding algorithms. Instead, the present disclosure may be implemented with any algorithm capable of detecting, locating, and decoding barcodes visible in a video frame captured by the first camera 26 or the second camera 28. For example, the barcode analysis algorithm may employ a single shot detector (SSD) algorithm to detect the presence of a barcode (as in Y. Ren and Z. Liu, “Barcode detection and decoding method based on deep learning”, 2 nd International Conference on Information Systems and Computer Aided Education (ICISCAE), 2019, 393-396).

[0083] The barcode processing module 108 is communicatively coupled to a product database 110 that is populated with a plurality of tuples that include a barcode ciphertext element for each product in the store's inventory and an identifier corresponding to each such product and a price for the product. At least 2,000 of the tuples also include an embedding vector, as will be discussed later in conjunction with the embedding module for each product / bulk product. Those skilled in the art will recognize that this number of tuples including embedding vectors is provided for illustrative purposes only. In particular, those skilled in the art will recognize that the self-service checkout device 10 of the present disclosure is not limited to this number of tuples in the product database 110 that include embedding vectors. Rather, the self-service checkout device 10 of the present disclosure may be implemented with any number of tuples in the product database 110 that include embedding vectors that is sufficient to allow identification of at least some of the products / bulk products in the store's inventory based on their appearance.

[0084] In the event that the barcode processing module 108 detects the presence of a barcode in the video frames of the first selected sequence VS1 or the second selected sequence VS2, and decodes any such detected barcode, the barcode processing module 108 is adapted to use the resulting barcode cryptogram to query the product database 110 to retrieve therefrom an identifier of the product corresponding to the barcode cryptogram and a price for the product. The barcode processing module 108 is further configured to send the identifier and the corresponding price to the controller module 102. In contrast, in the event that the barcode processing module 108 fails to detect the presence of a barcode in the video frames of the first selected sequence VS1 or the second selected sequence VS2, the barcode processing module 108 is communicatively coupled to the controller module 102 to send an "appearance activation" signal to the controller module 102.

[0085] Appearance interpretation module 114

[0086] The appearance interpretation module 114 includes an object detection module 116, which is configured to analyze the video frame sequence from the sequence selection module 106 to detect one or more objects therein. The appearance interpretation module 114 also includes a cropping module 118, which is configured to isolate one or more objects detected in the video frame sequence and extract visual features of the detected one or more objects. The appearance interpretation module 114 also includes an embedding module 120, which is configured to convert the extracted visual features of the detected one or more objects into an embedded feature vector. The appearance interpretation module 114 also includes an expert system module 122, which is configured to compare the embedded feature vector with the pre-stored feature vector in the embedding database 124, and identify the detected one or more objects based on the comparison. In this article, the identified one or more objects are registered as one or more products. In an embodiment of the present disclosure, the appearance interpretation module 114 uses a machine learning model to facilitate the detection, cropping, embedding and recognition processes.

[0087] Specifically, the appearance interpretation module 114 is communicatively coupled to the controller module 102 to receive the first selected sequence VS1 and the second selected sequence VS2; and the appearance activation signal from the controller module 102. The appearance interpretation module 114 is also communicatively coupled to the guidance module 130 as described later. Upon receiving the appearance activation signal, the appearance interpretation module 114 is adapted to send the first selected sequence VS1 and the second selected sequence VS2 to the object detection module 116.

[0088] Object detection module 116

[0089] Customers may approach the self-service checkout device 10 with products from the store and the customers' own personal belongings (e.g., handbags, shopping bags, wallets, mobile phones, etc.). When placing products on the detection plate, customers may accidentally allow their personal belongings to enter the field of view of either or both of the first camera 26 and the second camera 28. The goal of the object detection module 116 is to detect the presence of objects other than the detection plate in the received video frames; and determine whether the detected objects are products; and if the detected objects are all products, determine the location of the products.

[0090] For purposes of the present invention, the object detection module 116 implements an object detection algorithm that is configured to receive a video frame from the received first selected sequence VS1 and detect the presence of an object in the video frame. The object detection algorithm is also configured to classify the detected object as one of "sale product" or "other", where objects classified as "other" may include personal items of a customer. Similarly, the object detection algorithm is configured to receive a video frame from a second selected sequence VS2; detect the presence of an object in the video frame; and classify the detected object as one of "sale product" or "other". Therefore, for a given video frame Fr1(τ+iΔt) from the first selected sequence VS1 (where q st ≤i≤q st +α), the object detection module 116 is configured to output a first label vector in is the number of objects detected in the video frame Fr1(τ+iΔt) from the first selected sequence VS1; and is the label corresponding to the category of the jth detected object. Similarly, for a given video frame Fr2(τ+iΔt) from the second selected sequence VS2 (where q st ≤i≤q st +α), the object detection module 116 is configured to output a second label vector in is the number of objects detected in the video frame Fr2(τ+iΔt) from the second selected sequence VS2; and is the label corresponding to the category of the jth detected object.

[0091] The object detection algorithm is further configured to determine the coordinates of a bounding box arranged to enclose the object detected in the video frame. The coordinates of the bounding box are established relative to the coordinate system of the received video frames of the first selected sequence VS1 or the second selected sequence VS2, as the case may be. Specifically, for a given video frame Fr1(τ+iΔt) from the first selected sequence VS1 (where q st ≤i≤qst +α), the object detection algorithm is configured to output a set of bounding boxes One or more details, where N obj (τ+iΔt) is the number of objects detected in the video frame Fr1(τ+iΔt), and is the bounding box surrounding the jth detected product. Similarly, for a given video frame Fr2(τ+iΔt) from the second selected sequence VS2 (where q st ≤i≤q st +α), the object detection algorithm is configured to output a set of bounding boxes One or more details, where N obj (τ+iΔt) is the number of objects detected in the video frame Fr2(τ+iΔt), and is the bounding box surrounding the jth detected object. Each bounding box The details include four variables, namely [x, y], h and w, where [x, y] is the coordinate of the upper left corner of the bounding box relative to the upper left corner of the video frame (Fr1(τ+iΔt), Fr2(τ+iΔt)) (whose coordinates are [0, 0]); h and w are the height and width of the bounding box, respectively.

[0092] The number of objects detected in a video frame exceeds six (i.e., N obj (τ+iΔt)>6), the object detection module 116 is suitable for sending an "excessive object reminder" signal to the controller module 102. Those skilled in the art will understand that the above-mentioned number of objects detected in the video frame to cause the issuance of an excessive object reminder signal is provided for illustrative purposes only. In particular, those skilled in the art will understand that the self-service checkout device 10 of the present disclosure is not limited to the number of objects detected in the video frame to cause the issuance of an excessive object reminder signal. On the contrary, the self-service checkout device 10 of the present disclosure can be implemented with any number of objects detected in the video frame to cause the issuance of an excessive object reminder signal, which meets the requirements of maximizing the possibility of successfully identifying the product while maximizing the number of such products that can be simultaneously identified in this manner.

[0093] In one or more embodiments, the object detection algorithm implements a deep neural network whose architecture is substantially based on EfficientDet (as described in M. Tan, R. Pang, and QV Le, EfficientDet: Scalable and EfficientObject Detection, 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 10778-10787). The architecture of the deep neural network can also be based on YOLOv4 (as described in A Bochkovskiy, CY Wang, and HY M Liao, 2020arXiv:2004.10934). However, those skilled in the art will understand that these object detection deep neural network architectures are provided for exemplary purposes only. In particular, those skilled in the art will understand that the self-service checkout device 10 of the present invention is not limited to these deep neural network architectures. Instead, the self-service checkout device 10 can be implemented with any object detection architecture and / or training algorithm suitable for detecting, classifying, and locating products in an image or video frame.

[0094] It will be appreciated that the goal of training the deep neural network is to enable it to establish an internal representation of the product / bulk product, wherein the internal representation allows the deep neural network to recognize the presence of the product / bulk product in subsequently received video recordings. To meet this goal, the data set used to train the deep neural network includes video recordings of various scenarios in which one or more of each product / bulk product from the inventory of the retail environment is placed on the detection plate of the self-service checkout device 10, and / or other objects are placed on the detection plate of the self-service checkout device 10. Specifically, the video recordings may include a scene in which one or more personal items are placed on the detection plate 32 alone; a scene in which one or more products / bulk products are placed on the detection plate 32 alone; and a scene in which one or more products / bulk products and one or more personal items are placed on the detection plate 32.

[0095] Video recordings, hereinafter referred to as the training dataset, are gathered with the goal of providing robust, class-balanced information about a subject product / bulk product derived from different views of the product / bulk product obtained by different positions and orientations of the product relative to the first camera 26 and the second camera 28, wherein the different positions and orientations of the product / bulk product represent the expected usage environment of the self-service checkout device 10. The members of the training dataset are selected to create sufficient diversity to overcome the challenges posed to subsequent product / bulk product recognition by variations in lighting conditions, variations in viewing angles, cluttered backgrounds, and most importantly, intra-class variations.

[0096] In one or more examples, before the video recording is used in the training data set, the video recording is processed to remove very similar video frames / images. The members of the training data set may also be subjected to further data augmentation techniques (such as rotation, flipping, brightness changes) to increase their diversity, thereby increasing the robustness of the object detection model ultimately trained. In another preprocessing step, each image / video frame in the video recording of the training data set is provided with one or more bounding boxes, wherein each such bounding box is arranged to surround an object visible in the image / video frame. Each image / video frame is also provided with a label of "product" or "other", as the case may be, corresponding to each bounding box in the corresponding image / video frame.

[0097] The object detection module 116 is also configured to concatenate the bounding box details of each object detected in the video frame with the corresponding label classification of the detected object to form a detected object vector. Specifically, the output from the object detection module 116 is one or more first detected object vectors and one or more second detected object vectors The object detection module 116 is further configured to send the output to the controller module 102 .

[0098] Cropping module 118

[0099] The cropping module 118 is communicatively coupled to the controller module 102 to receive the first selected sequence VS1 and the second selected sequence VS2 from the controller module 102. The cropping module 118 is further configured to receive the first product vector PV1(y), the second product vector PV2(y); and the selection time stamp vector STS(y) from the controller module 102. The cropping module 118 is adapted to crop one or more first cropping regions from each video frame of the first selected sequence VS1, the periphery of the first cropping region being established by the bounding box coordinates of the first product vector PV1(y), the time stamp of the first product vector PV1(y) (as determined according to the selection time stamp vector STS(y)) matching the time stamp of the video frame. The cropping module 118 is also suitable for cropping one or more second cropping areas from each video frame of the second selected sequence VS2, the periphery of which is established by the bounding box coordinates of the second product vector PV2(y), and the timestamp of the second product vector PV2(y) (as determined based on the selected timestamp vector STS(y)) matches the timestamp of the video frame.

[0100] The cropping module 118 is also configured to resize each first cropping region and each second cropping region to the same predefined size. As described in the following paragraphs, this size is established empirically as the size delivered to the embedding module 120 for optimal product / bulk product recognition. For clarity, this size will be referred to as the "processed image size" hereinafter. Optionally, further data augmentation techniques (such as rotation, flipping, brightness change) can be applied. The cropping module 118 is also configured to transmit the resulting first cropping region and second cropping region to the embedding module 120.

[0101] Embedded module 120

[0102] The embedding module 120 is coupled to the cropping module 118 to receive the first cropped region and the second cropped region from the cropping module 118 during the run operation phase. In this paper, the embedding module 120 adopts a deep metric learning module, as reviewed in K. Musgrave, S Belongie and S.-N. Li, A Metric Learning Reality Check (retrieved from https: / / arxiv.org / abs / 2003.08505, August 19, 2020), to learn a unique representation (in the form of an embedding vector) of each product and bulk product in the store inventory from the video frames of each product and bulk product in the store inventory. This enables the identification of either or both of the products and bulk products that subsequently appear in the video frames of the first video stream VID1 or the second video stream VID2. Specifically, the deep metric learning module is configured to generate embedding vectors in response to images of products / bulk products, where the embedding vectors are close together (in the embedding space) if the images contain the same product, and the embedding vectors are far apart if the images contain different products, as measured by a similarity or distance function (e.g., dot product similarity or Euclidean distance). A query image can then be validated based on a similarity or distance threshold in the embedding space.

[0103] When in use, the embedded module 120 has two distinct phases of operation, namely an initial configuration phase and a run phase.

[0104] Initial configuration phase of embedded module 120

[0105] During the initial configuration phase, the embedding module 120 is trained to learn one or more embedding vectors E i , the one or more embedding vectors E i Form products included in the store's inventory / bulk products p i Therefore, the initial configuration phase includes several different phases, namely, a training data preparation phase and a network training phase. These phases are continuously implemented in a cyclic iteration manner to train the embedding module 120. Each of these phases is described in more detail in the following paragraphs.

[0106] Training data preparation phase

[0107] The dataset used to train the embedding module 120 includes video recordings of a scene in which one or more of each product / bulk product from the inventory of the retail environment is placed on the detection plate of the self-service checkout device 10. The video recordings, which will be referred to as the training dataset below, are gathered with the goal of providing robust, class-balanced information about the subject product / bulk product derived from different views of the product / bulk product obtained by different positions and orientations of the product relative to the first camera 26 and the second camera 28, wherein the different positions and orientations of the product / bulk product represent the expected use environment of the self-service checkout device 10. The members of the training dataset are selected to create sufficient diversity to overcome the challenges posed to subsequent product / bulk product recognition by changes in lighting conditions, changes in viewing angles, cluttered backgrounds, and most importantly, intra-class variations.

[0108] In one or more examples, before the video recording is used in the training data set, the video recording is processed to remove very similar video frames. The members of the training data set may also be subjected to further data augmentation techniques (such as rotation, flipping, brightness changes) to increase their diversity, thereby increasing the robustness of the final trained deep neural network of the embedding module 120. A polygonal region surrounding each product / bulk product visible in the video frame is cropped from the video frame. The cropped region is resized to the processed image size to produce a cropped product / bulk product image. A category label for identifying the corresponding product / bulk product is also set for each cropped product / bulk product image.

[0109] Model training phase

[0110] For simplicity, the deep neural network (not shown) in the embedding module 120 will be referred to as an "embedding neural network (ENN)" hereinafter. The embedding neural network includes a deep neural network (e.g., ResNet, Inception, EfficientNet) in which the final one or more layers (which typically output classification vectors) are replaced with a linear normalization layer that outputs a unit norm (embedding) vector of a desired dimension (where the dimension is a parameter established when creating the embedding neural network).

[0111] During the model training phase, positive and negative pairs of cropped product / bulk product images are constructed from the training dataset. A positive pair includes two cropped product / bulk product images with the same class label, and a negative pair includes two cropped product / bulk product images with different class labels. For brevity, the resulting cropped product / bulk product images will be referred to as “paired cropped images” hereinafter. According to the pairwise mining strategy (e.g., as described by R. Manmatha, C.-Y. Wu, A. J. Smoia, and P. Sampling Matters in Deep Embedded Learning, 2017 IEEE International Conference on Computer Vision (CCV2017) Venice, 2017, pp. 2859-2867, doi: 10.1109 / ICCV.2017.309) is used to sample paired cropped images. Paired metric learning loss is then calculated based on the sampled paired video frames (as described in K. Musgrave, S. Belongie and S.-N. Lim, A Metric Learning Reality Check, 2020, https: / / arxiv.org / abs / 2003.08505). The weights of the embedded neural network are then optimized using the back-propagation method that minimizes the pairwise metric learning loss value.

[0112] All paired crop images are processed by the embedding neural network to generate their corresponding embedding vectors. As a result, the embedding vectors are organized in a pairwise manner similar to the paired crop images. The resulting embedding vectors are stored in the embedding database 124. Thus, given an image of each product / bulk product in the store inventory, the trained embedding neural network uses the embedding vector E calculated for each such product / bulk product i Populate the embedding database 124. Thus, the embedding database 124 includes the embedding vector and each product / bulk product p in the store inventory. i The corresponding identifier Id i Multiple tuples of (E i ,Id i ).

[0113] Operation phase of embedded module 120

[0114] For clarity, the runtime is defined as the normal business hours of the relevant store. During runtime, an embedding neural network (not shown) generates an embedding vector for each product / bulk product visible in a video frame of the product / bulk product placed on the self-service checkout device 10 captured by the first camera 26 and the second camera 28. Therefore, the embedding vector generated based on the first cropped area received from the video recording captured by the first camera 26 will be referred to as the first query embedding QE1 hereinafter. Similarly, the embedding vector generated based on the second cropped area received from the video recording captured by the second camera 28 will be referred to as the second query embedding QE2 hereinafter. The embedding module 120 is communicatively coupled to the expert system module 122 to transmit either or both of the first query embedding QE1 and the second query embedding QE2 to the expert system module 122.

[0115] Expert System Module 122

[0116] The expert system module 122 is coupled to the embedding module 120 to receive either or both of the first query embedding QE1 and the second query embedding QE2 generated by the embedding neural network during the run operation phase of the embedding module 120. Upon receiving the first query embedding QE1 or the second query embedding QE2, the expert system module 122 queries the embedding database 124 to retrieve the embedding vector E therefrom. i The expert system module 122 uses a similarity or distance function (eg, dot product similarity or Euclidean distance) to associate the first query embedding QE1 or the second query embedding QE2 with the embedding vector E i The expert system module 122 uses a similarity or distance function (eg, dot product similarity or Euclidean distance) to compare each first query embedding QE1 or second query embedding QE2 with the retrieved embedding vector E i If a preconfigured percentage (Per) of the first query embeddings QE1 of the corresponding first product vectors PV1(y) from the first selected sequence VS1 is equal to the retrieved embedding vector E i If the similarity between them exceeds a pre-configured threshold (Th), it is concluded that the first query embedding QE1 is similar to the retrieved embedding vector E i Similarly, if a preconfigured percentage (Per) of the second query embeddings QE2 of the corresponding plurality of second product vectors PV2(y) from the second selected sequence VS2 matches the retrieved embedding vector E i If the similarity between them exceeds a pre-configured threshold (Th), it is concluded that the second query embedding QE2 is similar to the retrieved embedding vector E i match.

[0117] The values ​​of the percentage (Per) and threshold (Th) parameters used by the expert system module 122 are established using a grid search method whose goal is to be empirically defined based on the risk preference of the operator, which is balanced with the operator's desire to speed up the checkout process at the self-service checkout device 10 of the present disclosure. In addition, the embedding database 124 is repeatedly queried and the retrieved embedding vector E is i The process of comparing the received first query embedding QE1 and the received second query embedding QE2 is repeated until a match is found or until all embedding vectors E have been retrieved from the embedding database 124. i In the plurality of first query embeddings QE1 and embedding vectors E from the embedding database 124 i When a match is found between i In the following, it will be referred to as the first matching embedding ME1. Similarly, in the plurality of second query embeddings QE2 and the embedding vectors E from the embedding database 124, i When a match is found between i In the following it will be referred to as the second matching embedding ME2. In contrast, when the first query embedding QE1 is matched with the embedding vector E from the embedding database 124 i In the event that no match is found between the second query embedding QE2 and the embedding vector E from the embedding database 124, i When no match is found, the expert system module 122 is configured to send a “product not identified” signal to the controller module 102 .

[0118] The expert system module 122 is further adapted to retrieve a product identifier corresponding to the first matching embedding ME1 from the product database 110 using the first matching embedding, wherein the product identifier is an identifier of the product / bulk product represented by the first matching embedding ME1. For the sake of brevity, the product identifier will be referred to as the first matching category label hereinafter. Similarly, the expert system module 122 is further adapted to retrieve a product identifier corresponding to the second matching embedding ME2 from the product database 110, wherein the product identifier is an identifier of the product / bulk product represented by the second matching embedding ME2. For the sake of brevity, the product identifier will be referred to as the second matching category label hereinafter.

[0119] The expert system module 122 is also configured to compare each first matching class tag with all second matching class tags. In the event of a match between the first matching class tag and any of the second matching class tags, the expert system module 122 is configured to retrieve a price corresponding to the first matching class tag or the second matching class tag from the product database 110. The expert system module 122 is also configured to transmit the first matching class tag and the corresponding price to the controller module 102. In the event of a match between the first matching class tag and any of the second matching class tags, the expert system module 122 is also configured to determine whether the first matching class tag corresponds to a weight-related bulk product item. In the event that the first matching class tag corresponds to a weight-related bulk product item, the expert system module 122 is also configured to determine whether the first matching class tag corresponding to the first cropping area of ​​the first product vector PV1(y) and the second matching class tag corresponding to the second cropping area of ​​the second product vector PV2(y) include products that are not weight-related bulk product items. In other words, the expert system module 122 is configured to check whether the weight-dependent bulk product items and the products whose prices do not depend on the weight are placed on the detection plate 32 of the self-service checkout device 10 at the same time. In this case, the expert system module 122 is configured to send a "mixed product reminder" signal to the controller module 102. Otherwise, the expert system module 122 is configured to send a "weight-dependent product" signal to the controller module 102. In contrast, in the case of a mismatch between the first matching category tag and any second matching category tag, the expert system module 122 is configured to send a "product mismatch reminder" signal to the controller module 102.

[0120] Boot module 130

[0121] The guidance module 130 is communicatively coupled to the detection board 32 and the controller module 102 of the interactive unit 14. In use, the guidance module 130 is operable to activate the design display unit to display a design on the detection board 32 before detecting the motion near the automatic checkout device 10 or the detection board 32. The design includes at least two circular or oval shapes arranged in opposite quadrants of the upper surface of the detection board 32. The circles can be configured to have sufficient size to surround the bottom of a common bottle or can (such as Fig.10). Those skilled in the art will appreciate that the above designs are provided for illustrative purposes only. In particular, those skilled in the art will appreciate that the self-service checkout device 10 of the present disclosure is not limited to the details of the above patterns. Rather, the self-service checkout device 10 of the present disclosure may be implemented in any design that is used to provide some initial guidance about the positioning of the product on the detection plate 32 before detecting the motion of the self-service checkout device 10 or the detection plate 32. For example, the design may include a polygonal shape, rather than a circular or oval shape. Similarly, the design may include four or more such circular, oval, or polygonal shapes that are substantially evenly disposed on all four quadrants of the detection plate.

[0122] Once a motion trigger signal is issued by the motion detection module 104, the guidance module 130 is operable to receive the first product vector PV1(y) and the second product vector PV2(y) from the controller module 102. The guidance module 130 is configured to check the bounding boxes of the first product vector PV1(y) and the second product vector PV2(y) to determine whether the corresponding bounding boxes are set in more than one quadrant of the detection board 32 when there are two or more detected products. In the event that the guidance module 130 detects the opposite, the user is prompted to place all products in only one quadrant of the detection board 32. This may block the view of one or more products, thereby hindering their identification. Therefore, in the event that the guidance module 130 detects that the bounding boxes of two or more detected products are set in one quadrant of the detection board 32, the guidance module 130 is adapted to send a "quadrant misuse reminder" signal to the controller module 102.

[0123] The guidance module 130 is also operable to view the bounding boxes of the first product vector PV1(y) and the second product vector PV2(y) to compare the distance between adjacent bounding boxes. In the case where the distance is less than a predefined threshold (hereinafter referred to as the bounding box separation threshold), the guidance module 130 is configured to send a "product distance reminder" signal to the controller module 102. The "product distance reminder" includes the coordinates of the bounding boxes separated by a distance less than the predefined threshold. For simplicity, these bounding boxes will be referred to as "overly close bounding boxes" hereinafter. Therefore, the "product distance reminder" includes the coordinates of the overly close bounding boxes. The bounding box separation threshold is empirically established based on the size of the products typically provided for sale by the operator, the lighting conditions of the retail environment, and other conditions that will hinder the performance of the self-service checkout device 10. The predefined distance is configured to balance the need to maximize the accuracy of product identification and speed enhancement, which can be achieved by allowing customers to place several products on the detection plate 32 at the same time so that these products can be registered by the self-service checkout device 10 at the same time.

[0124] Weighing module 126

[0125] The weighing module 126 is communicatively coupled to the controller module 102 to receive an activation signal from the controller module 102. Upon receiving the activation signal, the weighing module 126 is configured to activate a weighing scale unit (not shown) of the automatic checkout device 10 to measure the weight of the bulk product items. The weighing module 126 is also configured to transmit the weight measurement value to the controller module 102.

[0126] Billing module 128

[0127] The billing module 128 is configured to receive from the controller module 102 the price of each product whose barcode is detected in the first selected sequence VS1 or the second selected sequence VS2. Alternatively, the billing module 128 is configured to receive from the controller module 102 the price of each product whose barcode is obtained by the 1D barcode reader 38 of the self-service checkout device 10. The billing module 128 is configured to sum these prices to calculate the total bill for the product. Otherwise, in the event that the barcode processing module 108 fails to detect the presence of a barcode in the video frame of the first selected sequence VS1 or the second selected sequence VS2, the billing module 128 is configured to receive from the controller module 102 the price of each product / bulk product identified by the appearance interpretation module 114 in the first selected sequence VS1 and the second selected sequence VS2, wherein the price is the price corresponding to the first matching category label determined by the expert system module 122. The billing module 128 is also configured to receive from the controller module 102 the weight measurement of the detected weight-related bulk product items. The billing module 128 is further configured to calculate a total bill for the weight-related bulk product items based on the weight measurements of the weight-related bulk product items and the price per unit weight of the weight-related bulk product items received from the controller module 102. The billing module 128 is further configured to sum the prices of all products identified by the appearance interpretation module 114 in the first selected sequence VS1 and the second selected sequence VS2 to calculate the total bill for the products.

[0128] Controller module 102

[0129] In this embodiment, the controller module 102 is configured to receive a motion trigger signal from the motion detection module 104, and send the motion trigger signal to the sequence selection module 106 to activate the sequence selection module 106. The controller module 102 is also configured to receive a first selected sequence VS1 and a second selected sequence VS2 from the sequence selection module 106, and send the first selected sequence VS1 and the second selected sequence VS2 to the barcode processing module 108. The controller module 102 is configured to receive an appearance activation signal from the barcode processing module 108, and send the appearance activation signal to the appearance interpretation module 114 to activate the appearance interpretation module 114.

[0130] The controller module 102 is also configured to receive the first detected object vector DO1(τ+iΔt) and the second detected object vector DO2(τ+iΔt) from the object detection module 116. In the case that any first label vector L1(τ+iΔt) of the first detected object vector DO1(τ+iΔt) or any second label vector L2(τ+iΔt) of the second detected object vector DO2(τ+iΔt) includes "other", the controller module 102 is configured to cause a message to be displayed on the display screen 34 of the interactive unit 14, wherein the message reminds the user that an item that is not a sales product is placed on the detection plate 32 and should be removed from the detection plate 32. The controller module 102 is also operable to activate the motion detection module 104 to detect motion in the area near the self-service checkout device 10 and the detection plate 32. If no motion trigger signal is received from the motion detection module 104 within a predefined time interval, the controller module 102 is operable to issue a reminder to the operator to indicate that the customer needs their assistance. For simplicity, the predefined time interval will be referred to as the "non-sale object reset period" hereinafter. However, upon receiving a motion trigger signal from the motion detection module 104 within the non-sale object reset period, the controller module 102 is operable to activate the object detection module 116 to view another first selected sequence VS1 and another second selected sequence VS2. In the event that any first label vector L1 (τ + iΔt) of the resulting first detected object vector DO1 (τ + iΔt) or any second label vector L2 (τ + iΔt) of the resulting second detected object vector DO2 (τ + iΔt) includes "other", the controller module 102 is also operable to issue a reminder to the operator to indicate that the customer needs their help. Similarly, upon receiving an excess object reminder signal from the object detection module 116, the controller module 102 is adapted to cause a message to be displayed on the display screen 34 of the interactive unit 14, wherein the message reminds the user that they should remove some of the products they have placed on the detection board 32.

[0131] For ease of understanding, for a given first selected sequence VS1 or another given first selected sequence VS1, let np1 video frames Fr1(τ+iΔt)(q st ≤i≤q st +α) contains only objects marked as sales products, in other words, np1≤α. Similarly, for a given second selected sequence VS2 or a given another second selected sequence VS2, let np2 video frames Fr2(τ+iΔt)(q st ≤i≤q st +α) contains only objects that are labeled as sales products, in other words, np2≤α. In addition, the first temporary product vector IPV1(w1) (1≤w1≤np1) is defined as the first detected object vector DO1(τ+iΔt), whose first label vector L1(τ+iΔt) only includes the "sale product" element; and the first temporary timestamp TS1(w1) (1≤w1≤np1) is defined as the timestamp (τ+iΔt) of the first detected object vector DO1(τ+iΔt) being mentioned. Therefore, for a given first temporary product vector IPV1(w1) (1≤w1≤np1), there is a matching first temporary timestamp TS1(w1) (1≤w1≤np1). In addition, the second temporary product vector IPV2(w2) (1≤w2≤np2) is defined as the second detected object vector DO1(τ+iΔt), whose second label vector L2(τ+iΔt) only includes the "sale product" element; and the second temporary timestamp TS2(w2) (1≤w2≤np2) is defined as the timestamp (τ+iΔt) of the first detected object vector DO2(τ+iΔt) being mentioned. Therefore, for a given second temporary product vector IPV2(w2) (1≤w2≤np2), there is a matching second temporary timestamp TS2(w2) (1≤w2≤np2). Ideally, np1 should match np2; however, depending on the complexity of the viewed scene, either the first camera 26 or the second camera 28 may detect the presence of an object on the detection plate that is not detected by the other camera.

[0132] In this article, the controller module 102 is configured to compare the first temporary timestamp TS1(w1)(1≤w1≤np1) and the second temporary timestamp TS2(w2)(1≤w2≤np2). The controller module 102 is also configured to select a first temporary product vector IPV1(w1)(1≤w1≤np1) and a second temporary product vector IPV2(w2)(1≤w2≤np2), wherein the first temporary timestamp TS1(w1)(1≤w1≤np1) corresponding to the first temporary product vector matches the second temporary timestamp TS2(w2)(1≤w2≤np2) corresponding to the second temporary product vector. For simplicity, the selected first temporary product vector IPV1(w1) and the second temporary product vector IPV2(w2) will be referred to as the first product vector PV1(y) and the second product vector PV2(y) respectively hereinafter, where 1≤y≤sel, sel is the number of the selected first product vector PV1(y) or second product vector PV2(y), and sel≤min(np1,np2). The timestamps of each first product vector PV1(y) and the corresponding second product vector PV2(y) are collated into a selection timestamp vector STS(y). This approach is adopted to exclude those video frames in which there are transient changes (such as light flicker or high-speed movement that are not detected by the motion detection module 104) from subsequent consideration of the self-service checkout device 10.

[0133] The controller module 102 is configured to transmit the first product vector PV1(y), the second product vector PV2(y), and the selected time stamp vector STS(y) to the guidance module 130. In the absence of a quadrant misuse reminder or a product distance reminder received from the guidance module 130, the controller module 102 is configured to transmit the first product vector PV1(y), the second product vector PV2(y), and the selected time stamp vector STS(y) to the cropping module 118. However, in the event of receiving a quadrant misuse reminder signal from the guidance module 130, the controller module 102 is configured to activate the design display unit of the interactive unit 14 so that the design display unit changes the design displayed on the inspection board 32 so that a circle, an oval, or a polygon is highlighted in each of the four quadrants of the inspection board 32. The controller module 102 is also configured to cause a message to be displayed on the display screen 34 of the interactive unit 14, wherein the message reminds the user that they should move some of the products they have placed on the inspection board 32 to other quadrants of the inspection board 32. The controller module 102 is also operable to activate the motion detection module 104 to detect the movement of the product. If no motion trigger signal is received from the motion detection module 104 within a predefined time interval, the controller module 102 is operable to issue a reminder to the operator to indicate that the customer needs their help. For the sake of simplicity, the predefined time interval will be referred to as the "first product movement reset period" below. However, once a motion trigger signal is received from the motion detection module 104 within the first product movement reset period, the controller module 102 is operable to activate the guidance module 130 to view the arrangement of the bounding box of the product on the detection board 32. In the case where the guidance module 130 re-issues the quadrant misuse reminder, the controller module 102 is operable to issue a reminder to the operator to indicate that the customer needs their help. In this article, the first product movement reset period and the second product movement reset period are each empirically determined according to the needs of the operator to give the user enough time to move his product to a new location while minimizing delays in the registration process.

[0134] Upon receiving a product distance reminder, the controller module 102 is configured to use the coordinates of the overly close bounding box included in the product distance reminder to identify the closest circle, oval or polygon displayed on the detection board 32 corresponding to the "overly close bounding box". For simplicity, each of these circles, ovals or polygons will be referred to as the "nearest guide handle" hereinafter. The controller module 102 is also configured to activate the design display unit of the interactive unit 14 so that the nearest guide handle moves to another position within a predefined distance of the coordinates of the overly close bounding box, and so that the nearest guide handle is separated by a distance exceeding the bounding box separation threshold. For clarity, the predefined distance of the movement of the nearest guide handle is referred to as the guide handle movement distance hereinafter. The guide handle movement distance is determined empirically based on the need to balance the accuracy of maximizing product identification with speed enhancement, which can be achieved by allowing customers to place several products on the detection board at the same time so that these products can be registered by the self-service checkout device 10 at the same time.

[0135] The controller module 102 is also configured to cause a message to be displayed on the display screen 34 of the interactive unit 14, wherein the message reminds the user that they should move the product closest to the nearest guide control key to the location of the nearest guide control key. The controller module 102 is operable to activate the motion detection module 104 to detect the movement of the product. If no motion trigger signal is received from the motion detection module 104 within a predefined time interval, the controller module 102 is operable to issue a reminder to the operator to indicate that the customer needs their help. For the sake of simplicity, the predefined time interval will be referred to as the "second product movement reset period" below. However, once a motion trigger signal is received from the motion detection module 104 within the second product movement reset period, the controller module 102 is operable to activate the guidance module 130 to view the arrangement of the bounding box of the product on the detection board. In the event that the guidance module 130 re-issues the product distance reminder, the controller module 102 is operable to issue a reminder to the operator to indicate that the customer needs their help.

[0136] The controller module 102 is also configured to receive an unidentified product reminder signal or a mixed product reminder signal or a product mismatch reminder signal from the expert system module 122. Upon receiving any of these signals, the controller module 102 is also configured to activate the display screen 34 of the self-service checkout device 10 to display a prompt to the user to present the product to the 1D barcode reader 38 of the self-service checkout device 10 so as to thereby read the barcode of the product. In the event that the 1D barcode reader 38 fails to read the barcode of the product, the controller module 102 is configured to issue a reminder to the operator to indicate that the customer needs their assistance.

[0137] The controller module 102 is also configured to receive a product label and a corresponding price from the barcode processing module 108; or to receive a first matching category label and a corresponding price from the expert system module 122. The controller module 102 is also configured to receive a "weight-related product" signal from the expert system module 122; and upon receiving the "weight-related product" signal, activate the weighing module 126 if a weighing scale unit is present in the automatic checkout device 10 of the present disclosure; or issue an alert to the operator to indicate that the customer needs their assistance. The controller module 102 is also configured to receive a weight measurement from the weighing module 126. The controller module 102 is also configured to transmit to the billing module 128 the price received from the barcode processing module 108; or the price received from the 1D barcode reader 38 of the self-service checkout device 10; or the price received from the expert system module 122; and the weight measurement (if any) from the weighing module 126. In response, the controller module 102 is also configured to receive from the billing module 128 a total bill for the products registered by the self-service checkout device 10 of the present disclosure.

[0138] In addition, the controller module 102 is configured to activate the display screen 34 of the self-service checkout device 10 to display any one of the following: (a) the product label and corresponding price received from the barcode processing module 108 or the 1D barcode reader (not shown) of the self-service checkout device 10; or (b) the first matching category label and corresponding price from the expert system module 122. The controller module 102 is configured to activate the display screen 34 of the self-service checkout device 10 to display a detailed list and a total bill of the product (e.g., Fig.10 shown).

[0139] Alternatively, in the event that the appearance interpretation module 114 identifies a weight-related bulk product item in the first selected sequence VS1 and the second selected sequence VS2, the billing module 128 is configured to activate the display screen 34 of the self-service checkout device 10 to display a first matching category label of the weight-related bulk product item and a corresponding total bill (e.g., Fig.10 Herein, the billing module 128 is also configured to activate the contactless card reader 40 to receive payment for the total bill of the identified product or the total bill of the bulk product items associated with the weight.

[0140] Management module 132

[0141] The management module 132 is adapted to allow an operator to access the software of the self-service checkout device 10 in order to update either or both of the software and its configuration, such as for updating a tuple of the product database 110; or refreshing the training of the embedded neural network of the embedding module 120 or the training of the deep neural network of the object detection module 116. The management module 132 may also include a PIN function or other access control mechanism to limit access to the software of the self-service checkout device 10 to certain designated operators.

[0142] The above discussion of the first embodiment of the software of the self-service checkout device 10 (eg Figure 6 10) focuses on a standalone implementation, where a given single self-service checkout device 10 is provided with its own product database 110 and embedded database 124. A single self-service checkout device 10 can operate without interacting with other infrastructure or other self-service checkout devices 10 in a store. The process of updating the software and configuration of each self-service checkout device 10 (including training the embedded neural network and object detection neural network of the self-service checkout device 10) is feasible when there are only a small number of them in the store. However, in a larger store including several such self-service checkout devices 10, the process of updating the software and configuration of each individual self-service checkout device 10 one by one is difficult. In this case, several self-service checkout devices 10 are individually communicatively coupled to a central controller, which includes a centralized software update scheduler and a centralized record of the store's entire inventory of products / bulk products. The central controller issues software configuration updates, including updates to the internal representations / embeddings formed in the embedded neural network and the object detection neural network.

[0143] Reference now Figure 7 , shows a schematic block diagram of a system 100 having a plurality of self-service checkout devices 10 according to a second exemplary embodiment of the present disclosure. Figure 7 As shown, combined with the description in the previous paragraph Figure 1-6, the system 700 has individual self-service checkout devices 10 communicatively coupled in a distributed network. The embedding neural network and object detection neural network of the self-service checkout devices 10 can be trained with different subsets of products / bulk products in the store's inventory, and the product database 110 and embedding database 124 of the individual self-service checkout devices 10 can be populated accordingly. The individual self-service checkout devices 10 are configured to share with each other the embedding representations formed by their respective embedding neural networks and object detection neural networks. The individual self-service checkout devices 10 are also configured to share with each other the members of their product databases 110 and embedding databases 124. While not requiring the embedding neural network and object detection neural network of each self-service checkout device 10 to be individually trained with members of the store's entire inventory of products / bulk products, the system 700 also eliminates the need to maintain a centralized software update scheduler and a centralized record of the store's entire inventory of products / bulk products.

[0144] Continue to refer Figure 8 , which shows a flow chart of a method 200 implemented by the self-service checkout device 10 of the present disclosure. In this article, it is considered Figure 1-7The method 200 is described using the components defined in . In step 202, the method 200 includes receiving a video recording including a plurality of video frames from each of the first camera 26 and the second camera 28. In step 204, the method 200 includes detecting the presence of motion in the received video recording. In step 206, the method 200 includes selecting a predefined number of video frames from the received video recording after detecting the end of motion in the received video recording. In step 208, the method 200 includes detecting and decoding a barcode visible in the selected video frame. In step 210, the method 200 includes calculating a total bill corresponding to the decoded barcode. In step 212, the method 200 includes detecting the presence of an object visible in the selected video frame if the barcode is not visible in the selected video frame. In step 214, the method 200 includes distinguishing between sale items and non-sale items in the detected objects. In step 216, the method 200 includes issuing a reminder when a non-sale item is detected, the reminder including a message to remove the non-sale item from the self-service checkout device 10. At step 218, method 200 includes determining the location of the detected sale items. At step 220, method 200 includes determining the distribution of the detected sale items based on the determined locations of the detected sale items. At step 222, method 200 includes issuing an alert when an inappropriate distribution of the detected sale items is detected. At step 224, method 200 includes cropping one or more regions substantially surrounding each detected sale item from each received video frame. At step 226, method 200 includes generating an embedded representation of the sale items visible in the cropped region based on each cropped region. At step 228, method 200 includes comparing the generated embedded representation with a record of embedded representations of products contained in a product inventory of the retail environment to find a match with any member in the record. At step 230, method 200 includes determining a price corresponding to a matching member in the record of embedded representations of products contained in a product inventory of the retail environment. At step 232, method 200 includes calculating a total bill for all products / bulk products visible in the received video recording, wherein for all products / bulk products visible in the received video recording, a member match is found to a record of embedded representations of products contained in the product inventory of the retail environment. At step 234, method 200 includes displaying the total bill to the user. And, at step 236, method 200 includes receiving payment for the total bill from the user. It will be understood that steps 202 to 236 as described herein are merely illustrative, and other alternatives may be provided in which one or more steps are added, one or more steps are removed, or one or more steps are provided in a different order without departing from the scope of the claims herein.

[0145] In one or more examples, the step 202 of receiving a video recording including a plurality of video frames from each of the first camera 26 and the second camera 28 is preceded by the step of displaying a design on the detection board 32 of the self-service checkout device 10. Preferably, the step of displaying a design includes the step of displaying any one or more of a circle, an oval, or a polygon in each quadrant of the detection board 32.

[0146] In one or more examples, the step 204 of detecting the presence of motion in the received video footage includes the step of using motion vectors obtained from decoding of H.264 video frames. Alternatively, the step 204 of detecting the presence of motion in the received video footage includes the following sub-steps: comparing consecutive samples of the video frames received from the first camera 26 to detect differences therebetween; comparing consecutive samples of the video frames received from the second camera 28 to detect differences therebetween; and if the detected difference exceeds a predefined threshold, determining that motion occurs in the period between the consecutive samples.

[0147] In one example, the threshold is configured to be mistaken for momentary changes in motion, such as light flicker. In one example, the intervals between successive samples of video frames received from the first camera 26 and successive samples of video frames received from the second camera 28 are configured to be of sufficient duration to avoid falsely detecting small, high-speed motions, such as finger movement, rather than larger motions corresponding to product placement or removal on the inspection board.

[0148] In one or more examples, in the event that a barcode is not visible in the selected video frame, step 212 of detecting the presence of an object visible in the selected video frame; and step 214 of distinguishing between sale items and non-sale items among the detected objects include the following sub-steps: prior to step 204 of detecting the presence of motion in the received video footage, training an object detection neural network model (e.g., training an EfficientDet neural network), the training being performed using a labeled training data set to form an internal representation of sale items and non-sale items placed on the detection plate 32 of the self-service checkout device 10; presenting the selected video frame to the trained object detection neural network model to form a representation of the sale items and non-sale items visible in the video frame; and obtaining labels corresponding to the sale items visible in the selected video frame and coordinates of a bounding box that substantially surrounds each sale item from the object detection neural network model.

[0149] In another example, in the case where the barcode is not visible in the selected video frame, the step 212 of detecting the presence of an object visible in the selected video frame; and the step 214 of distinguishing between sale items and non-sale items in the detected object further include the following sub-steps:

[0150] - Count the number of objects detected in any of the selected video frames;

[0151] - Sending an alert when the number of detected objects exceeds a predefined threshold, wherein the alert is a message requesting the removal of some products placed on the detection.

[0152] In one or more examples, step 222 of issuing an alert upon detecting an inappropriate distribution of the detected sales items includes the following sub-steps:

[0153] - Issuing an alert if two or more sales items are placed in only one quadrant of the detection plate 32 of the self-service checkout device 10; and

[0154] - Issuing an alert if the distance between adjacent sales items on the detection board 32 is less than a predefined threshold.

[0155] In addition, the step 222 of issuing a reminder when two or more sales items are detected to be set in only one quadrant of the detection plate 32 of the self-service checkout device 10 includes the following sub-steps:

[0156] - changing the design displayed on the detection plate 32 so that a circle, an oval or a polygon is highlighted in each of the four quadrants of the detection plate 32;

[0157] - displaying a message on the display screen 34 of the interactive unit 14, wherein the message reminds the user that they should move some of the sales items they have placed on the detection board 32 to other quadrants of the detection board 32;

[0158] - detecting whether a vend item has been moved to more than one quadrant of the detection plate 32; and

[0159] - If the merchandise item is not moved within a predefined time interval, an alert is issued to the operator indicating that the customer requires assistance.

[0160] In addition, the step 222 of issuing a reminder when it is detected that the distance between adjacent sales items on the detection plate 32 is less than a predefined threshold includes the following sub-steps:

[0161] - identifying a circle, oval or polygonal shape in the design displayed on the detection board 32 that is arranged closest to an adjacent sales item, the distance between which adjacent sales items is less than a predefined threshold;

[0162] - moving the identified circle, oval or polygon to another position within the guide control key movement distance of the adjacent sales items, the distance between the adjacent sales items being less than the predefined threshold;

[0163] - displaying a message on the display screen 34 of the interactive unit 14, wherein the message prompts the user that they should move the sales item closest to the moved circle, oval or polygon to the position of the moved circle, oval or polygon;

[0164] - Detecting whether the sales item has been moved to the moved circular, oval or polygonal position; and

[0165] - In the event that a sales item is not moved within a predefined time interval, an alert is issued to the operator indicating that the customer requires assistance.

[0166] In one or more examples, step 224 of cropping one or more regions from each received video frame that substantially surround each detected sale item includes: cropping one or more regions from each received video frame, the perimeter of the one or more regions being established by step 212 of detecting the presence of an object visible in the selected video frame when a bar code is not visible in the selected video frame; and coordinates of a bounding box generated by step 214 of distinguishing between sale items and non-sale items among the detected objects.

[0167] In one or more examples, step 226 of generating, based on each crop region, an embedded representation of the sale items visible in the crop region includes the following sub-steps:

[0168] - prior to the step 204 of detecting the presence of motion in the received video footage, training an embedding neural network, said training being performed using a training dataset to form an embedding representation of each of a plurality of products / bulk products in the store inventory;

[0169] - presenting the cropped region to a trained embedding neural network to form an embedding representation of the product or bulk product visible in the cropped region;

[0170] - compare the embedding formed from the cropped region with the embedding of each product / bulk product in the store inventory; and

[0171] - determining whether the embedded representation formed from the cropped region matches any of the embedded representations of products / bulk products in the store's inventory; and

[0172] - Retrieve the label corresponding to the matching embedded representation, along with the price or price per unit weight for that label.

[0173] Herein, in the event that no match is found between the embedding representation formed from the cropped region and any of the embedding representations of products in the store inventory, method 200 includes the following steps:

[0174] - displaying a message on the display screen 34 of the interactive unit 14, wherein the message requests the user to present the barcode of the product visible in the cropping area (if present) to the 1D barcode reader 38 of the self-service checkout device 10;

[0175] - activating the 1D barcode reader 38 to read the presented barcode; and

[0176] - In the event that the ID barcode reader 38 is unable to read the presented barcode, an alert is issued to the operator indicating that the customer requires assistance.

[0177] In addition, in the event that the embedded representation determined to match the cropped area generated based on the video recording from the first camera 26 does not match the corresponding embedded representation determined to match the cropped area generated based on the second camera 28, method 200 includes a step of issuing an alert to the operator, indicating that the customer needs assistance.

[0178] Furthermore, in the event that the embedded representation determined to match the cropped region generated from the video recording from the first camera 26 or the second camera 28 is an embedded representation of a bulk product, the method 200 includes the following steps:

[0179] - displaying a message on the display screen 34 of the interactive unit 14, which message requests the user to remove the remaining products placed on the detection plate 32 from the detection plate 32 of the automatic checkout device 10, in the case where the weighing scale unit is integrated with the detection plate 32;

[0180] - displaying a message on the display screen 34 of the interactive unit 14, in the case where the weighing scale unit is a separate component from the detection board 32 of the automatic checkout device 10, the message requests the user to place the item on the weighing scale unit of the automatic checkout device 10;

[0181] - activating the weighing scale unit to measure the weight of the bulk product; and

[0182] - Calculate the price of the bulk product by multiplying the measured weight of the bulk product by the retrieved price per unit weight of the bulk product.

[0183] Based on the method 200 described in the previous paragraphs, the self-service checkout device 10 of the present disclosure solves the following application scenarios:

[0184] a) The customer places the product / bulk products on the detection plate 32 where they are identified within 30 seconds by their barcode or their appearance. The self-service checkout device 10 displays guidance to the customer to assist in positioning the product / bulk products on the detection plate 32 to increase the likelihood that they will be correctly identified.

[0185] b) Using the 1D barcode scanner of the self-service checkout device 10 , the customer scans a product that is not automatically recognized by the software of the self-service checkout device 10 .

[0186] c) The customer places the bulk product on top of the weigh scale unit of the automated checkout device 10 to measure the weight of the bulk product. The weigh scale unit may be integrated with the detection plate 32, in which case the customer removes all other products from the detection plate 32 before the weigh scale unit measures the weight of the bulk product. The software calculates the price of the bulk product by multiplying its weight by its price per unit weight. The price of the bulk product is added to the bill which is calculated as the sum of the prices of the remaining identified products.

[0187] d) In the event that the customer has more than 6 products / bulk products, the customer places the first six products / bulk products on the detection board 32. When the products / bulk products are identified by the software of the self-service checkout device 10, the customer presses a multi-function button of the self-service checkout device 10 to allow additional products / bulk products to be included in the total bill, and places the additional products / bulk products on the detection board 32 where they are identified and their prices are added to the total bill.

[0188] e) In the event that the user encounters a problem, they can press the multi-function button to put the self-service checkout device 10 into standby mode and alert a store assistant to come to help. The store assistant can be alerted by illumination of a colored light mounted on either or both of the concave fixing member and the interactive unit of the self-service checkout device 10.

[0189] f) In the event that the self-service checkout device 10 does not receive payment for the product / bulk product detected on the detection plate 32, the self-service checkout device 10 is put into standby mode and a reminder is issued to the store associate regarding the self-service checkout device 10 being referred to investigate the event.

[0190] g) While assessing the reason why the self-service checkout device 10 entered standby mode, the store attendant may enter a PIN into the self-service checkout device 10, thereby causing it to be reset for further use.

[0191] h) By capturing a small number (up to 3) of video frames used to train the object detection neural network and the SKUs embedded in the neural network, the store attendant can add more products / bulk products for subsequent detection and identification by the software of the self-service checkout device 10. In a distributed network embodiment of the self-service checkout devices 10, the added products / bulk products are automatically propagated to the remaining self-service checkout devices 10 in the store, and if necessary, to self-service checkout devices 10 in other stores.

[0192] Reference now Fig. 9, showing a depiction of a first alternative embodiment of the self-service checkout device 10 of the present disclosure. As shown in the figure, according to the first alternative embodiment, the back side of the self-service checkout device 10 (as shown in reference Figure 1-3 In contrast to what is shown and described above, the self-service checkout device 10 may have a bracket unit 12 mounted on a lateral side (herein, the left side) of the interaction unit 14.

[0193] The present disclosure provides a self-service checkout device 10 with a small footprint, which allows for quick registration of one or more products, some of which may be bulk products. Specifically, the automatic checkout device 10 includes a detection plate 32, on which a plurality of products can be placed together. The self-service checkout device 10 also includes two cameras 26, 28, which are configured to capture video recordings of the detection plate 32 and the products placed thereon. The video recordings from the cameras 26, 28 are processed by computer vision to enable identification and registration of each product. The self-service checkout device 10 of the present invention implements a robust product recognition algorithm, which is configured to identify and recognize products placed on the board, regardless of the orientation of the product. Therefore, the product recognition algorithm allows identification of products without the need to scan the barcode of the product. The automatic checkout device 10 of the present invention also provides a weighing scale unit to allow for weighing of bulk product items related to weight. Therefore, the self-service checkout device 10 of the present invention allows for substantially simultaneous registration of up to six products in a given transaction, without the need for scanning or otherwise manually identifying items by the customer. Thus, the self-checkout device 10 of the present invention enables significantly faster check-in of multiple products in a single transaction, thereby reducing latency in high throughput sales environments.

[0194] The self-service checkout device 10 of the present invention can be implemented with a much smaller number of cameras than other solutions. Specifically, the self-service checkout device 10 of the present invention can be implemented with two cameras 26, 28 mounted on an upright concave fixed member 16. In contrast, other solutions require 4 to 6 cameras. In one example, the two cameras 26, 28 are mounted so that the first camera 26 is substantially disposed above the detection plate 32 so that it has a top-down view of the detection plate 32; and the second camera 28 is disposed so that it has a side view of the detection plate 32. The inward curvature of the concave fixed member 16 allows the second camera 28 to have a wide field of view. The inward curvature of the concave fixed member 16 also provides a larger access to the detection plate 32 for the user to place the product on the detection plate 32 than the access to the detection plate 32 that the vertical upright fixed member can provide. The concave fixed member 16 also provides a light diffusing housing member 24 (or reflector element) that further illuminates the product on the detection plate 32, thereby assisting the computer vision algorithm of the present disclosure. Herein, the inspection board 32 includes an upward-facing interactive display unit configured to display markings to guide the placement of products on the inspection board 32 so that they can be subsequently identified by a computer vision algorithm. The display unit can be configured to display a white background to eliminate reflections of products placed on the inspection board 32 on the display unit, and further eliminate reflections of the surrounding environment on the display unit. Herein, the positioning of the displayed markings is determined by a positioning algorithm that can be operated using video recording from the first camera 26. The positioning algorithm determines the optimal positioning of one or more products on the inspection board 32, wherein the optimal positioning is established to maximize the viewing angle of the cameras 26, 28 to the product, and the maximization is achieved by minimizing the occlusion of the product by other products placed on the inspection board 32.

[0195] The foregoing description of specific embodiments of the present disclosure has been presented for the purpose of illustration and description. Expressions such as "including", "comprising", "incorporating", "having", and "being" used to describe and claim the present disclosure are intended to be interpreted in a non-exclusive manner, that is, allowing items, parts, or elements that are not explicitly described to also exist. References to the singular should also be interpreted as involving the plural. The word "exemplary" is used herein to mean "used as an example, instance, or illustration". Any embodiment described as "exemplary" is not necessarily interpreted as being preferred or advantageous over other embodiments and / or excluding the combination of features from other embodiments. They are not intended to be exhaustive or limit the present disclosure to the precise form disclosed, and it is clear that many modifications and variations are possible in accordance with the above teachings. The exemplary embodiments are selected and described in order to best explain the principles of the present disclosure and its practical application, so that other technical personnel in the art can best utilize the present disclosure and various embodiments with various modifications suitable for the intended specific purpose.

Claims

1. A self-service checkout device, comprising: a test plate adapted to allow placement of a product on the test plate; one or more cameras, the one or more cameras being positioned to have a field of view that at least encompasses the inspection board, the one or more cameras being configured to provide video recording; a motion detection module, the motion detection module being configured to detect the presence of motion in the video recording; a sequence selection module configured to select a sequence of video frames within a time interval corresponding to a detection of motion in the video recording; an appearance interpretation module configured to register one or more products present in the sequence of video frames; a billing module configured to obtain prices of one or more registered products, generate a total bill based on the obtained prices, and process payment of the total bill; as well as A controller module operably connected to the one or more cameras and communicatively coupled to the motion detection module, the sequence selection module, the appearance interpretation module, and the billing module to control the operation of these modules and facilitate communication between these modules.

2. The self-service checkout device according to claim 1, wherein the appearance interpretation module comprises: an object detection module, the object detection module being configured to analyze the sequence of video frames to detect one or more objects in the sequence of video frames; a cropping module configured to isolate one or more detected objects in the sequence of video frames and extract visual features of the detected one or more objects; an embedding module configured to convert the extracted visual features of the detected one or more objects into an embedded feature vector; as well as an expert system module configured to compare the embedded feature vector with feature vectors pre-stored in an embedding database and identify the detected one or more objects based on the comparison; The identified one or more objects are registered as the one or more products.

3. The self-service checkout device of claim 2, wherein the appearance interpretation module employs a machine learning model to facilitate the detection, cropping, embedding and recognition processes.

4. The self-checkout device of claim 2, wherein the expert system module is further configured to determine whether any of the identified one or more objects is a weight-related bulk product item from among the one or more products.

5. The self-service checkout device of claim 4, further comprising a weighing module configured to activate a weighing scale unit to measure the weight of the weight-related bulk product items from among the one or more products placed on the detection plate, and wherein, The billing module is configured to generate the total bill based on the measured weight of the weight-related bulk product items.

6. The self-service checkout device according to claim 1 further comprises a barcode processing module, wherein the barcode processing module is configured to detect one or more barcodes in a selected video frame sequence and decode the detected barcode corresponding to the one or more registered products, and wherein the billing module is configured to obtain the price of the one or more registered products based on the decoded barcode.

7. The self-service checkout device according to claim 1 further comprises a guide module, which is operably connected to the design display unit, and the guide module is configured to activate the design display unit to display the design on the detection board, thereby providing visual guidance to the user to optimally place the product on the detection board.

8. The self-service checkout device according to claim 1, further comprising a concave fixing member disposed upright relative to the detection plate, wherein the one or more cameras are mounted on the concave fixing member.

9. The automatic checkout device according to claim 8, wherein: The concave fixing member accommodates a lighting device to illuminate the detection plate.

10. The self-checkout device of claim 1, wherein the one or more cameras include a first camera and a second camera, the first camera and the second camera being oriented at different angles to capture the video footage of the product from multiple perspectives.

11. The self-service checkout device of claim 1, wherein the billing module is further configured to generate a detailed list based on the registered one or more products.

12. The self-service checkout device according to claim 11, further comprising a display screen configured to display the detailed list and the total bill.

13. The self-service checkout device of claim 1, further comprising a management module configured to support updates to the configuration of the self-service checkout device, including updates to a product database of the self-service checkout device.

14. The self-service checkout device of claim 1, wherein the self-service checkout device operates as a stand-alone device.

15. A method implemented by a self-service checkout device, the method comprising: receiving video footage of a detection plate of the self-service checkout device from one or more cameras; detecting the presence of motion in the video recording by processing the video recording; selecting a sequence of video frames within a time interval corresponding to a detection of motion in the video recording; detecting and decoding one or more barcodes visible in the sequence of video frames; calculating a total bill corresponding to the decoded one or more bar codes; as well as The total bill is displayed on a display screen of the self-service checkout device.

16. The method according to claim 15, further comprising: detecting an item visible in the sequence of video frames when one or more barcodes are not visible in the sequence of video frames; Distinguishing detected items from items for sale and items not for sale; and A first reminder is issued upon detection of one or more non-sale items, the first reminder including a message to remove the non-sale items placed on the detection plate of the self-service checkout device.

17. The method according to claim 16, further comprising: determining a distribution of detected sales items on the detection plate of the self-service checkout device; as well as A second reminder is issued upon detecting that the determined distribution of the detected sales items is not suitable.

18. The method according to claim 17, further comprising: cropping, from each video frame in the sequence of video frames, one or more regions substantially surrounding each detected item for sale; generating, based on each of the cropped one or more regions, an embedded representation of a sale item visible in the region; comparing the generated embedding representation with the records of the embedding representation of the product to find the records of the embedding representation of the product that match; determining a price corresponding to the record of the embedded representation of the matching product; For all detected sale items, calculate the total bill as the sum of the determined prices corresponding to the records of the embedded representations of the matching products; as well as The total bill is displayed on the display screen.

19. The method of claim 18, further comprising receiving payment of the master bill.

20. A computer program product having machine-readable instructions stored therein, which when executed by one or more processing units cause the one or more processing units to perform the method according to claim 1.

Images