Method and device for product checkout in unmanned store

Abstract

A method and a device for product checkout in an unmanned store are disclosed. A method for product checkout in an unmanned store according to an embodiment of the present invention may comprise the steps of: recognizing, through a product recognition unit, a first product placed on a checkout counter; generating an image recognition result for the first product on the basis of a recognition result for the first product; generating a marker recognition result for a marker attached to the first product on the basis of the recognition result for the first product; comparing the image recognition result and the marker recognition result; and identifying the first product on the basis of a comparison result according to the comparison.

Description

Method and device for calculating products in an unmanned store

[0001] The present invention relates to a method and device for calculating products in an unmanned store.

[0002] The content described below merely provides background information related to one embodiment of the present invention and does not constitute prior art.

[0003] Efforts are being made to automate the entire logistics process—from product production, shipping, transportation, unloading, packaging, storage, and accounting—to reduce human resources and improve accuracy. Automating logistics can reduce costs through reduced manpower and prevent safety accidents. It also shortens logistics time and facilitates systematic management.

[0004] In particular, when calculating products in a store, there is a problem that the time required for calculating products increases as store employees perform barcode scanning on each product using a barcode scanning device, which increases unnecessary waiting time for malicious customers, increases store congestion, and increases store operation management costs due to hiring employees for calculating products.

[0005] Accordingly, there is a need for technology that can quickly and accurately perform calculations for products that customers wish to purchase without human intervention.

[0006] The purpose of the present invention is to provide a method for calculating products in an unmanned store.

[0007] In addition, the present invention aims to recognize a marker attached to a product, compare it with an image recognition result, and identify the product based on the comparison result.

[0008] In addition, the present invention aims to increase the recognition accuracy of products through multiple learning models.

[0009] In order to achieve the above-described object, an unmanned store product checkout method according to one embodiment of the present invention may include a step of recognizing a first product put into a checkout counter through a product recognition unit, a step of generating an image recognition result for the first product based on a recognition result for the first product, a step of generating a marker recognition result for a marker attached to the first product based on the recognition result for the first product, a step of comparing the image recognition result and the marker recognition result, and a step of identifying the first product based on a comparison result according to the comparison.

[0010] The step of recognizing the first product may include a step of identifying the size and shape of the first product using a camera, a step of measuring the weight of the first product using a weight sensor, and a step of measuring the temperature of the first product using a thermal imaging camera.

[0011] The step of identifying the size and shape of the first product may include the step of acquiring cross-sectional images of the first product using a plurality of cameras and the step of identifying the size and shape of the first product based on the cross-sectional images.

[0012] The step of generating an image recognition result for the first product based on the recognition result for the first product may include a step of selecting at least one learning model for generating an image recognition result for the first product based on the recognition result for the first product, a step of acquiring a first image for the first product through an image acquisition unit, and a step of generating an image recognition result for the first product based on the at least one learning model and the first image.

[0013] The step of selecting at least one learning model may include the step of calculating a first criterion corresponding to the recognition result for the first product, and the step of selecting at least one learning model from among a plurality of learning models based on the first criterion.

[0014] The step of generating an image recognition result for the first product based on the at least one learning model and the first image may include the step of producing a second image corresponding to the recognition result using the at least one learning model, the step of comparing the first image and the second image using image matching, and the step of identifying the first product as the second product having the highest degree of similarity according to the result of the image matching.

[0015] The step of generating the marker recognition result includes a step of identifying a marker for the first product from the cross-sectional images based on the recognition result for the first product, a step of receiving product information included in the marker based on the identified marker, and a step of generating a marker recognition result for the first product based on the product information, wherein the product information may include a type of product corresponding to the first product, a product name, a price, and a weight of the product.

[0016] The step of identifying the first product may include a step of identifying the first product using the marker recognition result when the image recognition result and the marker recognition result correspond as a comparison result for the comparison.

[0017] The above learning model is a model that is learned for each of a plurality of criteria for products classified based on the range of size, shape, weight, and temperature of the products, and the learning may include supervised learning that enables identification as the correct product when a specific product passes through the product recognition unit.

[0018] In addition, an unmanned store product calculation device according to one embodiment of the present invention includes a memory having at least one program recorded thereon and a processor executing the program, wherein the program may include instructions for performing a step of recognizing a first product put into a checkout counter through a product recognition unit, a step of generating an image recognition result for the first product based on a recognition result for the first product, a step of generating a marker recognition result for a marker attached to the first product based on the recognition result for the first product, a step of comparing the image recognition result and the marker recognition result, and a step of identifying the first product based on a comparison result according to the comparison.

[0019] According to the present invention, a method for calculating products in an unmanned store can be provided.

[0020] In addition, according to the present invention, a marker attached to a product can be recognized, compared with an image recognition result, and the product can be identified based on the comparison result.

[0021] Additionally, according to the present invention, the accuracy of product recognition can be improved through multiple learning models.

[0022] FIG. 1 is a block diagram showing entities for calculating products in an unmanned store according to one embodiment of the present invention.

[0023] Figure 2 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0024] Figure 3 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0025] Figure 4 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0026] Figure 5 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0027] Figure 6 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0028] Figure 7 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0029] Figure 8 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0030] FIG. 9 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0031] The present invention will be described in detail with reference to the attached drawings. Herein, repetitive descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. The embodiments of the present invention are provided to more fully explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

[0032] Although "first" or "second" are used to describe various components, these components are not limited by such terms. Such terms may only be used to distinguish one component from another. Accordingly, a first component referred to below may also be a second component within the technical scope of the present invention.

[0033] Throughout the specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0034] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0035] FIG. 1 is a block diagram showing entities for calculating products in an unmanned store according to one embodiment of the present invention.

[0036] Referring to FIG. 1, the entities for calculating unmanned store products according to one embodiment of the present invention include an unmanned store product calculation device (110) and a product recognition unit (120).

[0037] The unmanned store product calculation device (110) may refer to a device that receives a product recognition result from a product recognition unit (120) and generates an image recognition result and a marker recognition result for the product based on the recognition result.

[0038] The unmanned store product calculation device (110) may be a device that compares the image recognition result and the marker recognition result and identifies the product based on the comparison result.

[0039] The product recognition unit (120) may be a device that identifies the size and shape of a product using a camera, measures the weight of the product using a weight sensor, measures the temperature of the product using a thermal imaging camera, and provides the product recognition result to an unmanned store product calculation device (110).

[0040] The unmanned store product calculation device (110) and product recognition unit (120) can be interconnected through a communication network.

[0041] A communications network refers to a connection path that enables data to be transmitted and received between the above entities. For example, a communications network may encompass wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communications. However, the scope of communications networks applicable to the present invention is not limited thereto.

[0042] Figure 2 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0043] Referring to FIG. 2, the unmanned store product calculation method according to one embodiment of the present invention can first recognize a first product placed at a checkout counter through a product recognition unit (S210).

[0044] Next, an image recognition result for the first product can be generated based on the recognition result for the first product (S220).

[0045] Here, the image recognition result may mean the result of recognizing the product using the external image, weight, temperature, etc. of the first product.

[0046] Next, based on the recognition result for the first product, a marker recognition result for a marker attached to the first product can be generated (S230).

[0047] Here, the marker recognition result may mean the result of recognizing the product through the marker attached to the first product.

[0048] Next, the image recognition result and the marker recognition result can be compared (S240).

[0049] Next, based on the comparison result according to the above comparison, the first product can be identified (S250).

[0050] Figure 3 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0051]

[0052] *Referring to FIG. 3, the unmanned store product calculation method according to one embodiment of the present invention can first identify the size and shape of the first product using a camera (S310).

[0053] Next, the weight of the first product can be measured using a weight sensor (S320).

[0054] Next, the temperature of the first product can be measured using a thermal imaging camera (S330).

[0055] At this time, the temperature of the product can be measured to determine whether the first product is a refrigerated product or a frozen food.

[0056] Figure 4 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0057] Referring to FIG. 4, the unmanned store product calculation method according to one embodiment of the present invention can first acquire cross-sectional images of the first product using a plurality of cameras (S410).

[0058] Next, based on the cross-sectional images, the size and shape of the first product can be identified (S420).

[0059] According to one embodiment, each cross-sectional image of the first product can be acquired using a plurality of cameras installed in different directions and angles.

[0060] The unmanned store product calculation device can calculate the size of the first product by considering the angle difference between each camera and the distance between each camera and the first product.

[0061] Figure 5 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0062] Referring to FIG. 5, the unmanned store product calculation method according to one embodiment of the present invention may first select at least one learning model for generating an image recognition result for the first product based on the recognition result for the first product (S510).

[0063] Here, the learning model is a model learned when registering the first product, and may include models learned using different learning models for each of multiple categories.

[0064] Next, a first image for the first product can be acquired through an image acquisition unit (S520).

[0065] Next, based on the at least one learning model and the first image, an image recognition result for the first product can be generated (S530).

[0066] Here, the learning model may include a model learned for each of a plurality of criteria for products classified based on the range of size, shape, weight, and temperature of the products.

[0067] Additionally, the above learning may include supervised learning, which identifies a specific product as the correct one when it passes through the product recognition unit. Learning is performed by inputting a preset number of products that meet each criterion. If the product recognition rate exceeds the preset recognition rate, the learning process is marked as complete and further training can be discontinued. This prevents future recognition errors due to incorrectly recognized products being learned during product recognition.

[0068] Figure 6 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0069] Referring to FIG. 6, the unmanned store product calculation method according to one embodiment of the present invention can first calculate a first criterion corresponding to the recognition result for the first product (S610).

[0070] Next, based on the first criterion, at least one learning model among a plurality of learning models can be selected (S620).

[0071] Here, the first criterion may include a criterion for at least one category, and the learning model may select a learning model corresponding to each category.

[0072] Figure 7 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0073] Referring to FIG. 7, the unmanned store product calculation method according to one embodiment of the present invention can first produce a second image corresponding to the recognition result using at least one learning model (S710).

[0074] Next, the first image and the second image can be compared using image matching (S720).

[0075] Next, the first product can be identified as the second product with the highest similarity according to the results of the image matching (S730).

[0076] In one embodiment, a second image corresponding to the recognition result for each learning model can be generated. The unmanned store product checkout device can compare the first image with the second image for each learning model and calculate the similarity for each learning model.

[0077] Next, the first product can be determined as the second product with the highest similarity based on the image matching results for each learning model.

[0078]

[0079] *Unmanned store product checkout devices don't select a single learning model based on recognition results. Instead, they utilize multiple learning models corresponding to categories based on the recognition results. They compare the images produced by each learning model to determine the product with the highest similarity. This reduces misrecognition of products and improves recognition accuracy.

[0080] Figure 8 is a flowchart illustrating an unmanned store product calculation method according to one embodiment of the present invention.

[0081] Referring to FIG. 8, the unmanned store product calculation method according to one embodiment of the present invention first identifies a marker for the first product from the cross-sectional images based on the recognition result for the first product (S810).

[0082] Here, markers of different types may be attached depending on the classification of the first product.

[0083] Next, based on the identified marker, product information contained within the marker can be received (S820).

[0084] Here, the product information may include the type of product, product name, price, and weight of the product corresponding to the first product.

[0085] Next, based on the above product information, a marker recognition result for the first product can be generated (S830).

[0086] According to one embodiment, in step S250, if the image recognition result and the marker recognition result correspond as a comparison result for the comparison, the first product can be identified using the marker recognition result.

[0087] As an optional embodiment, when identifying a marker for a first product, a marker recognition score may be calculated, and whether the marker is recognized may be determined based on the marker recognition score.

[0088] Specifically, a marker recognition score for the first product can be calculated using an artificial intelligence learning model that learns the shape of the marker according to the angle of the cross-sectional images.

[0089] At this time, the marker recognition score can identify and reflect in the score any deformation of the marker shape due to crumpling, folding, alteration, damage, etc. of the marker.

[0090] Next, if the marker recognition score is greater than or equal to the first reference score, the marker can be determined to have been recognized.

[0091] At this time, if the marker recognition score is lower than the first reference score, the marker is judged not to have been recognized, and the user and administrator can be requested to reissue or re-recognize the marker.

[0092] Here, the unmanned store product checkout device can determine the main cause of the low marker recognition score and provide guidance.

[0093] If the primary cause is tampering or damage to the marker, you can request the administrator to reissue the marker. Additionally, if the primary cause is crumpling or folding, you can request the user to re-recognize the marker.

[0094] Through this, the main cause of the marker not being recognized can be determined, preventing users from gaining unfair advantage through tampering or damage, and guiding users to correctly recognize the marker.

[0095] As an optional embodiment, if the marker recognition score is higher than a second criterion score that is higher than the first criterion score, it may be determined as a best practice for marker recognition.

[0096] Specifically, the unmanned store product checkout device can store a first image captured by the user recognizing the marker as a best practice when the marker recognition score is higher than the second reference score.

[0097] At this time, the first product that recognizes the marker and the first image can be matched and stored.

[0098] Next, if another user recognizes the marker for the first product and the marker recognition score is lower than the first reference score, the first video can be provided as feedback while requesting the other user to recognize the marker again.

[0099] At this time, the marker part in the first video can be highlighted in a different color to make it more visible to other users.

[0100] This has the effect of providing feedback to other users using best practices for marker recognition on a product-by-product basis.

[0101] As an optional embodiment, if the cross-sectional images of the product include an identification code, the product can be identified through this.

[0102] Specifically, after acquiring cross-sectional images, it is possible to determine whether the cross-sectional images include an identification code for the first product.

[0103] Here, the identification code is included on the outer packaging of the product and may contain identification information about the product. For example, the identification code may include a barcode on the product or a QR code included on the outer packaging.

[0104] Next, if it is determined that an identification code is included, the identification code can be recognized and the first product can be determined as a third product corresponding to the identification code.

[0105] FIG. 9 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0106] An unmanned store product calculation device according to one embodiment of the present invention can be implemented in a computer system (1000) such as a computer-readable recording medium.

[0107] Referring to FIG. 9, a computer system (1000) may include one or more processors (1010), memory (1030), a user interface input device (1040), a user interface output device (1050), and storage (1060) that communicate with each other via a bus (1020). In addition, the computer system (1000) may further include a network interface (1070) connected to a network (1080). The processor (1010) may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory (1030) or the storage (1060). The memory (1030) and the storage (1060) may be various forms of volatile or non-volatile storage media. For example, the memory may include ROM (1031) or RAM (1032).

[0108] The specific implementations described in the present invention are merely exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely exemplary functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0109] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

Claims

1. A step of recognizing the first product placed at the checkout counter through the product recognition unit; A step of generating an image recognition result for the first product based on the recognition result for the first product; A step of generating a marker recognition result for a marker attached to the first product based on the recognition result for the first product; A step of comparing the image recognition result and the marker recognition result; and A step of identifying the first product based on the comparison result according to the above comparison. A method for calculating products in an unmanned store, including:

2. In paragraph 1, The step of recognizing the above first product is: A step of identifying the size and shape of the first product using a camera; A step of measuring the weight of the first product using a weight sensor; and A step of measuring the temperature of the first product using a thermal imaging camera A method for calculating products in an unmanned store, including:

3. In paragraph 2, The step of identifying the size and shape of the first product is: A step of acquiring cross-sectional images of the first product using multiple cameras; and A step of identifying the size and shape of the first product based on the cross-sectional images A method for calculating products in an unmanned store, including:

4. In paragraph 3, The step of generating an image recognition result for the first product based on the recognition result for the first product is as follows: A step of selecting at least one learning model for generating an image recognition result for the first product based on the recognition result for the first product; A step of acquiring a first image of the first product through an image acquisition unit; and A step of generating an image recognition result for the first product based on the at least one learning model and the first image. A method for calculating products in an unmanned store, including:

5. In paragraph 4, The step of selecting at least one learning model comprises: A step of calculating a first criterion corresponding to the recognition result for the first product; and A step of selecting at least one learning model from among a plurality of learning models based on the first criterion above. A method for calculating products in an unmanned store, including:

6. In paragraph 5, The step of generating an image recognition result for the first product based on the at least one learning model and the first image is as follows: A step of producing a second image corresponding to the recognition result using at least one learning model; A step of comparing the first image and the second image using image matching; and A step of identifying the first product as the second product having the highest similarity based on the results of the image matching. A method for calculating products in an unmanned store, including:

7. In paragraph 6, The step of generating the above marker recognition result is: A step of identifying a marker for the first product from the cross-sectional images based on the recognition result for the first product; A step of receiving product information contained within the marker based on the identified marker; and A step of generating a marker recognition result for the first product based on the above product information. Including, The above product information is, The type, product name, price, and weight of the product corresponding to the above first product A method for calculating products in an unmanned store, including:

8. In paragraph 7, The step of identifying the above first product is: A step of identifying the first product using the marker recognition result when the image recognition result and the marker recognition result correspond as a comparison result for the above comparison. A method for calculating products in an unmanned store, including:

9. In paragraph 4, The above learning model is, A model trained on one of multiple criteria for products classified based on the range of product size, shape, weight, and temperature, The above learning is a method for calculating products in an unmanned store, including supervised learning that enables identification of a specific product as the correct product when it passes through a product recognition unit.

10. Memory in which at least one program is recorded; and Processor that runs the above program Including, The above program is, A step of recognizing the first product being placed at the checkout counter through the product recognition unit; A step of generating an image recognition result for the first product based on the recognition result for the first product; A step of generating a marker recognition result for a marker attached to the first product based on the recognition result for the first product; A step of comparing the image recognition result and the marker recognition result; and A step of identifying the first product based on the comparison result according to the above comparison. An unmanned store product checkout device comprising commands for performing the following.

Images