A method and system for identifying shelf commodities

Abstract

The present invention discloses a method and system for identifying shelf commodities, the method comprising the following steps: collecting a shelf image, the content of the shelf image including a plurality of commodities and a plurality of price tags on the shelf; obtaining a plurality of price tags closest to a target commodity according to the shelf image; determining the price tag corresponding to the target commodity from the plurality of price tags according to the distance between each price tag in the plurality of price tags and the target commodity, and the feature similarity between the commodity corresponding to each price tag and the target commodity. The present invention can reduce the commodity feature retrieval range, improve the commodity identification accuracy, and reduce the time requirement for the commodity feature retrieval algorithm according to the comprehensive judgment of the distance and feature value similarity, thereby improving the difficulty of identifying adjacent commodities with high similarity.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for identifying shelf commodities. Background Art

[0002] With the development of artificial intelligence technology, smart retail has also developed rapidly. In order to achieve a more efficient, better service and better shopping experience, smart retail uses the Internet, Internet of Things, big data, artificial intelligence and other technologies to empower or upgrade existing supermarkets and convenience stores, manage them digitally and intelligently, and optimize the relationship between goods, users and payments, form big data, and further promote digitalization.

[0003] Digital shelves are an important part of the smart new retail business. The requirements of intelligent detection of shelves, intelligent identification of goods, intelligent alarm for out-of-stock, and display supervision optimization have given digital shelves the need for intelligent management. Among them, product identification is an important part of smart shelf management. It is also a prerequisite for applications such as shelf layout analysis and automatic settlement. It is a core technical point to solve offline digitalization. Therefore, the accuracy of product identification is crucial. However, there are huge difficulties in achieving high-precision product identification, mainly due to the wide variety of products, multi-angle differences, and the influence of ambient light. It requires massive data support and accurate retrieval technology. This is a very big obstacle to the real commercial implementation of technology and cannot enable smart retail to form a scale effect.

[0004] In order to achieve high-precision product recognition, the most effective technical means at present is to expand the product feature library, support massive data, and collect and store the image data of a certain product for a long time. That is, a large amount of product data is manually collected to cover as many types of products as possible, multi-angle feature information of products, and even accumulate long-tail data for many years, and finally build a very complete product feature database. On the basis of the feature database, the currently mature image retrieval algorithm (similar to image search) is used to achieve pure product recognition supported by massive product data.

[0005] However, the variety of commodities in the world is huge and complex, and the collection and accumulation of multi-angle feature information of a single category of commodities takes a long time and requires a lot of manpower investment, which is very costly. In addition, the pure visual commodity recognition method has very poor robustness when encountering situations such as reflection, partial occlusion, shadow, color difference, and high similarity between categories, which increases hardware costs and reduces algorithm efficiency, resulting in low algorithm recognition accuracy and difficulty in improving generalization ability, making it difficult to form a scale effect. Summary of the invention

[0006] The present invention provides a method and system for identifying shelf commodities, so as to solve the defects of low recognition accuracy and low efficiency in the prior art.

[0007] The present invention provides a method for identifying shelf commodities, comprising the following steps:

[0008] Acquire a shelf image, wherein the shelf image includes a plurality of commodities and a plurality of price tags on the shelf;

[0009] According to the shelf image, obtaining multiple price tags closest to the target product;

[0010] The price tag corresponding to the target product is determined from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0011] The present invention also provides a system for identifying shelf commodities, comprising:

[0012] A collection module, used for collecting shelf images, wherein the contents of the shelf images include a plurality of commodities and a plurality of price tags on the shelf;

[0013] An acquisition module, used for acquiring a plurality of price tags closest to the target product according to the shelf image;

[0014] The determination module is used to determine the price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0015] The embodiment of the present invention can reduce the product feature retrieval range, improve product recognition accuracy, and reduce the time requirement for the product feature retrieval algorithm based on the comprehensive judgment of distance and feature value similarity, thereby improving the difficulty of identifying adjacent products with high similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a method for identifying shelf merchandise in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of a scene for identifying shelf merchandise in an embodiment of the present invention;

[0018] Figure 3 The figure is a schematic diagram of the structure of a shelf commodity identification system in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The embodiment of the present invention provides a method for identifying shelf commodities, such as Figure 1 As shown, the following steps are included:

[0021] Step 101, collecting shelf images.

[0022] The contents of the shelf image include multiple commodities and multiple price tags on the shelf.

[0023] Step 102: acquiring a plurality of price tags closest to the target product according to the shelf image.

[0024] In this embodiment, after collecting the shelf image, each price tag and each commodity on the shelf can be detected and identified based on the shelf image; each commodity thumbnail can be cut out from the shelf image, and the features of each commodity thumbnail can be extracted.

[0025] Step 103: determining a price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0026] Specifically, the matching score of each price tag can be calculated based on the distance between each price tag among the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product; and the price tag with the highest matching score is determined as the price tag corresponding to the target product.

[0027] In this embodiment, the matching score of each price tag can be calculated using the following formula:

[0028] Score=a*(k-dis)+b*similarity

[0029] Wherein, Score is the matching score, dis is the distance between the target product and the price tag, similarity is the similarity between the feature value of the product corresponding to the price tag and the feature value of the target product, and a, b, k are weight coefficients used to adjust the weights of the distance and similarity in Score;

[0030] It should be noted that the characteristic value of the product corresponding to each price tag is specifically: the product characteristic value with the greatest similarity to the characteristic value of the target product in the product characteristic value set corresponding to the code bound to the price tag. The product characteristic value set corresponding to the code bound to the price tag may include multiple product characteristic values, and the code bound to the price tag may be an EAN (European Article Number) code.

[0031] In addition, after obtaining multiple price tags that are closest to the target product based on the shelf image, it is also possible to determine whether the product feature database contains product feature values ​​corresponding to the codes bound to the multiple price tags; if the product feature database does not contain product feature values ​​corresponding to the codes bound to the multiple price tags, then the price tag corresponding to the target product is determined from the multiple price tags based on the distance between each price tag in the multiple price tags and the target product; if the product feature database contains product feature values ​​corresponding to the codes bound to the multiple price tags, then the price tag corresponding to the target product is determined from the multiple price tags based on the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0032] The embodiment of the present invention can reduce the product feature retrieval range, improve product recognition accuracy, and reduce the time requirement for the product feature retrieval algorithm based on the comprehensive judgment of distance and feature value similarity, thereby improving the difficulty of identifying adjacent products with high similarity.

[0033] The embodiments of the present invention can be applied to a scene including an image acquisition device, a cloud server, and an edge device or a separate server device, wherein the image acquisition device can be a camera, which can be fixed on a shelf or mounted on a ceiling. The edge device or a separate server device can be used to deploy price tag detection, price tag recognition, commodity detection, and commodity feature extraction algorithms, and the cloud server can be used to deploy recognition and matching algorithms.

[0034] Based on the above scenario, an embodiment of the present invention provides a specific implementation process of a method for identifying shelf goods, including the following steps: initialize a feature library of all types of goods; use a camera to collect shelf images; call a price tag detection and recognition algorithm to detect and identify each price tag on the shelf; call a product detection algorithm to detect each product on the shelf and cut out each product thumbnail; call a feature extraction algorithm to extract the features of each product thumbnail; enter a fused product recognition process to cyclically identify each product; for each product, find the nearest price tags to it and calculate the distance dis to each price tag; check whether the EAN codes bound to the nearest price tags have feature values ​​recorded in the initialized product feature library. If the feature value of the EAN code does not exist, match only based on the distance between the product and the price tag; if the feature value of the EAN code exists, match by fusion distance and feature similarity.

[0035] The specific formula for calculating the matching score is as follows:

[0036] Score=a*(k-dis)+b*similarity

[0037] Among them, a, b, k are weight coefficients used to adjust the weights of the distance and similarity in the score; dis is the distance between the product and the price tag, and similarity is the value with the greatest similarity between the feature value of the target product and the feature value set of the product corresponding to the EAN code bound to the price tag.

[0038] like Figure 2 As shown, after calculating the scores between the target product and the nearest several price tags, the price tag corresponding to the maximum score can be determined as the final result of product identification, that is, the EAN code and product name of the target product are consistent with those bound to the matching price tag.

[0039] The embodiment of the present invention integrates the price tag position information, price tag EAN information, commodity position information and commodity feature information on the shelf, comprehensively calculates the matching score, and performs actual matching identification of the commodity according to the maximum score, thereby abandoning pure commodity identification and completing the identification calculation method of price tag position and feature similarity fusion, which can get rid of the difficulties such as high similarity, reflection, shadow, color difference, etc. that the pure commodity identification method cannot cope with, and greatly improve the accuracy of commodity identification on the shelf; and according to the comprehensive judgment of distance and feature value similarity, the search range in the retrieval technology is reduced to two to three EAN code retrieval targets, reducing the time requirement of the commodity feature retrieval algorithm, and realizing high-precision commodity identification on the shelf or in similar scenes in the case of a small amount of commodity feature data or even no feature data, reducing the accumulation of feature data and the complexity of the algorithm, so that the algorithm can be promoted and deployed in multiple scenarios, and has high actual commercial value; when the visual feature cannot be accurately determined, the nearest match or increased weight can be used for matching and identification according to the information of the nearest price tag, thereby improving the robustness of the algorithm.

[0040] like Figure 3 FIG. 1 is a schematic diagram of a system for identifying shelf commodities provided by an embodiment of the present invention, comprising:

[0041] The acquisition module 310 is used to acquire a shelf image, wherein the content of the shelf image includes a plurality of commodities and a plurality of price tags on the shelf.

[0042] The acquisition module 320 is used to acquire a plurality of price tags that are closest to the target product according to the shelf image.

[0043] The determination module 330 is used to determine the price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0044] The determination module 330 includes:

[0045] a calculation submodule, configured to calculate a matching score for each price tag according to a distance between each price tag in the plurality of price tags and the target product, and a feature similarity between the product corresponding to each price tag and the target product;

[0046] The determination submodule is used to determine the price tag with the highest matching score as the price tag corresponding to the target product.

[0047] Specifically, the calculation submodule is used to calculate the matching score of each price tag using the following formula:

[0048] Score=a*(k-dis)+b*similarity

[0049] Wherein, Score is the matching score, dis is the distance between the target product and the price tag, similarity is the similarity between the feature value of the product corresponding to the price tag and the feature value of the target product, and a, b, k are weight coefficients used to adjust the weights of the distance and similarity in Score;

[0050] The characteristic value of the commodity corresponding to each price tag is specifically: the commodity characteristic value with the greatest similarity to the characteristic value of the target commodity in the commodity characteristic value set corresponding to the code bound to the price tag.

[0051] In addition, the above system further comprises:

[0052] A recognition module, used to detect and recognize each price tag and each commodity on the shelf according to the shelf image;

[0053] The extraction module is used to cut out each commodity thumbnail from the shelf image and extract features of each commodity thumbnail.

[0054] A determination module, used to determine whether a commodity feature value corresponding to the codes bound to the multiple price tags is recorded in a commodity feature database;

[0055] Correspondingly, the determination module 330 is specifically used to determine the price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product if the product feature value corresponding to the code bound to the multiple price tags is not recorded in the product feature database; if the product feature value corresponding to the code bound to the multiple price tags is recorded in the product feature database, then determine the price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

[0056] The embodiment of the present invention can reduce the product feature retrieval range, improve product recognition accuracy, and reduce the time requirement for the product feature retrieval algorithm based on the comprehensive judgment of distance and feature value similarity, thereby improving the difficulty of identifying adjacent products with high similarity.

[0057] The steps in the method described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0058] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for identifying shelf commodities, characterized in that: The following steps are involved: Acquire a shelf image, wherein the shelf image includes a plurality of commodities and a plurality of price tags on the shelf; According to the shelf image, obtaining multiple price tags closest to the target product; Determine a price tag corresponding to the target product from the multiple price tags according to the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product; The determining, based on the distance between each price tag in the plurality of price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product, from the plurality of price tags specifically includes: Calculate a matching score for each price tag according to the distance between each price tag in the plurality of price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product; The price tag with the highest matching score is determined as the price tag corresponding to the target product.

2. The method according to claim 1, characterized in that The calculating the matching score of each price tag according to the distance between each price tag of the plurality of price tags and the target product and the feature similarity between the product corresponding to each price tag and the target product specifically includes: The matching score of each price tag is calculated using the following formula: Score=a*(k-dis)+b*similarity Wherein, Score is the matching score, dis is the distance between the target product and the price tag, similarity is the similarity between the feature value of the product corresponding to the price tag and the feature value of the target product, and a, b, k are weight coefficients used to adjust the weights of the distance and similarity in Score; The characteristic value of the commodity corresponding to each price tag is specifically: the commodity characteristic value with the greatest similarity to the characteristic value of the target commodity in the commodity characteristic value set corresponding to the code bound to the price tag.

3. The method according to claim 1, characterized in that After collecting the shelf image, the method further includes: Detect and identify each price tag and each commodity on the shelf according to the shelf image; Each product thumbnail is cut out from the shelf image, and features of each product thumbnail are extracted.

4. The method according to claim 1, characterized in that After acquiring a plurality of price tags closest to the target product according to the shelf image, the method further includes: Determine whether a commodity feature database contains commodity feature values ​​corresponding to the codes bound to the multiple price tags; If the commodity feature value corresponding to the code bound to the multiple price tags is not recorded in the commodity feature database, determining the price tag corresponding to the target commodity from the multiple price tags according to the distance between each price tag in the multiple price tags and the target commodity; The determining, based on the distance between each price tag in the plurality of price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product, from the plurality of price tags specifically includes: If the product feature database records product feature values ​​corresponding to the codes bound to the multiple price tags, then the price tag corresponding to the target product is determined from the multiple price tags based on the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

5. A system for identifying goods on a shelf, characterized in that: include: A collection module, used for collecting shelf images, wherein the contents of the shelf images include a plurality of commodities and a plurality of price tags on the shelf; An acquisition module, used for acquiring a plurality of price tags closest to the target product according to the shelf image; a determination module, configured to determine a price tag corresponding to the target product from the multiple price tags according to a distance between each price tag in the multiple price tags and the target product, and a feature similarity between the product corresponding to each price tag and the target product; The determination module comprises: a calculation submodule, configured to calculate a matching score for each price tag according to a distance between each price tag in the plurality of price tags and the target product, and a feature similarity between the product corresponding to each price tag and the target product; The determination submodule is used to determine the price tag with the highest matching score as the price tag corresponding to the target product.

6. The system according to claim 5, characterized in that The calculation submodule is specifically used to calculate the matching score of each price tag using the following formula: Score=a*(k-dis)+b*similarity Wherein, Score is the matching score, dis is the distance between the target product and the price tag, similarity is the similarity between the feature value of the product corresponding to the price tag and the feature value of the target product, and a, b, k are weight coefficients used to adjust the weights of the distance and similarity in Score; The characteristic value of the commodity corresponding to each price tag is specifically: the commodity characteristic value with the greatest similarity to the characteristic value of the target commodity in the commodity characteristic value set corresponding to the code bound to the price tag.

7. The system according to claim 5, characterized in that Also includes: A recognition module, used to detect and recognize each price tag and each commodity on the shelf according to the shelf image; The extraction module is used to cut out each commodity thumbnail from the shelf image and extract features of each commodity thumbnail.

8. The system according to claim 5, characterized in that Also includes: A determination module, used to determine whether a commodity feature value corresponding to the codes bound to the multiple price tags is recorded in a commodity feature database; The determination module is specifically used to determine the price tag corresponding to the target product from the multiple price tags based on the distance between each price tag in the multiple price tags and the target product if the product feature value corresponding to the code bound to the multiple price tags is not recorded in the product feature database; if the product feature value corresponding to the code bound to the multiple price tags is recorded in the product feature database, determine the price tag corresponding to the target product from the multiple price tags based on the distance between each price tag in the multiple price tags and the target product, and the feature similarity between the product corresponding to each price tag and the target product.

Images