Method, system, server, and robot for product recognition

Abstract

Disclosed in the present invention are a method, system, server, and robot for product recognition. The method comprises: an image collection device collecting a shelf image to be recognized; a server performing product segmentation and recognition on the shelf image to be recognized, and matching, with features in a feature library, each product feature extracted from the image; for products of which the matching degree is greater than a first threshold and less than a second threshold, generating a list of top N candidate products; a supermarket executing the step of determining whether a product matching each candidate product in the list and a bound electronic label are present in a supermarket, and if yes, sending a display command to the bound label, and sending a recollection command to the collection device; if the command has been received, displaying the label in a preset display mode; the collection device recollecting a shelf image to be recognized; and the server recognizing in the recollected image whether there is a label displayed in the display mode, and if yes, determining a final recognition result indicating that a product having a preliminary recognition result is the product bound to the label. The present invention can improve the accuracy of product recognition.

Description

Product identification methods, systems, servers, and robots

[0001] Related applications

[0002] This application claims priority to Chinese Patent Application No. 202411194687.7, filed on August 28, 2024, and incorporates the entire contents of the aforementioned patent application as part of this application. Technical Field

[0003] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, server, and robot for product recognition. Background Technology

[0004] This section is intended to provide background or context for embodiments of the present invention. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0005] Currently, electronic shelf labels are replacing traditional paper labels in large supermarkets worldwide. Besides displaying basic information, electronic shelf labels can be used for many applications, such as rapid order picking, out-of-stock management, quick inventory checks, and human-computer interaction with users. Improving the accuracy of product recognition is significant in supermarkets equipped with electronic shelf labels, as it involves cameras and robots moving and taking pictures in the shelf area, identifying products to construct product display maps, and monitoring out-of-stock status. Currently, the accuracy of product recognition is low. Summary of the Invention

[0006] This invention provides a product identification method applied to a first system to improve the accuracy of product identification. The method includes:

[0007] The image acquisition device captures images of the shelf to be identified and sends them to the image recognition server;

[0008] The image recognition server segments and recognizes goods in the shelf image to be recognized, obtaining a preliminary recognition result for each goods. Features of each goods with preliminary recognition results are extracted from the shelf image. Each extracted feature is matched with features in a preset product feature library. For goods with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list with the top N matching degree values ​​is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, a search command is generated to find goods matching the candidate product information and their associated electronic shelf tags in the first supermarket, and the search command is sent to the first supermarket.

[0009] When the first supermarket receives a search command, it performs the following steps for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command indicating the preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the image of the shelf to be identified through the image recognition server.

[0010] If the electronic shelf label receives a command instructing a preset display mode, it will display the label in that preset display mode.

[0011] When the image acquisition device receives a re-acquisition command, it re-acquisitions the image of the shelf to be identified and sends the re-acquisitioned image of the shelf to be identified to the image recognition server.

[0012] The image recognition server identifies whether there are electronic shelf labels displayed in a preset display mode in the re-captured shelf images. If so, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product recognition result that the product with preliminary recognition results is the product bound to the electronic shelf label.

[0013] This invention also provides a product recognition method applied to an image recognition server to improve the accuracy of product recognition. The method includes:

[0014] The product segmentation and recognition are performed on the shelf image to be identified to obtain a preliminary identification result for each product. The features of each product with a preliminary identification result are extracted from the shelf image to be identified. The shelf image to be identified is sent by an image acquisition device.

[0015] Each extracted feature is matched with product features in a pre-defined product feature library;

[0016] For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold.

[0017] A search command is generated based on the candidate product list to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket. The search command is sent to the first supermarket. Upon receiving the search command, the first supermarket performs the following steps for each candidate product in the candidate product list: to determine if a product matching the candidate product information and its associated electronic shelf label exists in the first supermarket. If so, a command instructing a preset display mode is sent to the corresponding associated electronic shelf label. The electronic shelf label displays the product in the preset display mode if it receives the command. A re-capture command is sent to the image acquisition device to re-capture the shelf image to be identified. Upon receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified and sends the re-captured shelf image to the image recognition server.

[0018] Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired images of the shelves to be identified;

[0019] If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

[0020] This invention also provides a product identification method applied to a first supermarket to improve the accuracy of product identification. The method includes:

[0021] The system receives a search command from an image recognition server, which generates a search order based on a candidate product list to find products matching the candidate product information and their associated electronic shelf labels in a first supermarket. The image recognition server performs product segmentation and recognition on the shelf image to be identified, obtaining preliminary recognition results for each product. It then extracts features from each product with preliminary recognition results from the shelf image. Each extracted feature is matched against product features in a preset product feature library. For products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list is generated, with N being a positive integer greater than 1, and the second matching degree threshold being greater than the first matching degree threshold. Based on the candidate product list, a search command is generated to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket, and this search command is sent to the first supermarket. The shelf image to be identified is provided by an image acquisition device.

[0022] Upon receiving a search command, for each candidate product in the candidate product list, the step is to determine whether there is a product in the first supermarket that matches the candidate product information and its associated electronic shelf label.

[0023] If present, a command instructing a preset display mode is sent to the corresponding bound electronic shelf label; a re-capture command to re-capture the shelf image to be identified is sent to the image acquisition device via the image recognition server; the electronic shelf label is used to display the shelf in the preset display mode if it receives the command instructing a preset display mode; the image acquisition device is also used to re-capture the shelf image to be identified when it receives the re-capture command, and send the re-captured shelf image to be identified to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, to determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and to obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0024] This invention also provides a product identification method applied to a second system to improve the accuracy of product identification. The method includes:

[0025] The second robot collects images of the shelves to be identified; it segments and identifies the products in the images to be identified, obtaining preliminary identification results for each product, and extracts features of each product with preliminary identification results from the images of the shelves to be identified; it matches each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, it generates a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; based on the candidate product list, it generates a search command to find products in the first supermarket that match the candidate product information and their associated electronic shelf tags, and sends the search command to the second supermarket.

[0026] When the second supermarket receives a search command, it performs the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command to the corresponding bound electronic shelf label to indicate the preset display mode; and sends a command to the second robot to re-capture the image of the shelf to be identified.

[0027] If the electronic shelf label receives a command instructing a preset display mode, it will display the label in that preset display mode.

[0028] When the second robot receives the re-collection command, it re-collects the image of the shelf to be identified. In the re-collected image of the shelf to be identified, it identifies whether there is an electronic shelf label displayed in a preset display mode. If there is, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0029] This invention also provides a product recognition method applied to a second robot to improve the accuracy of product recognition. The method includes:

[0030] Acquire images of the shelf to be identified;

[0031] The product segmentation and recognition are performed on the shelf image to be identified to obtain a preliminary recognition result for each product. The features of each product with a preliminary recognition result are extracted from the shelf image to be identified.

[0032] Each extracted feature is matched with product features in a pre-defined product feature library;

[0033] For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold.

[0034] Based on the candidate product list, a search command is generated to locate products matching the candidate product information and their associated electronic shelf labels in the second supermarket. This search command is then sent to the second supermarket. Upon receiving the search command, the second supermarket, for each candidate product in the candidate product list, performs the following steps: Does the second supermarket have a product matching the candidate product information and its associated electronic shelf label? If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; it also sends a command to re-capture the image of the shelf to be identified to the second robot. The electronic shelf label, upon receiving the command indicating the preset display mode, displays the product in the preset display mode.

[0035] Upon receiving a re-acquisition command, the image of the shelf to be identified is re-acquired;

[0036] Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired images of the shelves to be identified;

[0037] If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

[0038] This invention also provides a first system for product identification to improve the accuracy of product identification. The first system includes:

[0039] The image acquisition device is used to acquire images of the shelf to be identified and send them to the image recognition server; when a re-acquisition command is received, the image of the shelf to be identified is re-acquired and sent to the image recognition server.

[0040] An image recognition server is used to segment and recognize goods in the shelf image to be recognized, obtain a preliminary recognition result for each goods, extract features of each goods with preliminary recognition results from the shelf image to be recognized, match each extracted feature with goods features in a preset goods feature library, generate a candidate goods list with the top N matching scores for goods with a matching score greater than a first matching score threshold and less than a second matching score threshold, where N is a positive integer greater than 1 and the second matching score threshold is greater than the first matching score threshold, generate a search command based on the candidate goods list to find goods and their associated electronic shelf labels that match the candidate goods information in the first supermarket, and send the search command to the first supermarket, identify whether there are electronic shelf labels displayed in a preset display mode in the shelf image to be recognized, and if so, determine that the current candidate goods belong to the goods bound to the corresponding electronic shelf labels, and obtain the final goods recognition result that the goods with preliminary recognition results are the goods bound to the electronic shelf labels.

[0041] The first supermarket terminal, upon receiving a search command, performs the following steps for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the shelf image to be identified through the image recognition server.

[0042] Electronic shelf labels are used to display items in a preset display mode if a command is received instructing the display mode to be preset.

[0043] This invention also provides an image recognition server for product identification to improve the accuracy of product identification. The image recognition server includes:

[0044] The first feature extraction unit is used to segment and identify goods in the shelf image to be identified, obtain a preliminary identification result for each goods, and extract the features of each goods with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device;

[0045] The first matching processing unit is used to match each extracted feature with the product features in the preset product feature library;

[0046] The first candidate product list determination unit is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold.

[0047] The first generation unit is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket terminal. Upon receiving the search command, the first supermarket terminal performs the step of checking whether a product matching the candidate product information and its associated electronic shelf label exists in the first supermarket for each candidate product in the candidate product list. If it does, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label. The electronic shelf label is used to display the product in the preset display mode if it receives the command indicating the preset display mode. A re-capture command is sent to the image acquisition device to re-capture the shelf image to be identified. The image acquisition device, upon receiving the re-capture command, re-captures the shelf image to be identified and sends the re-captured shelf image to the image recognition server.

[0048] The first label detection unit is used to identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf image to be identified;

[0049] The first product identification unit is used to determine, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0050] This invention also provides a first supermarket terminal for product recognition to improve the accuracy of product recognition. The first supermarket terminal includes:

[0051] The first acquisition unit is used to acquire a search command sent by the image recognition server, which generates a search command based on a candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket. The image recognition server is used to segment and recognize products in the shelf image to be recognized, obtain preliminary recognition results for each product, and extract features of each product with preliminary recognition results from the shelf image to be recognized. Each extracted feature is matched with product features in a preset product feature library. For products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list with matching degree values ​​ranking in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. The search command is generated based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and the search command is sent to the first supermarket. The shelf image to be recognized is sent by the image acquisition device.

[0052] The first product detection unit is used to perform the step of whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list when a search command is received.

[0053] The first sending unit is configured to, if present, send a command indicating a preset display mode to the corresponding bound electronic shelf label; and send a re-capture command to the image acquisition device via the image recognition server to re-capture the image of the shelf to be identified; the electronic shelf label is configured to display the shelf in the preset display mode if it receives the command indicating the preset display mode; the image acquisition device is further configured to, upon receiving the re-capture command, re-capture the image of the shelf to be identified and send the re-captured image to the image recognition server; the image recognition server is further configured to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured image of the shelf to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product identification result that the product with preliminary identification result is the product bound to the electronic shelf label.

[0054] This invention also provides a second product recognition system to improve the accuracy of product recognition. The second system includes:

[0055] The second robot is used to collect images of shelves to be identified; to segment and identify products in the images of shelves to be identified, and to obtain preliminary identification results for each product; to extract features of each product with preliminary identification results from the images of shelves to be identified; to match each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, to generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; to generate a search command based on the candidate product list to find products in the second supermarket that match the candidate product information and their associated electronic shelf labels, and to send the search command to the second supermarket; when a re-collection command is received, to re-collect images of shelves to be identified, to identify whether there are electronic shelf labels displayed in a preset display mode in the re-collected images of shelves to be identified, and if so, to determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and to obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label;

[0056] The second supermarket terminal, upon receiving a search command, performs the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its associated electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; and sends a command to the second robot to re-capture the image of the shelf to be identified.

[0057] Electronic shelf labels are used to display items in a preset display mode if a command is received instructing the display mode to be preset.

[0058] This invention also provides a second robot for product recognition to improve the accuracy of product recognition. The second robot includes:

[0059] The acquisition unit is used to acquire images of the shelf to be identified; upon receiving a re-acquisition command, it re-acquisitions the images of the shelf to be identified.

[0060] The second feature extraction unit is used to segment and identify the goods in the initially acquired shelf image to be identified, obtain the preliminary identification result of each goods, and extract the features of each goods with the preliminary identification result from the shelf image to be identified.

[0061] The second matching processing unit is used to match each extracted feature with the product features in the preset product feature library;

[0062] The second candidate product list determination unit is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold.

[0063] The second generation unit is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the second supermarket, and send the search command to the second supermarket terminal. Upon receiving the search command, the second supermarket terminal performs the following steps for each candidate product in the candidate product list: to determine if a product matching the candidate product information and its associated electronic shelf label exists in the second supermarket. If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; and sends a command to re-capture the image of the shelf to be identified to the second robot. The electronic shelf label is used to display the product in the preset display mode if it receives the command indicating the preset display mode.

[0064] The second label detection unit is used to identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf image to be identified;

[0065] The second product identification unit is used to determine, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0066] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for identifying goods.

[0067] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described product identification method.

[0068] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying goods.

[0069] In this embodiment of the invention, the product identification scheme, compared with existing technologies with low product identification accuracy, involves: an image acquisition device acquiring an image of a shelf to be identified and sending it to an image recognition server; the image recognition server performing product segmentation and identification on the shelf image to obtain a preliminary identification result for each product, and extracting features of each product with a preliminary identification result from the shelf image; matching each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generating a candidate product list with matching degree values ​​ranking in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generating a search command based on the candidate product list to find products and their associated electronic shelf tags that match the candidate product information in the first supermarket, and sending the search command to the first supermarket; upon receiving the search command, the first supermarket, for the candidate product list... For each candidate product in the table, the process involves checking if a product matching the candidate product information and its associated electronic shelf label exists in the first supermarket. If so, a command instructing a preset display mode is sent to the corresponding electronic shelf label. A re-capture command to re-capture the shelf image to be identified is sent to the image acquisition device via the image recognition server. If the electronic shelf label receives the command instructing a preset display mode, it displays the product in that mode. Upon receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified and sends the re-captured image to the image recognition server. The image recognition server identifies whether an electronic shelf label displayed in the preset display mode exists in the re-captured shelf image. If so, it determines that the current candidate product belongs to the product associated with the corresponding electronic shelf label, thus obtaining the final product identification result: the product with preliminary identification results is the product associated with the electronic shelf label. This improves the accuracy of product identification. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0071] Figure 1 is a flowchart illustrating the product identification method applied to the first system in an embodiment of the present invention;

[0072] Figure 2 is a schematic diagram of the dual threshold values ​​in an embodiment of the present invention;

[0073] Figure 3 is a schematic diagram of the product list between MinCV and MaxCV outputs in an embodiment of the present invention;

[0074] Figure 4 is a schematic diagram of sending a command based on the ESL corresponding to the product list to find the ESL corresponding to this shelf in an embodiment of the present invention.

[0075] Figure 5 is a schematic diagram of the result of determining the product identification based on finding the corresponding ESL in an embodiment of the present invention;

[0076] Figure 6 is a flowchart illustrating the product recognition method applied to an image recognition server in an embodiment of the present invention;

[0077] Figure 7 is a flowchart illustrating the product identification method applied to the first supermarket in an embodiment of the present invention;

[0078] Figure 8 is a flowchart illustrating the product identification method applied to the second system in an embodiment of the present invention;

[0079] Figure 9 is a flowchart illustrating the product recognition method applied to the second robot in an embodiment of the present invention;

[0080] Figure 10 is a schematic diagram of the structure of the first product recognition system in an embodiment of the present invention;

[0081] Figure 11 is a schematic diagram of the structure of the image recognition server for commodity recognition in an embodiment of the present invention;

[0082] Figure 12 is a schematic diagram of the structure of the first supermarket terminal for product identification in an embodiment of the present invention;

[0083] Figure 13 is a schematic diagram of the structure of the second product recognition system in an embodiment of the present invention;

[0084] Figure 14 is a schematic diagram of the structure of the second robot for product recognition in an embodiment of the present invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0086] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0087] The accuracy of direct product recognition in existing technologies is not high, with an accuracy of only about 70%.

[0088] In the first system of this invention, an image acquisition device acquires an image of the shelf to be identified, and the image recognition server interacts with the supermarket and electronic tags. Candidate products with a matching degree between "MinCV" and "MaxCV" are searched and confirmed in the supermarket, thereby improving the accuracy of product identification. The product identification method applied to the first system is described in detail below.

[0089] Figure 1 is a flowchart illustrating the product identification method applied to the first system in an embodiment of the present invention. As shown in Figure 1, the method includes the following steps:

[0090] Step 101: The image acquisition device acquires an image of the shelf to be identified and sends it to the image recognition server;

[0091] Step 102: The image recognition server segments and recognizes the products in the shelf image to be recognized, obtains the preliminary recognition result of each product, and extracts the features of each product with the preliminary recognition result from the shelf image to be recognized; matches each extracted feature with the product features in the preset product feature library; for products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generates a candidate product list with the top N matching degree values, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command based on the candidate product list to find products that match the candidate product information and their bound electronic shelf tags in the first supermarket, and sends the search command to the first supermarket.

[0092] Step 103: When the first supermarket receives the search command, for each candidate product in the candidate product list, it performs the step of whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command indicating the preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the image of the shelf to be identified through the image recognition server.

[0093] Step 104: If the electronic shelf label receives a command instructing a preset display mode, it shall display the information in the preset display mode.

[0094] Step 105: When the image acquisition device receives the re-acquisition command, it re-acquisitions the image of the shelf to be identified and sends the re-acquisitioned image of the shelf to be identified to the image recognition server.

[0095] Step 106: The image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-acquired shelf image. If so, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product recognition result that the product with preliminary recognition results is the product bound to the electronic shelf label.

[0096] The product recognition method provided in this embodiment of the invention operates as follows: an image acquisition device acquires an image of a shelf to be recognized and sends it to an image recognition server; the image recognition server performs product segmentation and recognition on the shelf image to obtain a preliminary recognition result for each product, and extracts features of each product with a preliminary recognition result from the shelf image; each extracted feature is matched with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list with matching degree values ​​ranking in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; a search command is generated based on the candidate product list to find products in the first supermarket that match the candidate product information and their associated electronic shelf tags, and the search command is sent to the first supermarket; when the first supermarket receives the search command, it performs a search on each product in the candidate product list... The process involves determining whether a candidate product exists in the first supermarket and whether it is associated with a corresponding electronic shelf label. If so, a command instructing a preset display mode is sent to the corresponding electronic shelf label. A re-capture command for re-enhancing the shelf image to be identified is sent from the image recognition server to the image acquisition device. If the electronic shelf label receives the command instructing the preset display mode, it displays the product in that mode. Upon receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified and sends the re-captured image to the image recognition server. The image recognition server identifies whether an electronic shelf label displayed in the preset display mode exists in the re-captured shelf image. If so, it determines that the current candidate product belongs to the product associated with that corresponding electronic shelf label, thus obtaining the final product identification result: the product with the preliminary identification result is the product associated with the electronic shelf label.

[0097] Compared with existing technologies that suffer from low product recognition accuracy, the product recognition method provided in this invention can improve product recognition accuracy. The product recognition method will be described in detail below with reference to Figures 2 to 5.

[0098] The product identification method provided in this embodiment of the invention may include the following steps:

[0099] In step 101 above, the shelf camera (image acquisition device) can be fixed opposite or diagonally above the shelf being photographed, or mounted on the first robot moving through the supermarket. That is, in one embodiment, the image acquisition device is mounted on the shelf or on the first robot moving through the supermarket. After the camera captures an image of the shelf (shelf image), it sends it to the image recognition server, i.e., step 101 above.

[0100] In step 102 above, the image recognition server performs product segmentation and recognition to obtain each product in the shelf image and its preliminary recognition result. The image recognition server can use a pre-trained deep learning network model (the input of which can be the shelf image) to extract the features of each product in the image and match the extracted features with the features in the preset product feature library in the image recognition server (this feature library stores all features of all products, i.e., it has features corresponding to each product in all products). Taking a product in the shelf image as an example, for this product, the TOP-M recognition result will be returned after matching. By setting a dual threshold: the first matching degree threshold MinCV (Minimum Confidence Value) and the second matching degree threshold MaxCV (Maximum Confidence Value), the confidence level of the product recognition result can be classified. A schematic diagram of the dual threshold is shown in Figure 2. For all matching results (matching degree values) that are lower than or equal to the MinCV threshold, the product is reported as an unidentified product. In one embodiment, the product identification method may further include: the image recognition server reporting products with matching degrees less than or equal to a first matching degree threshold as unidentified products (i.e., unknown products); and confirming products with matching results greater than or equal to the MaxCV threshold as identified products. In one embodiment, the product identification method may further include: the image recognition server determining the preliminary identification result of a product with a matching degree greater than or equal to a second matching degree threshold (i.e., confirmed products) as the final product identification result. If multiple products have a matching degree higher than the second matching degree threshold, the product with the highest matching degree value is selected as the final product identification result. For products whose identification results fall between MinCV and MaxCV (i.e., products to be confirmed), the TOP-N results of this product are found (i.e., a candidate product list consisting of the top few product names with the highest matching values). Further confirmation is performed. That is, for products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated to further confirm whether the candidate products are real products in supermarkets. Thus, the product with the preliminary identification result is confirmed as a product in the candidate product list, so as to further improve the accuracy of product identification.

[0101] The product recognition output lists candidate products between MinCV and MaxCV (including multiple product recognition results to be confirmed) as shown in Figure 3.

[0102] In step 103 above, the image recognition server queries the PS (first supermarket terminal) to find the ESL (electronic shelf label) corresponding to each candidate product in the candidate product list. This electronic shelf label can communicate with the supermarket terminal PS via a base station AP. The PS then sends commands to the bound ESL. Commands sent by the PS to the bound ESL may include: instructing the ESL to flash a specified color light, or displaying special characters or graphics on the ESL's screen. The product information list and the product information bound to the ESL in the PS must be consistent, so that the corresponding ESL can be found through the product name information. The first supermarket terminal executes the following steps: if the product information in the candidate product list is not found in the corresponding store in the PS, the candidate product information is removed from the candidate product list. In one embodiment, the above product recognition method may further include: when the first supermarket terminal does not find a product consistent with the candidate product information in the supermarket, the current candidate product is removed from the candidate product list. If all are eliminated, the product is determined to be a new product, and its features need to be collected through a prompt. In one embodiment, the product identification method may further include: when all candidate products are eliminated from the candidate product list at the first supermarket end, the product with the preliminary identification result is determined to be a new product, and a prompt message to collect the new product's features is sent to the image acquisition device. If the PS finds the corresponding product and its associated ESL, it sends a command to instruct the ESL to display it in a preset manner. For example, the PS sends a command to the associated ESL, including instructing the ESL to flash a specified color light or display special characters or graphics on the ESL's screen.

[0103] Based on the product list, a command is sent to the ESL corresponding to the product in the product list to find the ESL corresponding to this shelf and product, as shown in Figure 4.

[0104] At the same time, PS sends commands to the camera via an image recognition server to capture images.

[0105] In step 104 above, if the electronic shelf label receives a command instructing a preset display mode, it will display the information in the preset display mode.

[0106] In step 105 above, when the image acquisition device (camera) receives the re-acquisition command, it re-acquisitions the image of the shelf to be identified and sends the re-acquisitioned image of the shelf to be identified to the image recognition server.

[0107] In step 106 above, the image recaptured by the camera (the re-acquired image of the shelf to be identified) is used to identify whether there is an electronic shelf label displayed in a preset display mode, such as whether the ESL is scheduled to display a light color, characters, or graphics. If no label is identified, the image server removes the product from the candidate product list. That is, in one embodiment, the above product identification method may further include: if the image recognition server does not identify an electronic shelf label displayed in a preset display mode in the re-acquired image of the shelf to be identified, it removes the current candidate product from the candidate product list. The PS sends commands to the corresponding multiple ESLs to obtain the images captured by the camera, identifies the special predetermined characters or images of the ESLs, locates the position of the candidate product in the image, and improves the product identification confidence in the candidate product list through a retrieval strategy, thereby transforming the product identification result to be confirmed into the product identification result already identified. If the ESL in the image fails to improve the product identification confidence in the candidate product list through the retrieval strategy in the PS, the product is marked as a new product, and the features of this new product need to be collected through a reminder prompt.

[0108] In practice, the above retrieval strategy involves determining the distance between the location of the product to be identified and the location of the identified ESL in the image by normalizing the two images. Considering the specific characteristics of the shelf, the product and ESL are positioned on the same shelf level, thus limiting the retrieval to that level. If an identified electronic shelf label (displayed in a preset display mode) appears within the target area of ​​the preset electronic shelf label corresponding to the initially identified product on the same shelf level (this target area can be a pre-defined area on the shelf near the location of the corresponding electronic shelf label, including locations that coincide with the preset electronic shelf label location), then the final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label displayed in the preset display mode. This also increases the confidence of this candidate product, thereby optimizing the product feature database. If multiple ESLs are matched simultaneously within the target area of ​​the preset electronic shelf label corresponding to a product with preliminary identification results, the product bound to the electronic shelf label displayed in the preset display mode that is closest to the product will be taken as the final product identification result. At the same time, the confidence of the corresponding candidate products will be increased according to the distance to optimize the product feature database.

[0109] As described above, in one embodiment, the image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-captured image of the shelf to be recognized. If so, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product recognition result that the product with preliminary recognition results is the product bound to the electronic shelf label. This can include:

[0110] If the image recognition server identifies an electronic shelf label displayed in a preset display mode in the re-captured image of the shelf to be recognized, it obtains the position of the electronic shelf label displayed in the preset display mode in the re-captured image of the shelf to be recognized. Specifically, the image recognition server identifies the preset display electronic price tag, thereby locating the position of the electronic price tag in the shelf image, and automatically obtains the position information of the electronic shelf label.

[0111] The system determines the position of the candidate product with preliminary recognition results in the initially captured shelf image to be recognized; similarly, the image recognition server identifies the product in the initially captured shelf image to be recognized, thereby locating the product's position in the initially captured shelf image to be recognized, and automatically obtains the product's position information;

[0112] The initially acquired image of the shelf to be identified and the re-acquired image of the shelf to be identified are normalized to the same coordinate system to obtain the normalized shelf image.

[0113] Based on the position of the electronic shelf label in the re-acquired shelf image to be identified and the position of the goods in the initially acquired shelf image to be identified, the distance between the goods with preliminary identification results and the electronic shelf label in the normalized shelf image is determined.

[0114] The final product identification result is obtained based on the distance, determining whether the product with preliminary identification results is the product bound to the electronic shelf label.

[0115] In practice, the system identifies whether the recaptured images contain characters or images that the pre-defined ESL should display. For newly acquired images, they need to be normalized compared to the previously identified product images. By combining the position of the image acquisition device with feature point matching of the shelf image, the coordinates of the two images are automatically normalized to the same coordinate system. Based on the distance between the product and the electronic shelf label in the normalized shelf image, the final product recognition result is obtained. For example, if the distance between the product and the electronic shelf label is less than or equal to a preset threshold, the final product recognition result is that the product with the preliminary recognition result is the product bound to the electronic shelf label. For example, if the actual distance between the product on the shelf and the electronic shelf label is 2 centimeters according to the shelf design, then this preset threshold can be 2 centimeters, thereby further improving the accuracy of product recognition.

[0116] Prior to this, as can be seen from the above, in one embodiment, obtaining the final product identification result—whether the product with the preliminary identification result is the product bound to the electronic shelf tag—based on the distance may include:

[0117] Based on the shelf shelf location information, determine whether the product with preliminary identification results and the electronic shelf label displayed in the preset display mode are located on the same shelf shelf;

[0118] If, within the target area of ​​a preset electronic shelf label corresponding to a product with preliminary identification results on the same shelf layer, an electronic shelf label displayed in a preset display mode appears, it is determined that the current candidate product belongs to the product bound to the electronic shelf label displayed in the preset display mode. The final product identification result is obtained: the product with preliminary identification results is the product bound to the electronic shelf label displayed in the preset display mode. For example, if a list of candidate products with matching scores in the top N is generated, and the candidate products are celery and chives, the step of confirming which candidate product the product identification result belongs to needs to be performed one by one: First, determine whether the identification result is celery. Then, when the image acquisition device receives the re-acquisition command, it re-captures the image in front of the shelf corresponding to the product with preliminary identification results. In the supermarket, when the electronic shelf label corresponding to celery receives the command indicating the preset display mode, If a green electronic shelf label appears in the target area corresponding to the preset electronic shelf label on the same shelf layer as the product "celery" in the re-captured shelf image, it is determined that the current candidate product belongs to the product (celery) bound to the corresponding electronic shelf label. The final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label. That is, the product with the preliminary identification result is celery. Conversely, if no green electronic shelf label appears in the re-captured shelf image, it is determined that the current candidate product does not belong to the product (celery) bound to the corresponding electronic shelf label. That is, the product with the preliminary identification result is not celery. Then, the steps to confirm which candidate product the product identification result belongs to are repeated to confirm whether the identification result is chives. This will not be repeated here.

[0119] Optionally, as can be seen from the above, in one embodiment, the above-described product identification method may include:

[0120] If multiple electronic shelf labels with preset display modes appear within the target area of ​​the preset electronic shelf label corresponding to a product with preliminary identification results on the same shelf layer, the electronic shelf label with the preset display mode that is closest to the product with preliminary identification results is selected.

[0121] The product associated with the electronic shelf label displayed in the preset display mode that is closest to the product with the preliminary identification result will be used as the final product identification result.

[0122] In practice, if the distance between the goods on the shelf and the electronic shelf label is exactly equal to the preset threshold (2 cm), it is possible that the goods and the electronic shelf label are not on the same layer, affecting the accuracy of the recognition result. Therefore, considering the special characteristics of the shelf, this embodiment of the invention positions the goods and ESL (electronic shelf label) on the same layer of the shelf, that is, excludes electronic shelf labels that are not on the same layer as the goods for goods recognition, which can further improve the accuracy of goods recognition. For example, if there are two goods in the candidate goods list in the above example, namely celery and leeks, if the electronic shelf labels corresponding to celery and leeks are displayed in a preset way within the preset electronic shelf label target area, for example, they are all displayed in green within the target area, then the distance between the position A1 of the electronic shelf label corresponding to celery and the position A2 of the electronic shelf label corresponding to leeks and the position of the goods is calculated. The goods bound to the electronic shelf label displayed in the preset display mode that is closest to the goods are taken as the final goods recognition result. For example, celery is closest to the goods, so the goods with preliminary recognition results are celery.

[0123] In specific implementation, when calculating the aforementioned distance, the embodiment of the present invention can use the center position of the product. For ease of calculation, the center position of the product frame obtained by target detection of the product can also be used. Similarly, the center position of the electronic shelf label can be used. For ease of calculation, the center position of the electronic shelf label frame obtained by target detection of the electronic shelf label can also be used.

[0124] Optionally, as can be seen from the above, in one embodiment, the above-mentioned product identification method may include: when obtaining the final product identification result that the product with preliminary identification result is the product bound to the electronic shelf label displayed in the preset display mode based on the distance, increasing the confidence of the candidate products in the preset product feature library to optimize the preset product feature library.

[0125] In practice, considering the unique characteristics of the shelves, products and ESLs are positioned on the same shelf level, thus limiting the search constraints to that level. If an ESL matching the candidate product information appears at the preset label location corresponding to the product on the same shelf level (this preset label location is a pre-set label location below the product with preliminary identification results, based on the shelf design), the confidence level of this candidate product is increased. If multiple product ESLs match simultaneously, the confidence level of the corresponding candidate product is increased based on their distance, thereby optimizing the preset product feature library.

[0126] The image recognition server repeats the above steps until it finds the light color, character, or graphic that the pre-defined ESL should display for each product. It then determines that the current product feature belongs to the product bound to this ESL. The result of product recognition, as shown in Figure 5, is obtained by finding the corresponding ESL. Simultaneously, for product recognition results that improve confidence, the image and product information are updated together in the product feature library. Upon receiving the image and product information, the product feature library automatically or manually registers the updated product image and information. It also distributes the product feature library to the image server. Subsequent product recognition will automatically associate the new product features, improving the accuracy of product recognition. In Figures 3 to 5, "confidence" represents the confidence level, "product" represents the candidate product, and "goods-01" to "goods17" represent candidate product identifiers, such as product name and number.

[0127] To facilitate understanding of how this invention is implemented, an example is given below where the image acquisition device is mounted on a first walking robot. Product recognition is performed locally on an image recognition server. For products with a matching degree higher than a threshold (MaxCV, the second matching degree threshold), the image recognition server obtains an accurate judgment result for the identified product. For products where all matching degrees are lower than the threshold (MinCV, the first matching degree threshold), the product is reported as unrecognized. For products with matching degrees falling between MinCV and MaxCV, the top few candidate products with the highest matching values ​​are identified (a list of candidate products with matching degree values ​​in the top N), and the shelf number corresponding to the image is recorded. After the first robot completes its patrol task, the image server commands the robot to return to the previously uncertain product shelf, and then:

[0128] 1) The image recognition server sends commands to the ESLs associated with the top few candidate products via the PS (Product Provider). If the PS does not find a corresponding product in the store, it removes the product from the candidate product list. If the PS finds a corresponding product and its associated ESL, it sends a command. The PS sends commands to the associated ESL, including instructing the ESL to flash a specified color light or display special characters or graphics on the ESL's screen.

[0129] 2) Simultaneously, the image recognition server sends commands to the robot to capture images. In the recaptured image (the image to be recognized), it checks for the light color, characters, or graphics that the pre-defined ESL should display. For newly acquired images, normalization processing is performed between them and the previously recognized product image. Combining the robot's position and image feature point matching, the coordinates of the two images are automatically normalized to the same coordinate system. If no product is recognized, the image server removes the product from the candidate product list.

[0130] 3) The image server repeats the above steps until it finds the corresponding light color, character, or graphic that the pre-defined ESL should display. It then uses the retrieval strategy mentioned above to determine that the current product feature belongs to the product bound to this ESL and updates the product feature database. Alternatively, it may exit the process after all candidate products have been traversed without finding the most matching product.

[0131] Further optional solutions according to the embodiments of the present invention are described below.

[0132] 1. Updating the product feature library can include adding new images to existing products while automatically removing low-quality images. By continuously filtering high-quality product images and information, a high-definition version of the product data feature library is built. Since the product feature library iterates frequently, there are usually daily incremental updates. By flexibly selecting a time window range, the incremental product images and information are merged into the full feature library. This allows for rapid application of the incremental feature library to the image recognition service, and if data issues are found, the problematic data can be quickly modified within the time window before merging.

[0133] 2. In addition to incremental updates, the product feature library also needs to continuously maintain data quality. By performing consistency checks on the product feature library, images that do not meet the quality standards can be quickly filtered out. Consistency checks include checking for problematic images based on factors such as image size and resolution.

[0134] 3. In addition to consistency checks, the product feature database also considers data eviction. By linking ESL product data with PS stores and maintaining image data within a window period, data outside the window is automatically evictioned. The window period is set to N days. If a product in a PS store is unlinked and does not appear within N days, it is considered eviction. This method prevents data from expanding too rapidly while ensuring increasingly higher data quality within the window period, thereby improving the accuracy of product recognition.

[0135] This invention also provides a method for product recognition applied to an image recognition server, as described in the following embodiments. Since the principle underlying this method is similar to that of the product recognition method applied to the first system, the implementation of this method for product recognition applied to an image recognition server can refer to the implementation of the product recognition method applied to the first system; repeated details will not be elaborated further.

[0136] Figure 6 is a flowchart illustrating the product recognition method applied to an image recognition server in an embodiment of the present invention. As shown in Figure 6, the method includes the following steps:

[0137] Step 201: Perform product segmentation and recognition on the shelf image to be identified, obtain preliminary recognition results for each product, and extract the features of each product with preliminary recognition results from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device;

[0138] Step 202: Match each extracted feature with the product features in the preset product feature library;

[0139] Step 203: For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold.

[0140] Step 204: Generate a search command based on the candidate product list to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket terminal; the first supermarket terminal, upon receiving the search command, performs the step of checking whether there is a product matching the candidate product information and its associated electronic shelf label in the first supermarket for each candidate product in the candidate product list. If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; the electronic shelf label is used to display the product in the preset display mode if it receives the command indicating a preset display mode; send a re-capture command to re-capture the image of the shelf to be identified to the image acquisition device; the image acquisition device, upon receiving the re-capture command, re-captures the image of the shelf to be identified and sends the re-captured image of the shelf to be identified to the image recognition server;

[0141] Step 205: Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf images;

[0142] Step 206: If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

[0143] In one embodiment, the above-mentioned product identification method may further include: for products with a matching degree greater than or equal to a second matching degree threshold, determining the preliminary identification result of the product as the final product identification result.

[0144] In one embodiment, the above-described product identification method may further include: for products with a matching degree less than or equal to a first matching degree threshold, reporting the product as an unidentified product.

[0145] In one embodiment, the above-mentioned product identification method may further include: when the first supermarket terminal does not find a product in the supermarket that matches the candidate product information, removing the current candidate product from the candidate product list.

[0146] In one embodiment, the above-mentioned product identification method may further include: when all candidate products are removed from the candidate product list after detection at the first supermarket end, determining that the product with preliminary identification results is a new product, and sending a prompt message to the image acquisition device to collect the features of the new product.

[0147] In one embodiment, the above-mentioned product identification method may further include: if no electronic shelf label displayed in a preset display mode is identified in the re-acquired shelf image to be identified, the current candidate product is removed from the candidate product list.

[0148] In one embodiment, the process involves identifying whether an electronic shelf label, displayed in a preset manner, exists in the re-captured image of the shelf to be identified. If so, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label. The final product identification result, which indicates that the product with the preliminary identification result is the product bound to the electronic shelf label, includes:

[0149] If an electronic shelf label displayed in a preset display mode is detected in the re-acquired shelf image, obtain the position of the electronic shelf label displayed in the preset display mode in the re-acquired shelf image;

[0150] Determine the position of the candidate product with preliminary identification results in the initially acquired shelf image to be identified;

[0151] The initially acquired image of the shelf to be identified and the re-acquired image of the shelf to be identified are normalized to the same coordinate system to obtain the normalized shelf image.

[0152] Based on the position of the electronic shelf label in the re-acquired shelf image to be identified and the position of the goods in the initially acquired shelf image to be identified, the distance between the goods with preliminary identification results and the electronic shelf label in the normalized shelf image is determined.

[0153] The final product identification result is obtained based on the distance, determining whether the product with preliminary identification results is the product bound to the electronic shelf label.

[0154] In one embodiment, obtaining the final product identification result—whether the product with the preliminary identification result is the product bound to the electronic shelf tag—based on the distance includes:

[0155] Based on the shelf shelf location information, determine whether the product with preliminary identification results and the electronic shelf label displayed in the preset display mode are located on the same shelf shelf;

[0156] If so, if an electronic shelf label with a preset display mode appears within the target area of ​​the preset electronic shelf label corresponding to a product with a preliminary identification result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label with the preset display mode, and the final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label with the preset display mode.

[0157] In one embodiment, the above-mentioned product identification method may further include:

[0158] If multiple electronic shelf labels with preset display modes appear within the target area of ​​the preset electronic shelf label corresponding to a product with preliminary identification results on the same shelf layer, the electronic shelf label with the preset display mode that is closest to the product with preliminary identification results is selected.

[0159] The product associated with the electronic shelf label displayed in the preset display mode that is closest to the product with the preliminary identification result will be used as the final product identification result.

[0160] In one embodiment, the above-mentioned product identification method may further include: when obtaining the final product identification result that the product with preliminary identification result is the product bound to the electronic shelf tag based on the distance, increasing the confidence of the candidate products in the preset product feature library to optimize the preset product feature library.

[0161] In one embodiment, the image acquisition device is mounted on a shelf or on a walking first robot.

[0162] This invention also provides a method for product identification applied to a first supermarket, as described in the following embodiments. Since the principle behind this method is similar to that of the product identification method applied to the first system, the implementation of this product identification method applied to the first supermarket can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0163] Figure 7 is a flowchart illustrating the product identification method applied to a first supermarket in an embodiment of the present invention. As shown in Figure 7, the product identification method applied to a first supermarket includes the following steps:

[0164] Step 301: Obtain a search command from the image recognition server, generated based on the candidate product list, to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket; the image recognition server is used to segment and recognize products in the shelf image to be recognized, obtain preliminary recognition results for each product, and extract features of each product with preliminary recognition results from the shelf image to be recognized; match each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with the top N matching degree values, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command from the candidate product list to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket; the shelf image to be recognized is sent by the image acquisition device;

[0165] Step 302: Upon receiving a search command, for each candidate product in the candidate product list, perform the step of checking whether there is a product in the first supermarket that matches the candidate product information and its associated electronic shelf label;

[0166] Step 303: If present, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a re-capture command to the image acquisition device via the image recognition server to re-capture the shelf image to be identified; the electronic shelf label is used to display the shelf in the preset display mode if it receives the command indicating a preset display mode; the image acquisition device is also used to re-capture the shelf image to be identified when it receives the re-capture command, and send the re-captured shelf image to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product identification result that the product with preliminary identification result is the product bound to the electronic shelf label.

[0167] In another product recognition system of the present invention, the robot itself can collect images of the shelf to be recognized and interact with the supermarket end and electronic tags. Candidate products with a matching degree between "MinCV and MaxCV" are searched and confirmed in the supermarket, thereby improving the accuracy of corresponding product recognition. That is, the embodiments of the present invention also provide a product recognition method applied to a second system, as described in the following embodiments. Since the principle of this method in solving the problem is similar to that of the product recognition method applied to the first system, the implementation of this product recognition method applied to the second system can refer to the implementation of the product recognition method applied to the first system. The product recognition method applied to the second system will be described in detail below.

[0168] Figure 8 is a flowchart illustrating the product identification method applied to the second system in an embodiment of the present invention. As shown in Figure 8, the method includes the following steps:

[0169] Step 401: The second robot acquires images of the shelves to be identified; it segments and identifies the products in the images of the shelves to be identified, obtaining preliminary identification results for each product, and extracts features of each product with preliminary identification results from the images of the shelves to be identified; it matches each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, it generates a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; based on the candidate product list, it generates a search command to find products in the first supermarket that match the candidate product information and their bound electronic shelf tags, and sends the search command to the second supermarket.

[0170] Step 402: When the second supermarket receives the search command, for each candidate product in the candidate product list, it performs the step of whether there is a product in the second supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command to the corresponding bound electronic shelf label to indicate the preset display mode; and sends a command to the second robot to re-capture the image of the shelf to be identified.

[0171] Step 403: If the electronic shelf label receives a command instructing a preset display mode, it shall display the label in the preset display mode.

[0172] Step 404: When the second robot receives the re-collection command, it re-collects the image of the shelf to be identified, and identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected image of the shelf to be identified. If there is, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0173] The product recognition method applied to a first system provided in this embodiment of the invention operates as follows: a second robot acquires an image of a shelf to be recognized; the shelf image is segmented and recognized to obtain a preliminary recognition result for each product; features of each product with a preliminary recognition result are extracted from the shelf image; each extracted feature is matched with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list with matching degree values ​​ranking in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; a search command is generated based on the candidate product list to find products in the first supermarket that match the candidate product information and their associated electronic shelf tags, and the search command is sent to the second supermarket; the second supermarket receives the search command. When a search command is received, for each candidate product in the candidate product list, the process involves checking if a product matching the candidate product information and its associated electronic shelf label exists in the second supermarket. If so, a command instructing a preset display mode is sent to the corresponding associated electronic shelf label; a command to re-capture the shelf image to be identified is sent to the second robot; if the electronic shelf label receives the command instructing a preset display mode, it displays the product in the preset display mode; when the second robot receives the re-capture command, it re-captures the shelf image to be identified and checks if there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image. If so, it determines that the current candidate product belongs to the product associated with the corresponding electronic shelf label, thus obtaining the final product identification result: the product with preliminary identification results is the product associated with the electronic shelf label. The following is a detailed description of the product identification method applied to the first system.

[0174] In step 401 above, the second robot (in this embodiment, the robot has the functions of the image acquisition device and image recognition server in the first system) takes pictures and identifies products while walking. Product recognition can be performed locally on the robot or on the image recognition server. For all matching results (matching degree values) that are lower than or equal to the MinCV threshold (first matching degree threshold), the product is reported as unrecognized. That is, in one embodiment, the above product recognition method may further include: for products with a matching degree less than or equal to the first matching degree threshold, reporting the product as unrecognized; for products with a matching degree higher than or equal to the MaxCV threshold (second matching degree threshold), obtaining an accurate judgment result of the identified product, directly judging it as a recognized result. That is, in one embodiment, the above product recognition method may further include: for products with a matching degree greater than or equal to the second matching degree threshold, determining the preliminary recognition result of the product as the final product recognition result. For product features whose matching degree falls between the MinCV and MaxCV thresholds for all products, the top few candidate products with the highest matching values ​​are identified. Specifically, for products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, a list of the top N candidate products with matching degree values ​​is generated. This further confirms whether the candidate products are actual products in the supermarket, thus confirming that the product with preliminary identification results is one of the products in the candidate product list, thereby improving the accuracy of product identification. If product identification is performed locally on the robot, the robot remains in front of the corresponding shelf. The robot (the second robot, which has image acquisition and recognition capabilities, as well as the functions shown in Figure 9) sends commands to the ESL associated with the corresponding TOP-N candidate products via the PS (second supermarket terminal).

[0175] In step 402 above, the execution steps of the second supermarket terminal are as follows: If the PS does not find the corresponding product in this store, the product is removed from the candidate product list. That is, in one embodiment, the above product identification method may further include: when the first supermarket terminal does not find a product that matches the candidate product information in the supermarket, the current candidate product is removed from the candidate product list. If all are removed, the product identification result is determined to be a new product, and the features of this new product need to be collected through a reminder prompt. That is, in one embodiment, the above product identification method may further include: when the first supermarket terminal detects and removes all candidate products from the candidate product list, the product with the preliminary identification result is determined to be a new product, and a prompt message for collecting the features of the new product is sent to the image acquisition device. If the PS finds the corresponding product and the bound ESL, a command instruction is sent. The PS issues a command to the bound ESL, including issuing an instruction to make the ESL flash a specified color light, or display special characters or graphics on the ESL screen.

[0176] In step 403 above, if the electronic shelf label receives a command instructing a preset display mode, it will display the information in the preset display mode.

[0177] In step 404 above, the robot captures images and identifies whether the recaptured images contain the light color, characters, or graphics that the pre-defined ESL should display. For newly acquired images, they need to be normalized with the images of the previously captured (initial or previous cycle) product identification images. Combining the robot's position and image feature point matching, the coordinates of the two images are automatically normalized to the same coordinate system. If no product is identified, the robot removes the product from the candidate product list. That is, in one embodiment, the above product identification method may further include: if the second robot does not identify the electronic shelf label displayed in a preset display mode in the recaptured shelf image to be identified, it removes the current candidate product from the candidate product list.

[0178] The second robot repeats the above steps until it finds the characters or images that the pre-defined ESL should display for each candidate product. It then uses a retrieval strategy to determine that the current product feature belongs to the product bound to this ESL and updates the product feature database. Alternatively, it exits after traversing all candidate products without finding the best match.

[0179] As can be seen from the above, in one embodiment, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result, which indicates that the product with preliminary identification results is the product bound to the electronic shelf label, may include:

[0180] If an electronic shelf label displayed in a preset display mode is detected in the re-acquired shelf image, obtain the position of the electronic shelf label displayed in the preset display mode in the re-acquired shelf image;

[0181] Determine the position of the candidate product with preliminary identification results in the initially acquired shelf image to be identified;

[0182] If an electronic shelf label displayed in a preset display mode is detected in the re-acquired shelf image, obtain the position of the electronic shelf label displayed in the preset display mode in the re-acquired shelf image;

[0183] Determine the position of the candidate product with preliminary identification results in the initially acquired shelf image to be identified;

[0184] The initially acquired image of the shelf to be identified and the re-acquired image of the shelf to be identified are normalized to the same coordinate system to obtain the normalized shelf image.

[0185] Based on the position of the electronic shelf label in the re-acquired shelf image to be identified and the position of the goods in the initially acquired shelf image to be identified, the distance between the goods with preliminary identification results and the electronic shelf label in the normalized shelf image is determined.

[0186] The final product identification result is obtained based on the distance, determining whether the product with preliminary identification results is the product bound to the electronic shelf label.

[0187] As can be seen from the above, in one embodiment, obtaining the final product identification result—whether the product with preliminary identification results is the product bound to the electronic shelf tag—based on the distance includes:

[0188] Based on the shelf shelf location information, determine whether the product with preliminary identification results and the electronic shelf label displayed in the preset display mode are located on the same shelf shelf;

[0189] If so, if an electronic shelf label with a preset display mode appears within the target area of ​​the preset electronic shelf label corresponding to a product with a preliminary identification result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label with the preset display mode, and the final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label with the preset display mode.

[0190] As can be seen from the above, in one embodiment, the product identification also includes:

[0191] If multiple electronic shelf labels with preset display modes appear within the target area of ​​the preset electronic shelf label corresponding to a product with preliminary identification results on the same shelf layer, the electronic shelf label with the preset display mode that is closest to the product with preliminary identification results is selected.

[0192] The product associated with the electronic shelf label displayed in the preset display mode that is closest to the product with the preliminary identification result will be used as the final product identification result.

[0193] As can be seen from the above, in one embodiment, the above-mentioned product identification further includes: when the final product identification result is obtained based on the distance and the product with preliminary identification result is the product bound to the electronic shelf label displayed in the preset display mode, the confidence of the candidate products in the preset product feature library is increased to optimize the preset product feature library.

[0194] In practice, the robot normalizes the two images by matching the robot's position and the feature points of the images, automatically normalizing the coordinates of the two images to the same coordinate system, which further improves the accuracy of product recognition.

[0195] This invention also provides a method for product recognition applied to a second robot, as described in the following embodiments. Since the principle underlying this method is similar to that of the product recognition method applied to the first system, the implementation of this method for product recognition applied to the second robot can refer to the implementation of the method for product recognition applied to the first system; repeated details will not be elaborated further.

[0196] Figure 9 is a flowchart illustrating the product recognition method applied to the second robot in an embodiment of the present invention. As shown in Figure 9, the method includes the following steps:

[0197] Step 501: Acquire an image of the shelf to be identified;

[0198] Step 502: Perform product segmentation and recognition on the shelf image to be identified, obtain preliminary recognition results for each product, and extract the features of each product with preliminary recognition results from the shelf image to be identified;

[0199] Step 503: Match each extracted feature with the product features in the preset product feature library;

[0200] Step 504: For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold.

[0201] Step 505: Generate a search command based on the candidate product list to find products matching the candidate product information and their associated electronic shelf labels in the second supermarket, and send the search command to the second supermarket terminal; the second supermarket terminal, upon receiving the search command, performs the step of checking whether there is a product matching the candidate product information and its associated electronic shelf label in the second supermarket for each candidate product in the candidate product list. If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; and sends a command to re-capture the image of the shelf to be identified to the second robot; the electronic shelf label is used to display the product in the preset display mode if it receives the command indicating the preset display mode.

[0202] Step 506: Upon receiving a re-acquisition command, re-acquire the image of the shelf to be identified;

[0203] Step 507: Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf images;

[0204] Step 508: If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

[0205] This invention also provides a first system for product identification, as described in the following embodiments. Since the principle underlying this first system is similar to the product identification method applied to the first system, the implementation of this first system can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0206] Figure 10 is a schematic diagram of the structure of the first product identification system in an embodiment of the present invention. As shown in Figure 10, the first system includes:

[0207] Image acquisition device 11 is used to acquire images of the shelf to be identified and send them to the image recognition server; when a re-acquisition command is received, the image of the shelf to be identified is re-acquired and the re-acquired image of the shelf to be identified is sent to the image recognition server.

[0208] Image recognition server 12 is used to segment and recognize goods in the shelf image to be recognized, obtain preliminary recognition results for each goods, extract features of each goods with preliminary recognition results from the shelf image to be recognized; match each extracted feature with the features in a preset product feature library; for goods with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command based on the candidate product list to find goods and their bound electronic shelf labels that match the candidate product information in the first supermarket, and send the search command to the first supermarket; identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf image to be recognized, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product recognition result that the goods with preliminary recognition results are the products bound to the electronic shelf labels;

[0209] The first supermarket terminal 13 is used to, when receiving a search command, perform the following steps for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the shelf image to be identified through the image recognition server.

[0210] Electronic shelf label 14 is used to display the information in the preset display mode if a command is received indicating a preset display mode is used.

[0211] This invention also provides an image recognition server for product identification, as described in the following embodiments. Since the principle by which this image recognition server solves the problem is similar to the product identification method applied to the first system, the implementation of this image recognition server can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0212] Figure 11 is a schematic diagram of the structure of the image recognition server for commodity recognition in an embodiment of the present invention. As shown in Figure 11, the server includes:

[0213] This invention also provides an image recognition server for product identification to improve the accuracy of product identification. The image recognition server includes:

[0214] The first feature extraction unit 121 is used to segment and identify goods in the shelf image to be identified, obtain a preliminary identification result for each goods, and extract the features of each goods with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device;

[0215] The first matching processing unit 122 is used to match each extracted feature with the product features in the preset product feature library;

[0216] The first candidate product list determination unit 123 is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold.

[0217] The first generation unit 124 is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket terminal; the first supermarket terminal is used to, upon receiving the search command, perform the step of checking whether there is a product that matches the candidate product information and its associated electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding associated electronic shelf label; the electronic shelf label is used to display the product in the preset display mode if it receives the command indicating a preset display mode; and send a re-capture command to re-capture the image of the shelf to be identified to the image acquisition device; the image acquisition device is used to re-capture the image of the shelf to be identified upon receiving the re-capture command, and send the re-captured image of the shelf to be identified to the image recognition server;

[0218] The first label detection unit 125 is used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-acquired image of the shelf to be identified;

[0219] The first product identification unit 126 is used to obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label.

[0220] This invention also provides a first supermarket terminal for product identification, as described in the following embodiments. Since the principle behind this first supermarket terminal's problem-solving is similar to the product identification method applied to the first system, the implementation of this first supermarket terminal can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0221] Figure 12 is a schematic diagram of the structure of the first supermarket terminal for product identification in an embodiment of the present invention. As shown in Figure 12, the first supermarket terminal includes:

[0222] The first acquisition unit 131 is used to acquire a search command sent by the image recognition server, which generates a search command based on a candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket. The image recognition server is used to segment and recognize products in the shelf image to be recognized, obtain preliminary recognition results for each product, extract features of each product with preliminary recognition results from the shelf image to be recognized, match each extracted feature with product features in a preset product feature library, generate a candidate product list with matching scores in the top N for products with matching scores greater than a first matching score threshold and less than a second matching score threshold, where N is a positive integer greater than 1 and the second matching score threshold is greater than the first matching score threshold, generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket. The shelf image to be recognized is sent by the image acquisition device.

[0223] The first commodity detection unit 132 is used to perform the step of whether there is a commodity in the first supermarket that matches the information of the candidate commodity and its bound electronic shelf label for each candidate commodity in the candidate commodity list when a search command is received;

[0224] The first sending unit 133 is configured to, if present, send a command indicating a preset display mode to the corresponding bound electronic shelf label; and send a re-capture command to the image acquisition device via the image recognition server to re-capture the image of the shelf to be identified; the electronic shelf label is configured to display the shelf in the preset display mode if it receives the command indicating the preset display mode; the image acquisition device is further configured to, upon receiving the re-capture command, re-capture the image of the shelf to be identified and send the re-captured image of the shelf to be identified to the image recognition server; the image recognition server is further configured to, in the re-captured image of the shelf to be identified, identify whether there is an electronic shelf label displayed in the preset display mode, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0225] This invention also provides a second product identification system, as described in the following embodiments. Since the principle of this second system in solving the problem is similar to the product identification method applied to the first system, the implementation of this second system can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0226] Figure 13 is a schematic diagram of the structure of the second product identification system in an embodiment of the present invention. As shown in Figure 13, the second system includes:

[0227] The second robot 21 is used to collect images of shelves to be identified; to segment and identify products in the images of shelves to be identified, and to obtain preliminary identification results for each product; to extract features of each product with preliminary identification results from the images of shelves to be identified; to match each extracted feature with product features in a preset product feature library; to generate a list of candidate products with matching scores greater than a first matching score threshold and less than a second matching score threshold for products with matching scores in the top N, where N is a positive integer greater than 1, and the second matching score threshold is greater than the first matching score threshold; to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the second supermarket, and to send the search command to the second supermarket; when a re-collection command is received, to re-collect images of shelves to be identified, to identify whether there are electronic shelf labels displayed in a preset display mode in the re-collected images of shelves to be identified, and if so, to determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and to obtain the final product identification result that the product with preliminary identification results is the product bound to the electronic shelf label;

[0228] The second supermarket terminal 22 is used to, upon receiving a search command, perform the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its bound electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a command to the second robot to re-capture the image of the shelf to be identified.

[0229] Electronic shelf label 23 is used to display the information in the preset display mode if a command is received indicating a preset display mode is used.

[0230] This invention also provides a second robot for product identification, as described in the following embodiments. Since the principle by which this second robot solves the problem is similar to the product identification method applied to the first system, the implementation of this second robot can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0231] Figure 14 is a schematic diagram of the structure of the second robot for product recognition in an embodiment of the present invention. As shown in Figure 14, the second robot includes:

[0232] The acquisition unit 211 is used to acquire images of the shelf to be identified; when a re-acquisition command is received, the image of the shelf to be identified is re-acquired.

[0233] The second feature extraction unit 212 is used to segment and identify the goods in the initially acquired shelf image to be identified, obtain the preliminary identification result of each goods, and extract the features of each goods with the preliminary identification result from the shelf image to be identified.

[0234] The second matching processing unit 213 is used to match each extracted feature with the product features in the preset product feature library;

[0235] The second candidate product list determination unit 214 is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold.

[0236] The second generation unit 215 is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the second supermarket, and send the search command to the second supermarket terminal; the second supermarket terminal is used to, upon receiving the search command, perform the step of checking whether there is a product that matches the candidate product information and its associated electronic shelf label in the second supermarket for each candidate product in the candidate product list; if so, send a command indicating a preset display mode to the corresponding associated electronic shelf label; and send a command to re-capture the image of the shelf to be identified to the second robot; the electronic shelf label is used to display the product in the preset display mode if it receives the command indicating the preset display mode.

[0237] The second label detection unit 216 is used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-acquired shelf image to be identified;

[0238] The second product identification unit 217 is used to obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label.

[0239] This invention also provides a method for product identification applied to a second supermarket, as described in the following embodiments. Since the principle behind this method is similar to that of the product identification method applied to the first system, its implementation can be found in the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0240] The product identification method applied to the second-tier supermarket includes the following steps:

[0241] The system receives a command from a second robot to search for products matching candidate product information and their associated electronic shelf labels in a first supermarket. The second robot collects images of the shelves to be identified. It segments and identifies products in the initially collected shelf images to obtain preliminary identification results for each product. Features of each product with preliminary identification results are extracted from the shelf images. The extracted features of each product with preliminary identification results are matched with product features in a preset product feature library. For products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a list of candidate products with the top N feature matching degree values ​​(N is a positive integer greater than 1) matching the product with the preliminary identification results is identified. The second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, a command is generated to search for products matching candidate product information and their associated electronic shelf labels in the first supermarket, and this command is sent to the second supermarket.

[0242] Upon receiving the command, for each candidate product in the candidate product list, the step of checking whether there is a product in the first supermarket that matches the candidate product information and its associated electronic shelf label is executed;

[0243] If present, a command instructing a preset display mode is sent to the corresponding bound electronic shelf label; a command to re-capture the shelf image to be identified is sent to the second robot; the electronic shelf label is used to display the shelf in the preset display mode if it receives the command instructing a preset display mode; the second robot is also used to acquire the shelf image to be identified when it receives the command to re-capture the shelf image to be identified; to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, to determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and to obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0244] This invention also provides a second supermarket terminal for product identification, as described in the following embodiments. Since the principle behind this second supermarket terminal's problem-solving is similar to the product identification method applied to the first system, the implementation of this second supermarket terminal can refer to the implementation of the product identification method applied to the first system; repeated details will not be elaborated further.

[0245] The second supermarket terminal for product identification includes:

[0246] The second acquisition unit is used to acquire a command from the second robot to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket. The second robot is used to acquire images of the shelves to be identified. The acquired images of the shelves to be identified are segmented and identified to obtain preliminary identification results for each product. Features of each product with preliminary identification results are extracted from the images of the shelves to be identified. The features of each product with preliminary identification results are matched with product features in a preset product feature library. For products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a list of candidate products with the top N feature matching degree values ​​of products with preliminary identification results is found, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, a command is generated to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and the command is sent to the second supermarket.

[0247] The second product detection unit is used to, upon receiving the command, perform the following step for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label.

[0248] The second sending unit is configured to, if present, send a command indicating a preset display mode to the corresponding bound electronic shelf label; and send a command to re-capture the image of the shelf to be identified to the second robot; the electronic shelf label is configured to display the shelf in the preset display mode if it receives the command indicating the preset display mode; the second robot is further configured to, upon receiving the command to re-capture the image of the shelf to be identified, acquire the image of the shelf to be identified; identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured image of the shelf to be identified; if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0249] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for identifying goods.

[0250] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described product identification method.

[0251] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for identifying goods.

[0252] The product identification solution provided by this invention achieves the following:

[0253] 1) In the first system of the present invention, the image acquisition device or robot improves the accuracy of identifying uncertain product identification results by interacting with the ESL system (which may include: a first supermarket terminal, an AP, and multiple electronic shelf labels).

[0254] 2) A dual-threshold method is used to filter out the list of products to be identified. By using dual thresholds to discover unknown products, products to be identified, and products to be confirmed, the accuracy of product identification is improved.

[0255] 3) In the second system of the present invention, the robot performs normalization processing on the two images (the initial acquisition and the re-acquisition), and automatically normalizes the coordinates of the two images to the same coordinate system by matching the robot's position and the feature points of the images.

[0256] 4) Map product information to ESL, filter product information with low confidence, and aggregate the corresponding price tags in the ESL system.

[0257] 5) By combining the camera system with methods such as flashing / page cutting to capture and identify images, the specified ESL location and ESL number can be found.

[0258] 6) Association Method: By analyzing the distance relationship between the product display and a specified ESL location, mismatches in the candidate list can be eliminated to determine the final product identification result in the candidate list. If the electronic price tag frame falls within a preset area below the product frame to be detected, the association is successful.

[0259] 7) By combining the product information and window period strategy bound to ESL in PS, the data in the product feature library is automatically eliminated, so that the data does not expand too quickly, while ensuring that the data quality within the window period becomes higher and higher, thereby improving the accuracy of product recognition.

[0260] In summary, the product recognition scheme provided by the embodiments of the present invention can improve the accuracy of product recognition.

[0261] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0262] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0263] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0264] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0265] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for product identification, characterized in that, The product identification method is applied to the first system, and the method includes: The image acquisition device captures images of the shelf to be identified and sends them to the image recognition server; The image recognition server segments and recognizes goods in the shelf image to be recognized, obtaining a preliminary recognition result for each goods. Features of each goods with preliminary recognition results are extracted from the shelf image. Each extracted feature is matched with features in a preset product feature library. For goods with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a candidate product list with the top N matching degree values ​​is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, a search command is generated to find goods matching the candidate product information and their associated electronic shelf tags in the first supermarket, and the search command is sent to the first supermarket. When the first supermarket receives a search command, it performs the following steps for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command indicating the preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the image of the shelf to be identified through the image recognition server. If the electronic shelf label receives a command instructing a preset display mode, it will display the label in that preset display mode. When the image acquisition device receives a re-acquisition command, it re-acquisitions the image of the shelf to be identified and sends the re-acquisitioned image of the shelf to be identified to the image recognition server. The image recognition server identifies whether there are electronic shelf labels displayed in a preset display mode in the re-captured shelf images. If so, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product recognition result that the product with preliminary recognition results is the product bound to the electronic shelf label.

2. The method as described in claim 1, characterized in that, The image acquisition device is mounted on a shelf or on the first walking robot.

3. A method for product identification, characterized in that, The product recognition method is applied to an image recognition server, and the method includes: The product segmentation and recognition are performed on the shelf image to be identified to obtain a preliminary identification result for each product. The features of each product with a preliminary identification result are extracted from the shelf image to be identified. The shelf image to be identified is sent by an image acquisition device. Each extracted feature is matched with product features in a pre-defined product feature library; For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. A search command is generated based on the candidate product list to find products matching the candidate product information and their associated electronic shelf labels in the first supermarket. The search command is sent to the first supermarket. Upon receiving the search command, the first supermarket performs the following steps for each candidate product in the candidate product list: to determine if a product matching the candidate product information and its associated electronic shelf label exists in the first supermarket. If so, a command instructing a preset display mode is sent to the corresponding associated electronic shelf label. The electronic shelf label displays the product in the preset display mode if it receives the command. A re-capture command is sent to the image acquisition device to re-capture the shelf image to be identified. Upon receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified and sends the re-captured shelf image to the image recognition server. Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired images of the shelves to be identified; If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

4. The method as described in claim 3, characterized in that, Also includes: For products with a matching degree greater than or equal to the second matching degree threshold, the preliminary identification result of the product is determined as the final product identification result.

5. The method as described in claim 3, characterized in that, Also includes: For products with a matching degree less than or equal to the first matching degree threshold, report the product as an unidentified product.

6. The method as described in claim 3, characterized in that, Also includes: When the first supermarket does not find a product that matches the candidate product information in the supermarket, the current candidate product will be removed from the candidate product list.

7. The method as described in claim 6, characterized in that, Also includes: When all candidate products are removed from the candidate product list after detection at the first supermarket end, the product with preliminary identification results is determined to be a new product, and a prompt message to collect the features of the new product is sent to the image acquisition device.

8. The method as described in claim 3, characterized in that, Also includes: If the electronic shelf label displayed in the preset display mode is not detected in the re-captured shelf image, the current candidate product is removed from the candidate product list.

9. The method as described in claim 3, characterized in that, If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result, which has a preliminary identification result, is the product bound to the electronic shelf label, including: If an electronic shelf label displayed in a preset display mode is detected in the re-acquired shelf image, obtain the position of the electronic shelf label displayed in the preset display mode in the re-acquired shelf image; Determine the position of the candidate product with preliminary identification results in the initially acquired shelf image to be identified; The initially acquired image of the shelf to be identified and the re-acquired image of the shelf to be identified are normalized to the same coordinate system to obtain the normalized shelf image. Based on the position of the electronic shelf label in the re-acquired shelf image to be identified and the position of the goods in the initially acquired shelf image to be identified, the distance between the goods with preliminary identification results and the electronic shelf label in the normalized shelf image is determined. The final product identification result is obtained based on the distance, determining whether the product with preliminary identification results is the product bound to the electronic shelf label.

10. The method as described in claim 9, characterized in that, The final product identification result, based on the distance, determines whether the product with preliminary identification results is the product bound to the electronic shelf tag. This includes: Based on the shelf shelf location information, determine whether the product with preliminary identification results and the electronic shelf label displayed in the preset display mode are located on the same shelf shelf; If so, if an electronic shelf label with a preset display mode appears within the target area of ​​the preset electronic shelf label corresponding to a product with a preliminary identification result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label with the preset display mode, and the final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label with the preset display mode.

11. The method as described in claim 10, characterized in that, Also includes: If multiple electronic shelf labels with preset display modes appear within the target area of ​​the preset electronic shelf label corresponding to a product with preliminary identification results on the same shelf layer, the electronic shelf label with the preset display mode that is closest to the product with preliminary identification results is selected. The product associated with the electronic shelf label displayed in the preset display mode that is closest to the product with the preliminary identification result will be used as the final product identification result.

12. The method of claim 9, further comprising: When the final product identification result is obtained based on the distance and the product with preliminary identification result is the product bound to the electronic shelf label displayed in the preset display mode, the confidence of the candidate products in the preset product feature library is increased to optimize the preset product feature library.

13. A method for product identification, characterized in that, The product identification method is applied to the second system, and the method includes: The second robot collects images of the shelves to be identified; it segments and identifies the products in the images to be identified, obtaining preliminary identification results for each product, and extracts features of each product with preliminary identification results from the images of the shelves to be identified; it matches each extracted feature with product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, it generates a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; based on the candidate product list, it generates a search command to find products in the first supermarket that match the candidate product information and their associated electronic shelf tags, and sends the search command to the second supermarket. When the second supermarket receives a search command, it performs the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command to the corresponding bound electronic shelf label to indicate the preset display mode; and sends a command to the second robot to re-capture the image of the shelf to be identified. If the electronic shelf label receives a command instructing a preset display mode, it will display the label in that preset display mode. When the second robot receives the re-collection command, it re-collects the image of the shelf to be identified. In the re-collected image of the shelf to be identified, it identifies whether there is an electronic shelf label displayed in a preset display mode. If there is, it determines that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtains the final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

14. A method for product identification, characterized in that, The product recognition method is applied to a second robot, and the method includes: Acquire images of the shelf to be identified; The product segmentation and recognition are performed on the shelf image to be identified to obtain a preliminary recognition result for each product. The features of each product with a preliminary recognition result are extracted from the shelf image to be identified. Each extracted feature is matched with product features in a pre-defined product feature library; For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, a search command is generated to locate products matching the candidate product information and their associated electronic shelf labels in the second supermarket. This search command is then sent to the second supermarket. Upon receiving the search command, the second supermarket, for each candidate product in the candidate product list, performs the following steps: Does the second supermarket have a product matching the candidate product information and its associated electronic shelf label? If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; it also sends a command to re-capture the image of the shelf to be identified to the second robot. The electronic shelf label, upon receiving the command indicating the preset display mode, displays the product in the preset display mode. Upon receiving a re-acquisition command, the image of the shelf to be identified is re-acquired; Identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired images of the shelves to be identified; If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is obtained that the product with preliminary identification results is the product bound to the electronic shelf label.

15. The method as described in claim 14, characterized in that, If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result, which has a preliminary identification result, is the product bound to the electronic shelf label, including: If an electronic shelf label displayed in a preset display mode is detected in the re-acquired shelf image, obtain the position of the electronic shelf label displayed in the preset display mode in the re-acquired shelf image; Determine the position of the candidate product with preliminary identification results in the initially acquired shelf image to be identified; The initially acquired image of the shelf to be identified and the re-acquired image of the shelf to be identified are normalized to the same coordinate system to obtain the normalized shelf image. Based on the position of the electronic shelf label in the re-acquired shelf image to be identified and the position of the goods in the initially acquired shelf image to be identified, the distance between the goods with preliminary identification results and the electronic shelf label in the normalized shelf image is determined. The final product identification result is obtained based on the distance, determining whether the product with preliminary identification results is the product bound to the electronic shelf label.

16. A first system for product identification, characterized in that, include: An image acquisition device is used to acquire images of the shelf to be identified and send them to an image recognition server. Upon receiving a re-acquisition command, the image of the shelf to be identified is re-acquisitioned, and the re-acquisitioned image of the shelf to be identified is sent to the image recognition server. An image recognition server is used to segment and recognize goods in the image of a shelf to be recognized, obtain a preliminary recognition result for each goods, extract features of each goods with a preliminary recognition result from the image of the shelf to be recognized, and match each extracted feature with the features in a preset product feature library. For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, generate a search command to find products that match the candidate product information and their associated electronic shelf tags in the first supermarket, and send the search command to the first supermarket. In the re-acquired images of the shelves to be identified, identify whether there are electronic shelf labels displayed in a preset display mode. If there are, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain the final product identification result that the product with preliminary identification results is the product bound to the electronic shelf label. The first supermarket terminal, upon receiving a search command, performs the following steps for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a re-capture command to the image acquisition device to re-capture the shelf image to be identified through the image recognition server. Electronic shelf labels are used to display items in a preset display mode if a command is received instructing the display mode to be preset.

17. An image recognition server for commodity recognition, characterized in that, include: The first feature extraction unit is used to segment and identify goods in the shelf image to be identified, obtain a preliminary identification result for each goods, and extract the features of each goods with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device; The first matching processing unit is used to match each extracted feature with the product features in the preset product feature library; The first candidate product list determination unit is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold. The first generation unit is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the first supermarket, and send the search command to the first supermarket. When the first supermarket terminal receives a search command, it performs the following step for each candidate product in the candidate product list: whether there is a product in the first supermarket that matches the candidate product information and its bound electronic shelf label. If there is, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label. The electronic shelf label is used to display the product in the preset display mode if it receives the command indicating a preset display mode. Send a re-acquisition command to the image acquisition device to re-capture the image of the shelf to be identified; The image acquisition device is used to re-acquire the image of the shelf to be identified when a re-acquisition command is received, and send the re-acquired image of the shelf to be identified to the image recognition server. The first label detection unit is used to identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf image to be identified; The first product identification unit is used to determine, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label.

18. A second system for product identification, characterized in that, include: The second robot is used to collect images of the shelves to be identified; The product segmentation and recognition are performed on the shelf image to be identified to obtain a preliminary recognition result for each product. Features of each product with a preliminary recognition result are extracted from the shelf image to be identified. Each extracted feature is matched with product features in a preset product feature library. For products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold. Based on the candidate product list, generate a search command to find products that match the candidate product information and their associated electronic shelf labels in the second supermarket, and send the search command to the second supermarket. When a re-acquisition command is received, the image of the shelf to be identified is re-acquisitioned. In the re-acquisitioned image of the shelf to be identified, it is identified whether there is an electronic shelf label displayed in a preset display mode. If there is, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label. The final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label. The second supermarket terminal, upon receiving a search command, performs the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its associated electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding associated electronic shelf label; and sends a command to the second robot to re-capture the image of the shelf to be identified. Electronic shelf labels are used to display items in a preset display mode if a command is received instructing the display mode to be preset.

19. A second robot for product recognition, characterized in that, include: The acquisition unit is used to acquire images of the shelves to be identified. Upon receiving a re-acquisition command, the image of the shelf to be identified is re-acquired; The second feature extraction unit is used to segment and identify the goods in the initially acquired shelf image to be identified, obtain the preliminary identification result of each goods, and extract the features of each goods with the preliminary identification result from the shelf image to be identified. The second matching processing unit is used to match each extracted feature with the product features in the preset product feature library; The second candidate product list determination unit is used to generate a candidate product list with matching degree values ​​ranking in the top N for products with matching degree values ​​greater than the first matching degree threshold and less than the second matching degree threshold, where N is a positive integer greater than 1 and the second matching degree threshold is greater than the first matching degree threshold. The second generation unit is used to generate a search command based on the candidate product list to find products that match the candidate product information and their associated electronic shelf labels in the second supermarket, and send the search command to the second supermarket. When the second supermarket terminal receives a search command, it performs the following steps for each candidate product in the candidate product list: whether there is a product in the second supermarket that matches the candidate product information and its bound electronic shelf label. If so, it sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a command to the second robot to re-capture the image of the shelf to be identified. The electronic shelf label is used to display the product in the preset display mode if it receives the command indicating a preset display mode. The second label detection unit is used to identify whether there are electronic shelf labels displayed in a preset display mode in the re-acquired shelf image to be identified; The second product identification unit is used to determine, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product identification result is that the product with the preliminary identification result is the product bound to the electronic shelf label.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 15.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 15.

22. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 15.

Images