Commodity identification method, system, server and robot

Through the collaborative work of image recognition technology and electronic shelf labels, the dual threshold value and re-acquisition confirmation mechanism are used to solve the problem of low product recognition accuracy, and efficient and accurate identification and out-of-stock monitoring are achieved in the electronic shelf environment.

CN118710997BActive Publication Date: 2024-11-15HANSHOW TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411194687.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-15
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

In the prior art, the product recognition accuracy is low, especially in electronic shelf environments, and it is difficult to accurately identify the goods on the shelf and monitor the out-of-stock status.

Method used

The shelf images are collected through the image acquisition device. The image recognition server segments and recognizes the image, extracts features and matches them with the preset library, uses double threshold values ​​to filter the candidate product list, and re-acquisition and confirmation in combination with the display method of electronic shelf labels. Finalize the product identification results.

Benefits of technology

It improves the accuracy of product identification, ensures accurate identification and out-of-stock monitoring of shelf products in electronic shelf environments, and improves the system's identification efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710997B_ABST
    Figure CN118710997B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system, server and robot for commodity identification, the method comprising: an image acquisition device acquires a shelf image to be identified; a server performs commodity segmentation and identification on the shelf image to be identified, and matches each commodity feature extracted from the image with the feature in the feature library; for commodities with a matching degree greater than a first and less than a second threshold, a list of top N candidate commodities is generated; a supermarket end executes the steps of determining whether there is a commodity consistent with each candidate commodity in the list and a bound electronic tag in the supermarket, and if so, sends a display command to the bound tag, and sends a re-collection command to the acquisition device; if the tag receives the command, it is displayed in a preset display mode; the acquisition device re-collects the shelf image to be identified; the server identifies whether there is a label displayed in the display mode in the re-collected image, and if so, determines that the commodity with the preliminary recognition result is the final recognition result of the commodity bound to the label. The present invention can improve the accuracy of commodity identification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, server and robot for commodity identification. Background Art

[0002] This section is intended to provide a background or context for embodiments of the present invention. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Currently, electronic shelf labels will replace traditional paper labels in large supermarkets around the world. In addition to displaying general information, electronic shelf labels can also be used for many applications, such as fast picking, out-of-stock management, fast inventory, and human-computer interaction between users. For cameras taking pictures in the shelf area, robots walking and taking pictures in the shelf area, identifying products to build shelf product display maps, and monitoring out-of-stock status on shelves, it is very meaningful to improve the accuracy of product recognition in supermarkets equipped with electronic price tags. Currently, the accuracy of product recognition is low. Summary of the invention

[0004] An embodiment of the present invention provides a method for commodity identification, which is applied to a first system to improve the accuracy of commodity identification. The method includes:

[0005] The image acquisition device acquires the image of the shelf to be identified and sends it to the image recognition server;

[0006] The image recognition server performs commodity segmentation and recognition on the shelf image to be recognized, obtains a preliminary recognition result of each commodity, extracts features of each commodity with the preliminary recognition result from the shelf image to be recognized; matches each extracted feature with a commodity feature in a preset commodity feature library; generates a candidate commodity list with a matching value in the top N for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with consistent candidate commodity information and their bound electronic shelf labels in a first supermarket according to the candidate commodity list, and sends the search command to the first supermarket end;

[0007] When the first supermarket terminal receives the search command, for each candidate product in the candidate product list, the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label is performed. If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; a re-capture command for re-shooting the shelf image to be identified is sent to the image acquisition device through the image recognition server;

[0008] If the electronic shelf label receives a command indicating a preset display mode, it displays in the preset display mode;

[0009] When receiving the re-capture command, the image acquisition device re-captures the image of the shelf to be identified, and sends the re-captured image of the shelf to be identified to the image recognition server;

[0010] The image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified. If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0011] The embodiment of the present invention further provides a method for commodity identification, which is applied to an image recognition server to improve the accuracy of commodity identification. The method includes:

[0012] Segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by an image acquisition device;

[0013] Match each extracted feature with the product features in the preset product feature library;

[0014] For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0015] Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list, and send the search command to the first supermarket end; the first supermarket end is used to, upon receiving the search command, execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used to display in the preset display mode if receiving the command indicating the preset display mode; send a re-capture command for re-shooting the image of the shelf to be identified to the image acquisition device; the image acquisition device is used to re-capture the image of the shelf to be identified when receiving the re-capture command, and send the re-captured image of the shelf to be identified to the image recognition server;

[0016] Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0017] If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0018] The embodiment of the present invention further provides a method for commodity identification, which is applied to a first supermarket terminal to improve the accuracy of commodity identification. The method includes:

[0019] A search command is obtained from an image recognition server for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in a first supermarket based on a list of candidate products; the image recognition server is used to segment and identify products on the shelf image to be identified, obtain a preliminary identification result for each product, extract features of each product with a preliminary identification result from the shelf image to be identified; match each extracted feature with a product feature in a preset product feature library; for products whose matching degree is greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with matching values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; a search command is generated from the candidate product list for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in a first supermarket, and the search command is sent to the first supermarket end; the shelf image to be identified is sent by an image acquisition device;

[0020] When receiving the search command, for each candidate product in the candidate product list, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label;

[0021] If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; a re-capture command for reshooting the shelf image to be identified is sent to the image acquisition device through the image recognition server; the electronic shelf label is used to display in the preset display mode if the command indicating the preset display mode is received; the image acquisition device is also used to re-capture the shelf image to be identified upon receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0022] The embodiment of the present invention further provides a method for commodity identification, which is applied to the second system to improve the accuracy of commodity identification. The method includes:

[0023] The second robot collects the shelf image to be identified; performs commodity segmentation and identification on the shelf image to be identified to obtain a preliminary identification result of each commodity, and extracts the features of each commodity with the preliminary identification result from the shelf image to be identified; matches each extracted feature with the commodity feature in the preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generates a candidate commodity list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with consistent candidate commodity information and their bound electronic shelf labels in the first supermarket according to the candidate commodity list, and sends the search command to the second supermarket end;

[0024] When the second supermarket terminal receives the search command, for each candidate product in the candidate product list, the second supermarket terminal performs the steps of determining whether there is a product with the same information as the candidate product and its bound electronic shelf label in the second supermarket, and if so, sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a command to re-shoot the image of the shelf to be identified to the second robot;

[0025] If the electronic shelf label receives a command indicating a preset display mode, it displays in the preset display mode;

[0026] When the second robot receives the re-collection command, it re-collects the shelf image to be identified, and identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified. If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained, indicating that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0027] The embodiment of the present invention further provides a method for commodity identification, which is applied to a second robot to improve the accuracy of commodity identification. The method includes:

[0028] Collect images of shelves to be identified;

[0029] Segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, and extract features of each commodity with a preliminary identification result from the shelf image to be identified;

[0030] Match each extracted feature with the product features in the preset product feature library;

[0031] For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0032] Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the second supermarket according to the candidate product list, and send the search command to the second supermarket end; the second supermarket end is used to perform the steps of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the second supermarket for each candidate product in the candidate product list upon receiving the search command, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating a preset display mode;

[0033] When receiving a re-capture command, re-capture the image of the shelf to be identified;

[0034] Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0035] If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0036] The embodiment of the present invention further provides a first system for commodity identification, which is used to improve the accuracy of commodity identification. The first system includes:

[0037] The image acquisition device is used to acquire the image of the shelf to be identified and send it to the image recognition server; when receiving a re-acquisition command, the image of the shelf to be identified is re-acquired, and the re-acquired image of the shelf to be identified is sent to the image recognition server;

[0038] The image recognition server is used to segment and identify commodities in the shelf image to be identified, obtain a preliminary identification result of each commodity, extract the features of each commodity with the preliminary identification result from the shelf image to be identified; match each extracted feature with the commodity features in a preset commodity feature library; generate a candidate commodity list with the top N matching values ​​for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for commodities with the same information as the candidate commodities and their bound electronic shelf labels in a first supermarket according to the candidate commodity list, and send the search command to the first supermarket end; identify whether there is an electronic shelf label displayed in a preset display mode in the shelf image to be identified, and if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label, obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label;

[0039] The first supermarket terminal is used for, upon receiving the search command, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a re-capture command for re-shooting the shelf image to be identified to the image acquisition device through the image recognition server;

[0040] The electronic shelf label is used to display in the preset display mode if a command indicating the preset display mode is received.

[0041] The embodiment of the present invention further provides an image recognition server for commodity identification, which is used to improve the accuracy of commodity identification. The image recognition server includes:

[0042] The first feature extraction unit is used to segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with the preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device;

[0043] A first matching processing unit, used to match each extracted feature with a commodity feature in a preset commodity feature library;

[0044] A first candidate product list determination unit, for generating a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0045] The first generating unit is used to generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket according to the candidate product list, and send the search command to the first supermarket end; the first supermarket end is used to, upon receiving the search command, execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used to display in the preset display mode upon receiving the command indicating the preset display mode; send a re-capture command for re-shooting the shelf image to be identified to the image acquisition device; the image acquisition device is used to re-capture the shelf image to be identified upon receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server;

[0046] A first label detection unit, used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0047] The first commodity identification unit is used to obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label.

[0048] The embodiment of the present invention further provides a first supermarket terminal for commodity identification, which is used to improve the accuracy of commodity identification. The first supermarket terminal includes:

[0049] The first acquisition unit is used to acquire a search command sent by the image recognition server for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list; the image recognition server is used to segment and identify the products on the shelf image to be identified, obtain a preliminary identification result for each product, extract the features of each product with the preliminary identification result from the shelf image to be identified; match each extracted feature with the product features in a preset product feature library; for products whose matching degree is greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with matching values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list, and send the search command to the first supermarket end; the shelf image to be identified is sent by the image acquisition device;

[0050] The first commodity detection unit is configured to, upon receiving a search command, perform a step of determining, for each candidate commodity in the candidate commodity list, whether there is a commodity consistent with the candidate commodity information and a bound electronic shelf label thereof in the first supermarket;

[0051] The first sending unit is used to send a command indicating a preset display mode to a corresponding bound electronic shelf label if it exists; send a re-capture command for reshooting the shelf image to be identified to the image acquisition device through the image recognition server; the electronic shelf label is used to display in the preset display mode if it receives the command indicating the preset display mode; the image acquisition device is also used to re-capture the shelf image to be identified when receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain a final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0052] The embodiment of the present invention further provides a second system for commodity identification, which is used to improve the accuracy of commodity identification. The second system includes:

[0053] The second robot is used to collect the shelf image to be identified; perform commodity segmentation and identification on the shelf image to be identified to obtain a preliminary identification result of each commodity, extract the features of each commodity with the preliminary identification result from the shelf image to be identified; match each extracted feature with the commodity features in the preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate commodity list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for commodities with the same information as the candidate commodities and their bound electronic shelf labels in the second supermarket according to the candidate commodity list, and send the search command to the second supermarket end; upon receiving the re-collection command, re-collect the shelf image to be identified, identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified, and if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label, obtain a final commodity recognition result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label;

[0054] The second supermarket terminal is used for, upon receiving the search command, executing the step of determining whether there is a product in the second supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a command to retake the image of the shelf to be identified to the second robot;

[0055] The electronic shelf label is used to display in the preset display mode if a command indicating the preset display mode is received.

[0056] The embodiment of the present invention further provides a second robot for commodity identification, which is used to improve the accuracy of commodity identification. The second robot includes:

[0057] The acquisition unit is used to acquire the image of the shelf to be identified; when receiving a re-acquisition command, the image of the shelf to be identified is re-acquired;

[0058] The second feature extraction unit is used to segment and identify commodities on the initially collected shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified;

[0059] A second matching processing unit, used to match each extracted feature with a commodity feature in a preset commodity feature library;

[0060] A second candidate product list determination unit, for generating a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0061] The second generating unit is used to generate a search command for searching for a commodity consistent with the candidate commodity information and its bound electronic shelf label in the second supermarket according to the candidate commodity list, and send the search command to the second supermarket end; the second supermarket end is used to execute the step of determining whether there is a commodity consistent with the candidate commodity information and its bound electronic shelf label in the second supermarket for each candidate commodity in the candidate commodity list upon receiving the search command, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating a preset display mode;

[0062] A second label detection unit is used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0063] The second commodity identification unit is used to obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label.

[0064] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned commodity identification method when executing the computer program.

[0065] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned commodity identification method is implemented.

[0066] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for identifying a product is implemented.

[0067] In the embodiment of the present invention, the product identification solution, compared with the technical solution of low product identification accuracy in the prior art, is as follows: an image acquisition device acquires a shelf image to be identified and sends it to an image recognition server; the image recognition server performs product segmentation and identification on the shelf image to be identified, obtains a preliminary identification result of each product, extracts the feature of each product with the preliminary identification result from the shelf image to be identified; matches each extracted feature with a product feature in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generates a candidate product list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for products with consistent candidate product information and electronic shelf labels bound thereto in a first supermarket according to the candidate product list, and sends the search command to the first supermarket end; when the first supermarket end receives the search command, for the candidate product list, For each candidate product in the table, the steps of determining whether there is a product consistent with the candidate product information and its bound electronic shelf label in the first supermarket, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; sending a re-capture command for reshooting the shelf image to be identified to the image acquisition device through the image recognition server; if the electronic shelf label receives the command indicating the preset display mode, it is displayed in the preset display mode; when receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified, and sends the re-captured shelf image to be identified to the image recognition server; the image recognition server identifies whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label, so as to improve the accuracy of product recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0069] Figure 1 It is a flowchart of a method for commodity identification applied to the first system in an embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of a double threshold value in an embodiment of the present invention;

[0071] Figure 3This is a schematic diagram of a commodity list between MinCV and MaxCV outputted by commodity recognition in an embodiment of the present invention;

[0072] Figure 4 In the embodiment of the present invention, a command is sent according to the ESL corresponding to the commodity list to find the ESL schematic diagram corresponding to the shelf;

[0073] Figure 5 This is a schematic diagram of a result of determining commodity identification based on finding the corresponding ESL in an embodiment of the present invention;

[0074] Figure 6 It is a flowchart of a method for commodity identification applied to an image recognition server in an embodiment of the present invention;

[0075] Figure 7 Schematic diagram of a flow chart of a method for commodity identification applied to a first supermarket terminal in an embodiment of the present invention;

[0076] Figure 8 It is a flowchart of a method for commodity identification applied to the second system in an embodiment of the present invention;

[0077] Fig. 9 is a flow chart of a method for commodity identification applied to a second robot in an embodiment of the present invention;

[0078] Fig.10 This is a schematic diagram of the structure of the first system for commodity identification in an embodiment of the present invention;

[0079] Fig.11 A schematic diagram of the structure of an image recognition server for commodity recognition in an embodiment of the present invention;

[0080] Fig.12 This is a structural diagram of a first supermarket terminal for commodity identification in an embodiment of the present invention;

[0081] Fig.13 This is a schematic diagram of the structure of the second system for commodity identification in an embodiment of the present invention;

[0082] Fig.14 Schematic diagram of the structure of the second robot for commodity recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0084] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of laws and regulations.

[0085] The accuracy of direct commodity identification in the existing technology is not high, and the identification accuracy is only about 70%.

[0086] In the first system of the present invention, the image acquisition device acquires the shelf image to be identified, and the image recognition server interacts with the supermarket end and the electronic tag, and the candidate products with a matching degree between "MinCV and MaxCV" are searched and confirmed in the supermarket to improve the recognition accuracy of the corresponding products. The product recognition method applied to the first system is introduced in detail below.

[0087] Figure 1 FIG. 1 is a flow chart of a method for commodity identification applied to the first system in an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0088] Step 101: an image acquisition device acquires an image of a shelf to be identified and sends it to an image recognition server;

[0089] Step 102: The image recognition server performs commodity segmentation and recognition on the shelf image to be recognized, obtains a preliminary recognition result for each commodity, extracts features of each commodity with a preliminary recognition result from the shelf image to be recognized; matches each extracted feature with a commodity feature in a preset commodity feature library; generates a candidate commodity list with a matching value in the top N for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with information consistent with the candidate commodities and their bound electronic shelf labels in the first supermarket according to the candidate commodity list, and sends the search command to the first supermarket end;

[0090] Step 103: When the first supermarket terminal receives the search command, for each candidate product in the candidate product list, the first supermarket terminal performs the step of determining whether there is a product with the same information as the candidate product and its bound electronic shelf label in the first supermarket. If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; and a re-capture command for re-shooting the image of the shelf to be identified is sent to the image acquisition device through the image recognition server;

[0091] Step 104: If the electronic shelf label receives a command indicating a preset display mode, it is displayed in the preset display mode;

[0092] Step 105: When receiving the re-capture command, the image acquisition device re-captures the image of the shelf to be identified, and sends the re-captured image of the shelf to be identified to the image recognition server;

[0093] Step 106: The image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified. If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product identification result is obtained, indicating that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0094] The method for identifying goods provided by the embodiment of the present invention, when working: an image acquisition device acquires a shelf image to be identified and sends it to an image recognition server; the image recognition server performs product segmentation and identification on the shelf image to be identified to obtain a preliminary identification result of each product, extracts the features of each product with the preliminary identification result from the shelf image to be identified; matches each extracted feature with a product feature in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generates a candidate product list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for products with consistent candidate product information and electronic shelf labels bound thereto in a first supermarket according to the candidate product list, and sends the search command to the first supermarket end; when the first supermarket end receives the search command, for each product in the candidate product list, A candidate product executes the steps of determining whether there is a product consistent with the candidate product information and its bound electronic shelf label in the first supermarket, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; sending a re-capture command for reshooting the shelf image to be identified to the image acquisition device through the image recognition server; if the electronic shelf label receives the command indicating the preset display mode, it is displayed in the preset display mode; when receiving the re-capture command, the image acquisition device re-captures the shelf image to be identified, and sends the re-captured shelf image to be identified to the image recognition server; the image recognition server identifies whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0095] Compared with the technical solutions with low commodity identification accuracy in the prior art, the commodity identification method provided by the embodiment of the present invention can improve the accuracy of commodity identification. Figures 2 to 5 , the method of identifying the product is introduced in detail.

[0096] The method for commodity identification provided by the embodiment of the present invention may include the following steps:

[0097] In the above step 101, the shelf camera (image acquisition device) can be fixed opposite to or diagonally above the shelf being photographed, or set on the first robot walking in the supermarket. That is, in one embodiment, the image acquisition device is set on the shelf, or on the first robot walking. After the camera captures the picture of the shelf (shelf image), it is sent to the image recognition server, that is, the above step 101.

[0098] In the above step 102, the image recognition server performs product segmentation and recognition to obtain each product in the shelf image and its preliminary recognition result. The picture (image) recognition server can use the pre-trained deep learning network model (the input of the model can be the shelf image) to extract the features of each product in the image, and match and recognize the extracted features with the features in the preset product feature library in the image recognition server (the feature library stores all the features of all products). Taking a product in the shelf image as an example, for this product, the TOP-M recognition result will be returned after matching. By setting a double threshold value: the first matching degree threshold MinCV (Minimum Confidence Value), and the second matching degree threshold MaxCV (Maximum Confidence Value), the confidence level of the product recognition results can be classified. The schematic diagram of the double threshold value is shown as follows. Figure 2 For all matching results (matching values) that are lower than or equal to the MinCV threshold, the product is reported as an unrecognized product. In one embodiment, the above product identification method may further include: the image recognition server reports the product as an unrecognized product (i.e., an unknown product) for a product with a matching value less than or equal to the first matching value threshold; for a recognition result that is greater than or equal to the MaxCV threshold, the product is confirmed as a recognized product. In one embodiment, the above product identification method may further include: the image recognition server determines the preliminary recognition result of the product as a final recognition result for a product with a matching value greater than or equal to the second matching value threshold (i.e., a confirmed product). For the product recognition result, if there are multiple products with a matching degree higher than the second matching degree threshold, the product with the highest matching degree value is selected as the final product recognition result; for the product whose recognition result falls between MinCV and MaxCV (i.e., the product to be confirmed), find the TOP-N results of this product, find the list of top candidate products with the highest matching values, and further perform confirmation operations, that is, for products with a matching degree greater than the first matching degree threshold and less than the second matching degree threshold, generate a list of candidate products with matching degree values ​​in the top N, further confirm whether the candidate product is a real product on the supermarket side, and then confirm that the product with the preliminary recognition result is a product in the candidate product list to improve the accuracy of product recognition.

[0099] Product recognition outputs a list of candidate products between MinCV and MaxCV, such as Figure 3 shown.

[0100] In the above step 103, the image recognition server queries the top several candidate products for the product identification to be confirmed through the PS (the first supermarket end) and the bound ESL (electronic shelf label, which can be connected to the supermarket end PS through the base station AP), and sends a command to the bound ESL through the PS. The command issued by the PS to the bound ESL may include: issuing an instruction to make the ESL flash a specified color, or displaying special characters or graphics on the screen of the ESL. The product information list and the product information bound to the ESL in the PS need to be consistent, so that the corresponding ESL can be queried through the product name information. The first supermarket end executes the following steps: if the product information in the candidate product list does not find the corresponding product information in the corresponding store in the PS, then remove the candidate product information from the candidate product list, that is, in one embodiment, the above product identification method may also include: when the first supermarket end does not find a product consistent with the candidate product information in the supermarket, remove the current candidate product from the candidate product list. If all are eliminated, the commodity identification result is judged to be a new commodity, and the characteristics of the new commodity need to be collected by means of reminders. That is, in one embodiment, the above commodity identification method may also include: when all candidate commodities are removed from the candidate commodity list after detection at the first supermarket end, the commodity with the preliminary identification result is judged to be a new commodity, and a prompt message for collecting the characteristics of the new commodity is sent to the image acquisition device. If the PS finds the corresponding commodity and the bound ESL, a command is sent to instruct the ESL to display in a preset manner. For example, the PS sends a command to the bound ESL, including sending an instruction to let the ESL flash a specified color, or display special characters or graphics on the screen of the ESL.

[0101] Send commands according to the ESL corresponding to the product list to find the ESL corresponding to this shelf, such as Figure 4 shown.

[0102] At the same time, the PS sends a command to the camera to capture the image through the image recognition server.

[0103] In the above step 104, if the electronic shelf label receives a command indicating a preset display mode, it is displayed in the preset display mode.

[0104] In the above step 105, when the image acquisition device (camera) receives the re-acquisition command, it re-acquires the image of the shelf to be identified, and sends the re-acquired image of the shelf to be identified to the image recognition server.

[0105] In the above step 106, it is identified whether there is an electronic shelf label displayed in a preset display mode in the image recaptured by the camera (the re-collected shelf image to be identified), for example, whether there is a predetermined ESL that should display a light color, character or graphic. If not identified, the image server removes the product from the candidate product list, that is, in one embodiment, the above product identification method may also include: if the image recognition server does not identify the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified, the current candidate product is removed from the candidate product list. Send a command to the corresponding multiple ESLs through the PS, obtain the image taken by the camera, identify the special predetermined characters or pictures of the ESL, locate the position of the candidate product in the image, and improve the product identification confidence in the candidate product list through the retrieval strategy, so as to transform the product identification result to be confirmed into the identified product identification result. If the ESL in the PS fails to improve the product identification confidence in the candidate product list through the retrieval strategy in the image, the product is identified as a new product, and the characteristics of the new product need to be collected by reminders.

[0106] In specific implementation, the above retrieval strategy is to determine the distance between the position of the commodity to be confirmed and the position of the ESL in the image by normalizing the two images, and at the same time, considering the particularity of the shelf, locate the commodity and the ESL on the same layer of the shelf, so as to limit the retrieval constraint to the same layer of the shelf. If the identified electronic shelf label (electronic shelf label displayed in a preset display mode) appears within the preset electronic shelf label target area range corresponding to the preliminary identified commodity in the same shelf layer (the target area range can be a pre-set target area range on the shelf near the position of the electronic shelf label corresponding to the commodity, and can include a position that coincides with the preset electronic shelf label position), the final commodity recognition result is obtained that the commodity with the preliminary recognition result is bound to the electronic shelf label displayed in the preset display mode, and the confidence of this candidate commodity is improved to optimize the commodity feature database. If there are multiple bound ESL matches within the preset electronic shelf label target area corresponding to the product with the preliminary recognition result, the product bound to the electronic shelf label displayed in the preset display mode closest to the product will be used as the final product recognition result. At the same time, the confidence of the corresponding candidate product will be improved according to the distance to optimize the product feature database.

[0107] As can be seen from the above, in one embodiment, the image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be recognized, and if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, the final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label is obtained, which may include:

[0108] If the image recognition server identifies an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be recognized, the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be recognized is obtained; specifically, the image recognition server identifies the electronic price tag displayed in a preset manner, thereby locating the position of the electronic price tag in the shelf image, and automatically obtains the position information of the electronic shelf label;

[0109] Determine the position of the product with the preliminary recognition result corresponding to the current candidate product in the initially collected shelf image to be recognized; similarly, the image recognition server recognizes the product in the initially collected shelf image to be recognized, thereby locating the position of the product in the initially collected shelf image to be recognized, and automatically obtains the position information of the product;

[0110] Normalizing the initially acquired shelf image to be identified and the re-acquired shelf image to be identified to the same coordinate system to obtain a normalized shelf image;

[0111] Determine the distance between the product with the preliminary recognition result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be recognized and the position of the product in the initially collected shelf image to be recognized;

[0112] A final commodity recognition result is obtained according to the distance to determine whether the commodity with the preliminary recognition result is the commodity bound to the electronic shelf label.

[0113] In specific implementation, it is identified whether there are characters or pictures that the predetermined ESL should display in the recaptured picture. For the newly acquired picture, it is necessary to perform normalization processing with the picture of the last commodity identification. In combination with the position of the image acquisition device and the feature point matching of the shelf image, the coordinates of the two pictures are automatically normalized to the same coordinate system. According to the distance between the commodity and the electronic shelf label in the shelf image after normalization processing, the final commodity identification result is obtained. For example, when the distance between the commodity and the electronic shelf label is less than or equal to the preset threshold, the final commodity identification result is obtained that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label. For example, according to the shelf design, the actual distance between the commodity on the shelf and the electronic shelf label is 2 cm, then this preset threshold can be 2 cm, thereby further improving the accuracy of commodity identification.

[0114] Preferably, from the above, it can be known that in one embodiment, obtaining the final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label according to the distance may include:

[0115] Determine, based on the shelf layer position information, whether the product with the preliminary recognition result and the electronic shelf label displayed in the preset display mode are located on the same layer of the shelf;

[0116] If so, if an electronic shelf label displayed in a preset display mode appears within the preset electronic shelf label target area corresponding to the product with the preliminary recognition result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label displayed in the preset display mode, and the final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label displayed in the preset display mode is obtained. For example: for example, a list of candidate products with matching values ​​in the top N is generated, and the candidate products are celery and leeks. It is necessary to execute the steps of confirming which candidate product the product recognition result is one by one: first, determine whether the recognition result is celery, then, when the image acquisition device receives the re-capture command, it re-takes the image in front of the shelf corresponding to the product with the preliminary recognition result. In the supermarket, when the electronic shelf label corresponding to the celery receives the command indicating the preset display mode, If an electronic shelf label displayed in green appears in the target area corresponding to the preset electronic shelf label located on the same shelf layer as the product "celery" in the re-photographed shelf image to be identified, it is determined that the current candidate product belongs to the product (celery) bound to the corresponding electronic shelf label, and the final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label, that is, the product with the preliminary identification result is celery. On the contrary, if no electronic shelf label displayed in green appears in the re-photographed shelf image to be identified, it is determined that the current candidate product does not belong to the product (celery) bound to the corresponding electronic shelf label, that is, the product with the preliminary identification result is not celery, and then according to the above step of confirming which candidate product the product identification result is, it is confirmed whether the identification result is leek, which will not be repeated here.

[0117] Preferably, from the above, it can be known that in one embodiment, the above commodity identification method may include:

[0118] If there are multiple electronic shelf labels displayed in a preset display mode within the preset electronic shelf label target area corresponding to the commodity with the preliminary recognition result in the same shelf layer, the electronic shelf label displayed in the preset display mode closest to the commodity with the preliminary recognition result is obtained;

[0119] The product bound to the electronic shelf label displayed in a preset display mode closest to the product with the preliminary recognition result is used as the final product recognition result.

[0120] In specific implementation, if the distance between the commodity and the electronic shelf label on the shelf is exactly equal to the preset threshold value (2 cm), it is also possible that the commodity and the electronic shelf label are not on the same layer, which affects the accuracy of the recognition result. Therefore, considering the particularity of the shelf, the embodiment of the present invention locates the commodity and the ESL (electronic shelf label) on the same layer of the shelf, that is, excludes the electronic shelf label that is not located on the same layer as the commodity, and performs commodity recognition, which can further improve the accuracy of commodity recognition. For example, in the above example, there are two commodities in the candidate commodity list, namely celery and leek. If the electronic shelf labels corresponding to celery and leek are both displayed in a preset manner within the preset electronic shelf label target area range, for example, they are both displayed in green within the target area range, then the distances between the position A1 of the electronic shelf label corresponding to celery and the position A2 of the electronic shelf label corresponding to leek and the position of the commodity are calculated, and the commodity bound to the electronic shelf label displayed in the preset display mode closest to the commodity is used as the final commodity recognition result. For example, celery is closest to the commodity, and then the final recognition result that the commodity with the preliminary recognition result is celery is obtained.

[0121] During specific implementation, when calculating the above-mentioned distance, the embodiment of the present invention may use the center position of the product. For the convenience of calculation, the center position of the product frame obtained by performing target detection on the product may also be used. Similarly, the center position of the electronic shelf label may be used. For the convenience of calculation, the center position of the electronic shelf label frame obtained by performing target detection on the electronic shelf label may also be used.

[0122] Preferably, from the above, it can be seen that in one embodiment, the above-mentioned method of product identification may include: when the final product identification result of the product with the preliminary identification result obtained according to the distance is the product bound to the electronic shelf label displayed in a preset display mode, the confidence of the candidate product in the preset product feature library is improved to optimize the preset product feature library.

[0123] In the specific implementation, considering the particularity of the shelf, the product and ESL are located on the same shelf layer, so that the search constraints are limited to the same shelf layer. If an ESL bound to a product that is consistent with the candidate product information appears at the preset label position corresponding to the product in the same shelf layer (the preset label position is a label position pre-set under the product with preliminary recognition results according to the shelf design), the confidence of this candidate product is improved. If the bound ESLs of multiple products are matched at the same time, the confidence of the corresponding candidate products is improved according to the distance, realizing the optimization of the preset product feature library.

[0124] The image recognition server repeats the above steps until it finds the light color, character or graphic that the predetermined ESL corresponding to each product should display, then determines that the current product feature belongs to the product bound to this ESL, finds the corresponding ESL, and determines the result of product recognition. Figure 5 As shown. At the same time, for the product recognition results with improved confidence, the image and product information will be updated to the product feature library in pairs. After receiving the image and product information, the product feature library will automatically or manually register the updated product image and product information to the product feature library, and send the product feature library to the image server. In the subsequent product recognition, the new product features will be automatically associated to improve the accuracy of product recognition. Figures 3 to 5 In the above figure, confidence represents confidence, product represents candidate products, and goods-01 to goods17 represent candidate product identifiers, such as product names and numbers.

[0125] In order to facilitate understanding of how the present invention is implemented, an example is given below in which an image acquisition device is set on a first walking robot. Commodity identification is performed locally on an image recognition server, and the image recognition server obtains accurate judgment results of the identified commodities for those with a matching degree higher than a threshold (MaxCV, the second matching degree threshold). For commodity features whose matching degree with all commodities is less than a threshold (MinCV, the first matching degree threshold), find the top candidate commodities with the highest matching values ​​(a list of candidate commodities with matching values ​​in the top N), and record the shelf number corresponding to the image. After the first robot completes the inspection task, the image server commands the robot to return to the previously uncertain commodity shelf, and then:

[0126] 1) The image recognition server sends commands to the ESLs bound to the top candidate products through the PS. If the PS does not find the corresponding product in this store, the product is removed from the candidate product list. If the PS finds the corresponding product and the bound ESL, it sends a command instruction. The PS sends commands to the bound ESL, including sending instructions to make the ESL flash a specified color, or display special characters or graphics on the ESL screen.

[0127] 2) At the same time, the image recognition server sends a command to the robot to capture the image. In the recaptured image (the image to be recognized), it is determined whether there is a light color, character or graphic that the predetermined ESL should display. For the newly acquired image, it is necessary to normalize it with the image of the last product recognition. Combined with the position of the robot and the matching of the feature points of the image, the coordinates of the two images are automatically normalized to the same coordinate system. If it is not recognized, the image server removes the product from the candidate product list.

[0128] 3) The image server repeats the above steps until it finds the light color, character or graphic that the corresponding predetermined ESL should display, and determines that the current product feature belongs to the product bound to this ESL through the retrieval strategy mentioned above, and updates the product feature library at the same time; or it exits after all candidate products are traversed and the most matching product is not found.

[0129] The following describes further preferred solutions of the embodiments of the present invention.

[0130] 1. The update of the product feature library can include adding new images to existing products, and automatically eliminating existing products with poor quality. By continuously screening out high-quality product images and product information, a high-definition version of the product data feature library is constructed. Since the iteration of the product feature library is a frequent operation, there will be incremental updates every day under normal circumstances. By flexibly selecting the time window range, the incremental product images and product information are merged into the full feature library. On the one hand, the incremental feature library can be quickly applied to the image recognition service. On the other hand, if problems are found in the data, the problematic data can be quickly modified through the time window before merging.

[0131] 2. In addition to incremental updates, the product feature library also needs to continuously maintain data quality. By performing consistency checks on the product feature library, you can quickly filter out images of unqualified quality. The consistency check includes image size, image resolution, etc. to check problematic images.

[0132] 3. In addition to consistency checks, the product feature library also considers data elimination. By binding ESL product data with PS stores and maintaining image data during the window period, data outside the window is automatically eliminated. By setting the window period to N days, if the PS cancels the binding of the product of this store and it does not appear for N days, it is considered that this product can be eliminated. In this way, the data will not expand too quickly, and the data quality within the window period will be guaranteed to be higher and higher, thereby improving the accuracy of product identification.

[0133] The embodiment of the present invention also provides a method for commodity identification applied to an image recognition server, as described in the following embodiment. Since the principle of solving the problem by this method is similar to that of the method for commodity identification applied to the first system, the implementation of the method for commodity identification applied to the image recognition server can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0134] Figure 6 FIG. 1 is a flow chart of a method for commodity identification applied to an image recognition server in an embodiment of the present invention. Figure 6 As shown, the method comprises the following steps:

[0135] Step 201: segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with the preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by an image acquisition device;

[0136] Step 202: Match each extracted feature with a commodity feature in a preset commodity feature library;

[0137] Step 203: for products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, generate a candidate product list whose matching degree values ​​are in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0138] Step 204: Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list, and send the search command to the first supermarket end; the first supermarket end is used to, upon receiving the search command, execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used to display in the preset display mode upon receiving the command indicating a preset display mode; send a re-capture command for re-shooting the shelf image to be identified to the image acquisition device; the image acquisition device is used to re-capture the shelf image to be identified upon receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server;

[0139] Step 205: Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0140] Step 206: If it is determined that the current candidate product is a product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is a product bound to the electronic shelf label.

[0141] In one embodiment, the above commodity identification method may further include: for a commodity whose matching degree is greater than or equal to a second matching degree threshold, determining the preliminary identification result of the commodity as the final commodity identification result.

[0142] In one embodiment, the above commodity identification method may further include: for a commodity whose matching degree is less than or equal to a first matching degree threshold, reporting the commodity as an unidentified commodity.

[0143] In one embodiment, the above-mentioned commodity identification method may further include: when the first supermarket terminal fails to find a commodity that is consistent with the candidate commodity information in the supermarket, removing the current candidate commodity from the candidate commodity list.

[0144] In one embodiment, the above-mentioned product identification method may also include: when all candidate products are removed from the candidate product list after detection at the first supermarket end, the product with the preliminary identification result is judged to be a new product, and a prompt information for collecting new product features is sent to the image acquisition device.

[0145] In one embodiment, the above-mentioned commodity identification method may further include: if the electronic shelf label displayed in the preset display mode is not identified in the re-collected shelf image to be identified, removing the current candidate commodity from the candidate commodity list.

[0146] In one embodiment, it is identified whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified, and if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label, including:

[0147] If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, obtaining a position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified;

[0148] Determine the position of the product with a preliminary recognition result corresponding to the current candidate product in the initially collected shelf image to be recognized;

[0149] Normalizing the initially acquired shelf image to be identified and the re-acquired shelf image to be identified to the same coordinate system to obtain a normalized shelf image;

[0150] Determine the distance between the product with the preliminary recognition result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be recognized and the position of the product in the initially collected shelf image to be recognized;

[0151] A final commodity recognition result is obtained according to the distance to determine whether the commodity with the preliminary recognition result is the commodity bound to the electronic shelf label.

[0152] In one embodiment, obtaining a final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label according to the distance includes:

[0153] Determine, based on the shelf layer position information, whether the product with the preliminary recognition result and the electronic shelf label displayed in the preset display mode are located on the same layer of the shelf;

[0154] If so, if an electronic shelf label displayed in a preset display mode appears within the preset electronic shelf label target area corresponding to the product with the preliminary recognition result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label displayed in the preset display mode, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label displayed in the preset display mode.

[0155] In one embodiment, the above commodity identification method may further include:

[0156] If there are multiple electronic shelf labels displayed in a preset display mode within the preset electronic shelf label target area corresponding to the commodity with the preliminary recognition result in the same shelf layer, the electronic shelf label displayed in the preset display mode closest to the commodity with the preliminary recognition result is obtained;

[0157] The product bound to the electronic shelf label displayed in a preset display mode closest to the product with the preliminary recognition result is used as the final product recognition result.

[0158] In one embodiment, the above-mentioned method for identifying goods may further include: when a final product identification result is obtained based on the distance that the product with the preliminary identification result is the product bound to the electronic shelf label, improving the confidence of the candidate products in the preset product feature library to optimize the preset product feature library.

[0159] In one embodiment, the image acquisition device is disposed on the shelf, or on the first walking robot.

[0160] The embodiment of the present invention also provides a method for identifying goods at the first supermarket end, as described in the following embodiment. Since the principle of solving the problem by this method is similar to that of the method for identifying goods at the first system, the implementation of the method for identifying goods at the first supermarket end can refer to the implementation of the method for identifying goods at the first system, and the repeated parts will not be repeated.

[0161] Figure 7 FIG. 1 is a flow chart of a method for commodity identification applied to a first supermarket terminal in an embodiment of the present invention. Figure 7 As shown, the method for commodity identification applied to the first supermarket terminal includes the following steps:

[0162] Step 301: Obtain a search command sent by an image recognition server for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in a first supermarket based on a candidate product list; the image recognition server is used to segment and identify products on the shelf image to be identified, obtain a preliminary identification result for each product, extract features of each product with a preliminary identification result from the shelf image to be identified; match each extracted feature with a product feature in a preset product feature library; for products whose matching degree is greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in a first supermarket based on the candidate product list, and send the search command to the first supermarket end; the shelf image to be identified is sent by an image acquisition device;

[0163] Step 302: upon receiving the search command, for each candidate product in the candidate product list, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label;

[0164] Step 303: If it exists, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a re-capture command for reshooting the shelf image to be identified to the image acquisition device through the image recognition server; the electronic shelf label is used to display in the preset display mode if it receives the command indicating the preset display mode; the image acquisition device is also used to re-capture the shelf image to be identified when receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain a final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0165] In another commodity identification system of the present invention, the robot itself can collect images of shelves to be identified and interact with supermarkets and electronic tags, and improve the accuracy of corresponding commodity identification by searching and confirming candidate commodities with a matching degree between "MinCV and MaxCV" in supermarkets. That is, a method for commodity identification applied to a second system is also provided in an embodiment of the present invention, as described in the following embodiment. Since the principle of solving the problem in this method is similar to that of the commodity identification method applied to the first system, the implementation of the commodity identification method applied to the second system can refer to the implementation of the commodity identification method applied to the first system. The commodity identification method applied to the second system is introduced in detail below.

[0166] Figure 8 FIG. 1 is a flow chart of a method for commodity identification applied to the second system in an embodiment of the present invention. Figure 8 As shown, the method comprises the following steps:

[0167] Step 401: The second robot collects a shelf image to be identified; performs commodity segmentation and identification on the shelf image to be identified to obtain a preliminary identification result for each commodity, extracts features of each commodity with a preliminary identification result from the shelf image to be identified; matches each extracted feature with a commodity feature in a preset commodity feature library; generates a candidate commodity list with a matching value in the top N for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with information consistent with the candidate commodities and their bound electronic shelf labels in the first supermarket according to the candidate commodity list, and sends the search command to the second supermarket end;

[0168] Step 402: When the second supermarket terminal receives the search command, for each candidate product in the candidate product list, the second supermarket terminal performs the step of determining whether there is a product with the same information as the candidate product and its bound electronic shelf label in the second supermarket. If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; and a command to re-shoot the image of the shelf to be identified is sent to the second robot.

[0169] Step 403: If the electronic shelf label receives a command indicating a preset display mode, it is displayed in the preset display mode;

[0170] Step 404: When receiving the re-collection command, the second robot re-collects the shelf image to be identified, and identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified. If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0171] The method for commodity identification applied to the first system provided by the embodiment of the present invention, when working: the second robot collects the shelf image to be identified; the shelf image to be identified performs commodity segmentation and identification to obtain a preliminary identification result of each commodity, and extracts the feature of each commodity with the preliminary identification result from the shelf image to be identified; each extracted feature is matched with the commodity feature in a preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, a list of candidate commodities with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; a search command is generated in the first supermarket based on the candidate commodity list to search for commodities with consistent candidate commodity information and their bound electronic shelf labels, and the search command is sent to the second supermarket end; the second supermarket end receives the When the search command is received, for each candidate product in the candidate product list, the step of determining whether there is a product with the same information as the candidate product and its bound electronic shelf label in the second supermarket is executed. If there is, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; a command to re-shoot the shelf image to be identified is sent to the second robot; if the electronic shelf label receives the command indicating a preset display mode, it is displayed in the preset display mode; when the second robot receives the re-collection command, it re-collects the shelf image to be identified, and identifies whether there is an electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified. If there is, it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and the final product identification result is obtained that the product with the preliminary identification result is the product bound to the electronic shelf label. The method for identifying products applied to the first system is described in detail below.

[0172] In the above step 401, the second robot (the robot in this embodiment has the functions of the image acquisition device and the image recognition server in the first system) takes photos to identify commodities while walking. Commodity recognition can be performed locally on the robot or on the image recognition server. For all matching results (matching values) that are lower than or equal to the MinCV threshold, report the commodity as an unrecognized commodity, that is, in one embodiment, the above commodity recognition method may also include: for commodities with a matching degree less than or equal to the first matching degree threshold, report the commodity as an unrecognized commodity; for accurate judgment results of the identified commodities with a matching degree greater than or equal to the MaxCV threshold, directly judge them as recognized results, that is, in one embodiment, the above commodity recognition method may also include: for commodities with a matching degree greater than or equal to the second matching degree threshold, determine the preliminary recognition result of the commodity as the final commodity recognition result. For product features that match all products between the MinCV and MaxCV thresholds, find the top candidate products with the highest matching values, that is, for products with a matching degree greater than the first matching threshold and less than the second matching threshold, generate a list of candidate products with matching values ​​in the top N, and further confirm whether the candidate product is a real product on the supermarket side, and then confirm that the product with the preliminary recognition result is a product in the candidate product list to improve the accuracy of product recognition. If the product recognition is performed locally on the robot, the robot stays in front of the corresponding shelf at this time. The robot (a second robot, which has image acquisition, i.e., image recognition function, and Fig. 9 The function shown) sends a command to the ESL bound to the corresponding TOP-N candidate product through PS (the second supermarket end).

[0173] In the above step 402, the execution step of the second supermarket end is: if the corresponding product is not found in the PS in this store, then the product is removed from the candidate product list, that is, in one embodiment, the above product identification method may also include: when the first supermarket end does not find the product consistent with the candidate product information in the supermarket, the current candidate product is removed from the candidate product list. If all are removed, it is judged that the product identification result is a new product, and the characteristics of the new product need to be collected by means of reminders, that is, in one embodiment, the above product identification method may also include: when all candidate products are removed from the candidate product list after detection at the first supermarket end, it is judged that the product with the preliminary recognition result is a new product, and a prompt information for collecting the characteristics of the new product is sent to the image acquisition device. If the PS finds the corresponding product and the bound ESL, a command instruction is sent, and the PS sends a command to the bound ESL, including sending an instruction to make the ESL flash a specified color light color, or display special characters or graphics on the screen of the ESL.

[0174] In the above step 403, if the electronic shelf label receives a command indicating a preset display mode, it displays in the preset display mode.

[0175] In the above step 404, the robot captures the image, and identifies whether there is a light color, character or graphic that the predetermined ESL should display in the recaptured image. For the newly acquired image, it is necessary to normalize it with the image of the product recognition last time (the first acquisition or the last cycle acquisition), and automatically normalize the coordinates of the two images to the same coordinate system in combination with the position of the robot and the matching of the feature points of the image. If it is not recognized, the robot removes the product from the candidate product list, that is, in one embodiment, the above product recognition method may also include: if the second robot does not recognize the electronic shelf label displayed in the preset display mode in the recaptured shelf image to be recognized, remove the current candidate product from the candidate product list.

[0176] The second robot repeats the above steps until it finds the characters or pictures that should be displayed in the predetermined ESL corresponding to each candidate product, determines through the retrieval strategy that the current product features belong to the product bound to this ESL, and updates the product feature library at the same time; or exits after all candidate products are traversed and the most matching product is not found.

[0177] As can be seen from the above, in one embodiment, if it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, obtaining the final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label may include:

[0178] If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, obtaining a position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified;

[0179] Determine the position of the product with a preliminary recognition result corresponding to the current candidate product in the initially collected shelf image to be recognized;

[0180] If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, obtaining a position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified;

[0181] Determine the position of the product with a preliminary recognition result corresponding to the current candidate product in the initially collected shelf image to be recognized;

[0182] Normalizing the initially acquired shelf image to be identified and the re-acquired shelf image to be identified to the same coordinate system to obtain a normalized shelf image;

[0183] Determine the distance between the product with the preliminary recognition result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be recognized and the position of the product in the initially collected shelf image to be recognized;

[0184] A final commodity recognition result is obtained according to the distance to determine whether the commodity with the preliminary recognition result is the commodity bound to the electronic shelf label.

[0185] As can be seen from the above, in one embodiment, obtaining a final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label according to the distance includes:

[0186] Determine, based on the shelf layer position information, whether the product with the preliminary recognition result and the electronic shelf label displayed in the preset display mode are located on the same layer of the shelf;

[0187] If so, if an electronic shelf label displayed in a preset display mode appears within the preset electronic shelf label target area corresponding to the product with the preliminary recognition result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label displayed in the preset display mode, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label displayed in the preset display mode.

[0188] As can be seen from the above, in one embodiment, the above commodity identification also includes:

[0189] If there are multiple electronic shelf labels displayed in a preset display mode within the preset electronic shelf label target area corresponding to the commodity with the preliminary recognition result in the same shelf layer, the electronic shelf label displayed in the preset display mode closest to the commodity with the preliminary recognition result is obtained;

[0190] The product bound to the electronic shelf label displayed in a preset display mode closest to the product with the preliminary recognition result is used as the final product recognition result.

[0191] From the above, it can be seen that in one embodiment, the above-mentioned product identification also includes: when the final product identification result of the product with the preliminary identification result obtained according to the distance is the product bound to the electronic shelf label displayed in the preset display mode, the confidence of the candidate product in the preset product feature library is improved to optimize the preset product feature library.

[0192] During the specific implementation, the robot normalizes the two pictures. Through the robot's position and matching the feature points of the images, the coordinates of the two pictures are automatically normalized to the same coordinate system, further improving the accuracy of product recognition.

[0193] The embodiment of the present invention also provides a method for commodity identification applied to the second robot, as described in the following embodiment. Since the principle of solving the problem by this method is similar to that of the method for commodity identification applied to the first system, the implementation of the method for commodity identification applied to the second robot can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0194] Fig. 9 FIG. 1 is a flow chart of a method for commodity identification applied to a second robot in an embodiment of the present invention. Fig. 9 As shown, the method comprises the following steps:

[0195] Step 501: Collect the image of the shelf to be identified;

[0196] Step 502: segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, and extract features of each commodity with a preliminary identification result from the shelf image to be identified;

[0197] Step 503: Match each extracted feature with a commodity feature in a preset commodity feature library;

[0198] Step 504: for the products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, generate a candidate product list whose matching degree values ​​are in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0199] Step 505: Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the second supermarket based on the candidate product list, and send the search command to the second supermarket end; the second supermarket end is used to execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the second supermarket for each candidate product in the candidate product list upon receiving the search command, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating a preset display mode;

[0200] Step 506: upon receiving a re-capture command, re-capture the image of the shelf to be identified;

[0201] Step 507: Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0202] Step 508: If it is determined that the current candidate product belongs to the product bound to the corresponding electronic shelf label, a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0203] The embodiment of the present invention also provides a first system for commodity identification, as described in the following embodiment. Since the principle of solving the problem by the first system is similar to the commodity identification method applied to the first system, the implementation of the first system can refer to the implementation of the commodity identification method applied to the first system, and the repeated parts will not be repeated.

[0204] Fig.10 FIG. 1 is a schematic diagram of the structure of the first system for commodity identification in an embodiment of the present invention. Fig.10 As shown, the first system includes:

[0205] The image acquisition device 11 is used to acquire the image of the shelf to be identified and send it to the image recognition server; when receiving a re-acquisition command, the image of the shelf to be identified is re-acquired, and the re-acquired image of the shelf to be identified is sent to the image recognition server;

[0206] The image recognition server 12 is used to segment and identify commodities in the shelf image to be identified, obtain a preliminary identification result of each commodity, extract features of each commodity with a preliminary identification result from the shelf image to be identified; match each extracted feature with a commodity feature in a preset commodity feature library; generate a candidate commodity list with a matching value in the top N for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for commodities with information consistent with the candidate commodities and their bound electronic shelf labels in the first supermarket according to the candidate commodity list, and send the search command to the first supermarket end; identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified, and if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label, obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label;

[0207] The first supermarket terminal 13 is used for, upon receiving the search command, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a re-capture command for re-shooting the shelf image to be identified to the image acquisition device through the image recognition server;

[0208] The electronic shelf label 14 is used to display in the preset display mode if a command indicating the preset display mode is received.

[0209] The embodiment of the present invention also provides an image recognition server for commodity identification, as described in the following embodiment. Since the principle of solving the problem by the image recognition server is similar to the commodity identification method applied to the first system, the implementation of the image recognition server can refer to the implementation of the commodity identification method applied to the first system, and the repeated parts will not be repeated.

[0210] Fig.11 FIG. 1 is a schematic diagram of the structure of an image recognition server for commodity recognition in an embodiment of the present invention. Fig.11 As shown, the server includes:

[0211] The embodiment of the present invention further provides an image recognition server for commodity identification, which is used to improve the accuracy of commodity identification. The image recognition server includes:

[0212] The first feature extraction unit 121 is used to segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device;

[0213] A first matching processing unit 122, used to match each extracted feature with a commodity feature in a preset commodity feature library;

[0214] A first candidate product list determining unit 123 is configured to generate a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0215] The first generating unit 124 is used to generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket according to the candidate product list, and send the search command to the first supermarket end; the first supermarket end is used to, upon receiving the search command, execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used to display in the preset display mode upon receiving the command indicating a preset display mode; send a re-capture command for re-shooting the shelf image to be identified to the image acquisition device; the image acquisition device is used to re-capture the shelf image to be identified upon receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server;

[0216] A first label detection unit 125, used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0217] The first commodity identification unit 126 is configured to obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label.

[0218] The embodiment of the present invention also provides a first supermarket terminal for commodity identification, as described in the following embodiment. Since the principle of solving the problem by the first supermarket terminal is similar to the method for commodity identification applied to the first system, the implementation of the first supermarket terminal can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0219] Fig.12 FIG. 1 is a schematic diagram of the structure of the first supermarket terminal for commodity identification in an embodiment of the present invention. Fig.12 As shown, the first supermarket terminal includes:

[0220] The first acquisition unit 131 is used to acquire a search command sent by the image recognition server for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list; the image recognition server is used to segment and identify the products on the shelf image to be identified, obtain a preliminary identification result of each product, extract the features of each product with the preliminary identification result from the shelf image to be identified; match each extracted feature with the product features in a preset product feature library; for products with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate product list with matching degree values ​​in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list, and send the search command to the first supermarket end; the shelf image to be identified is sent by the image acquisition device;

[0221] The first commodity detection unit 132 is used for executing the step of determining whether there is a commodity consistent with the candidate commodity information and its bound electronic shelf label in the first supermarket for each candidate commodity in the candidate commodity list upon receiving the search command;

[0222] The first sending unit 133 is used to send a command indicating a preset display mode to the corresponding bound electronic shelf label if it exists; send a re-capture command for re-shooting the shelf image to be identified to the image acquisition device through the image recognition server; the electronic shelf label is used to display in the preset display mode if it receives the command indicating the preset display mode; the image acquisition device is also used to re-capture the shelf image to be identified when receiving the re-capture command, and send the re-captured shelf image to be identified to the image recognition server; the image recognition server is also used to identify whether there is an electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain a final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0223] The embodiment of the present invention also provides a second system for commodity identification, as described in the following embodiment. Since the principle of solving the problem by the second system is similar to the commodity identification method applied to the first system, the implementation of the second system can refer to the implementation of the commodity identification method applied to the first system, and the repeated parts will not be repeated.

[0224] Fig.13 FIG. 1 is a schematic diagram of the structure of the second system for commodity identification in an embodiment of the present invention. Fig.13 As shown, the second system includes:

[0225] The second robot 21 is used to collect the shelf image to be identified; perform commodity segmentation and identification on the shelf image to be identified to obtain a preliminary identification result of each commodity, extract the features of each commodity with the preliminary identification result from the shelf image to be identified; match each extracted feature with the commodity features in the preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generate a candidate commodity list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a search command for searching for commodities with the same information as the candidate commodities and their bound electronic shelf labels in the second supermarket according to the candidate commodity list, and send the search command to the second supermarket end; upon receiving the re-collection command, re-collect the shelf image to be identified, identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified, and if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label, obtain a final commodity recognition result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label;

[0226] The second supermarket terminal 22 is used for, upon receiving the search command, executing the step of determining whether there is a product in the second supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a command to retake the image of the shelf to be identified to the second robot;

[0227] The electronic shelf label 23 is used to display in the preset display mode if a command indicating the preset display mode is received.

[0228] The embodiment of the present invention also provides a second robot for commodity identification, as described in the following embodiment. Since the principle of the second robot to solve the problem is similar to the method for commodity identification applied to the first system, the implementation of the second robot can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0229] Fig.14 FIG. 1 is a schematic diagram of the structure of a second robot for commodity recognition in an embodiment of the present invention. Fig.14 As shown, the second robot comprises:

[0230] The acquisition unit 211 is used to acquire the image of the shelf to be identified; when receiving a re-acquisition command, the image of the shelf to be identified is re-acquired;

[0231] The second feature extraction unit 212 is used to segment and identify commodities on the initially collected shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified;

[0232] A second matching processing unit 213 is used to match each extracted feature with a commodity feature in a preset commodity feature library;

[0233] A second candidate product list determining unit 214 is configured to generate a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold;

[0234] The second generating unit 215 is used to generate a search command for searching for a commodity consistent with the candidate commodity information and its bound electronic shelf label in the second supermarket according to the candidate commodity list, and send the search command to the second supermarket end; the second supermarket end is used to execute the step of determining whether there is a commodity consistent with the candidate commodity information and its bound electronic shelf label in the second supermarket for each candidate commodity in the candidate commodity list upon receiving the search command, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating a preset display mode;

[0235] A second label detection unit 216, used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified;

[0236] The second commodity identification unit 217 is configured to obtain a final commodity identification result that the commodity with the preliminary identification result is the commodity bound to the electronic shelf label if it is determined that the current candidate commodity belongs to the commodity bound to the corresponding electronic shelf label.

[0237] The embodiment of the present invention also provides a method for commodity identification applied to the second supermarket end, as described in the following embodiment. Since the principle of solving the problem by this method is similar to that of the method for commodity identification applied to the first system, the implementation of this method can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0238] The method for commodity identification applied to the second supermarket terminal comprises the following steps:

[0239] Obtain a command from the second robot to search for a commodity consistent with the candidate commodity information and its bound electronic shelf label in the first supermarket; the second robot is used to collect a shelf image to be identified; perform commodity segmentation and identification on the initially collected shelf image to be identified to obtain a preliminary identification result of each commodity, extract the feature of each commodity with a preliminary identification result from the shelf image to be identified; match the extracted feature of each commodity with a preliminary identification result with the commodity feature in a preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, find a candidate commodity list whose feature matching degree values ​​with the commodity with the preliminary identification result are in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a command to search for commodities consistent with the candidate commodity information and its bound electronic shelf label in the first supermarket according to the candidate commodity list, and send the command to the second supermarket end;

[0240] When the command is received, for each candidate product in the candidate product list, a step is performed to determine whether there is a product in the first supermarket that is consistent with the candidate product information and a bound electronic shelf label thereof;

[0241] If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; a command to re-shoot the shelf image to be identified is sent to the second robot; the electronic shelf label is used to display in the preset display mode if the command indicating a preset display mode is received; the second robot is also used to collect the shelf image to be identified when receiving the command to re-shoot the shelf image to be identified; identify whether there is an electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain a final product identification result that the product with the preliminary identification result is the product bound to the electronic shelf label.

[0242] The embodiment of the present invention also provides a second supermarket terminal for commodity identification, as described in the following embodiment. Since the principle of solving the problem by the second supermarket terminal is similar to the method for commodity identification applied to the first system, the implementation of the second supermarket terminal can refer to the implementation of the method for commodity identification applied to the first system, and the repeated parts will not be repeated.

[0243] The second supermarket terminal for product identification includes:

[0244] The second acquisition unit is used to acquire a command sent by the second robot to search for a commodity consistent with the candidate commodity information and its bound electronic shelf label in the first supermarket; the second robot is used to collect a shelf image to be identified; perform commodity segmentation and identification on the initially collected shelf image to be identified to obtain a preliminary identification result of each commodity, and extract the feature of each commodity with a preliminary identification result from the shelf image to be identified; match the extracted feature of each commodity with a preliminary identification result with the commodity feature in a preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, find a candidate commodity list whose feature matching degree values ​​with the commodity with the preliminary identification result are in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generate a command to search for commodities consistent with the candidate commodity information and its bound electronic shelf label in the first supermarket according to the candidate commodity list, and send the command to the second supermarket end;

[0245] A second commodity detection unit is configured to, upon receiving the command, perform a step of determining, for each candidate commodity in the candidate commodity list, whether there is a commodity consistent with the candidate commodity information and a bound electronic shelf label thereof in the first supermarket;

[0246] The second sending unit is used to send a command indicating a preset display mode to the corresponding bound electronic shelf label if it exists; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating the preset display mode; the second robot is also used to collect the shelf image to be identified when receiving the command to re-shoot the shelf image to be identified; identify whether there is an electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified, and if so, determine that the current candidate product belongs to the product bound to the corresponding electronic shelf label, and obtain a final product recognition result that the product with the preliminary recognition result is the product bound to the electronic shelf label.

[0247] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned commodity identification method when executing the computer program.

[0248] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned commodity identification method is implemented.

[0249] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for identifying a product is implemented.

[0250] The product identification solution provided by the present invention realizes:

[0251] 1) In the first system of the present invention, the image acquisition device or robot improves the accuracy of identifying uncertain product recognition results by interacting with the ESL system (which may include: the first supermarket terminal, AP, and multiple electronic shelf labels).

[0252] 2) The dual threshold method filters out the list of products to be confirmed. The dual threshold method can be used to discover unknown products and products to be confirmed, as well as confirm products, thus improving the accuracy of product identification.

[0253] 3) In the second system of the present invention, the robot normalizes the two images (the first captured and the re-captured) and automatically normalizes the coordinates of the two images to the same coordinate system through the robot's position and matching of feature points of the images.

[0254] 4) Map product information to ESL, filter out product information with low confidence, and aggregate the price tags corresponding to the ESL system.

[0255] 5) By combining the camera system with shooting, acquiring and recognizing images through flashing lights / cutting pages, the designated ESL position and ESL number are found.

[0256] 6) Association method: The distance between the product layout and the specified ESL position can be used to exclude the unmatched items in the candidate list and determine the final product recognition result in the product candidate list. If the electronic price tag box meets the preset range below the product box to be confirmed, the association is successful.

[0257] 7) Combined with the product information and window period strategy bound to ESL in PS, the data in the product feature library is automatically eliminated to prevent the data from expanding too quickly. At the same time, the data quality within the window period is ensured to be higher and higher, thereby improving the accuracy of product identification.

[0258] In summary, the commodity identification solution provided by the embodiment of the present invention can improve the accuracy of commodity identification.

[0259] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0260] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0261] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0262] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0263] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for commodity identification, characterized in that: The commodity identification method is applied to the first system, and the method includes: The image acquisition device acquires the image of the shelf to be identified and sends it to the image recognition server; The image recognition server performs commodity segmentation and recognition on the shelf image to be recognized, obtains a preliminary recognition result of each commodity, extracts features of each commodity with the preliminary recognition result from the shelf image to be recognized; matches each extracted feature with a commodity feature in a preset commodity feature library; generates a candidate commodity list with a matching value in the top N for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with consistent candidate commodity information and their bound electronic shelf labels in a first supermarket according to the candidate commodity list, and sends the search command to the first supermarket end; When the first supermarket terminal receives the search command, for each candidate product in the candidate product list, the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label is performed. If so, a command indicating a preset display mode is sent to the corresponding bound electronic shelf label; a re-capture command for re-shooting the shelf image to be identified is sent to the image acquisition device through the image recognition server; If the electronic shelf label receives a command indicating a preset display mode, it displays in the preset display mode; When receiving the re-capture command, the image acquisition device re-captures the image of the shelf to be identified, and sends the re-captured image of the shelf to be identified to the image recognition server; The image recognition server identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be recognized. If an electronic shelf label displayed in a preset display mode is recognized in the re-collected shelf image to be recognized, the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be recognized is obtained; the position of the commodity with a preliminary recognition result corresponding to the current candidate commodity in the initially collected shelf image to be recognized is determined; the initially collected shelf image to be recognized and the re-collected shelf image to be recognized are normalized to the same coordinate system to obtain a normalized shelf image; according to the position of the electronic shelf label in the re-collected shelf image to be recognized and the position of the commodity in the initially collected shelf image to be recognized, the distance between the commodity with a preliminary recognition result and the electronic shelf label in the normalized shelf image is determined; and according to the distance, a final commodity recognition result of whether the commodity with a preliminary recognition result is the commodity bound to the electronic shelf label is obtained.

2. The method according to claim 1, characterized in that The image acquisition device is arranged on the shelf, or on the first walking robot.

3. A method for commodity identification, characterized in that: The commodity recognition method is applied to an image recognition server, and the method comprises: Segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by an image acquisition device; Match each extracted feature with the product features in the preset product feature library; For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the first supermarket based on the candidate product list, and send the search command to the first supermarket end; the first supermarket end is used to, upon receiving the search command, execute the step of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the first supermarket for each candidate product in the candidate product list, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used to display in the preset display mode if receiving the command indicating the preset display mode; send a re-capture command for re-shooting the image of the shelf to be identified to the image acquisition device; the image acquisition device is used to re-capture the image of the shelf to be identified when receiving the re-capture command, and send the re-captured image of the shelf to be identified to the image recognition server; Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified; If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified is obtained; the position of the commodity with a preliminary recognition result corresponding to the current candidate commodity in the initially collected shelf image to be identified is determined; the initially collected shelf image to be identified and the re-collected shelf image to be identified are normalized to the same coordinate system to obtain a normalized shelf image; according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified, the distance between the commodity with a preliminary recognition result and the electronic shelf label in the normalized shelf image is determined; and according to the distance, a final commodity recognition result of whether the commodity with a preliminary recognition result is the commodity bound to the electronic shelf label is obtained.

4. The method according to claim 3, characterized in that Also includes: For a commodity whose matching degree is greater than or equal to the second matching degree threshold, the preliminary identification result of the commodity is determined as the final commodity identification result.

5. The method according to claim 3, characterized in that Also includes: For a product whose matching degree is less than or equal to the first matching degree threshold, the product is reported as an unidentified product.

6. The method according to claim 3, characterized in that Also includes: When the first supermarket terminal fails to find a product that is consistent with the candidate product information in the supermarket, the current candidate product is removed from the candidate product list.

7. The method according to claim 6, characterized in that Also includes: When all candidate commodities are removed from the candidate commodity list after detection at the first supermarket end, the commodity with the preliminary recognition result is determined to be a new commodity, and prompt information for collecting features of the new commodity is sent to the image acquisition device.

8. The method according to claim 3, characterized in that Also includes: If the electronic shelf label displayed in the preset display mode is not recognized in the re-collected shelf image to be recognized, the current candidate product is removed from the candidate product list.

9. The method according to claim 3, characterized in that The final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label is obtained according to the distance, including: Determine, based on the shelf layer position information, whether the product with the preliminary recognition result and the electronic shelf label displayed in the preset display mode are located at the same layer of the shelf; If so, if an electronic shelf label displayed in a preset display mode appears within the preset electronic shelf label target area corresponding to the product with the preliminary recognition result in the same shelf layer, it is determined that the current candidate product belongs to the product bound to the electronic shelf label displayed in the preset display mode, and a final product recognition result is obtained that the product with the preliminary recognition result is the product bound to the electronic shelf label displayed in the preset display mode.

10. The method according to claim 9, characterized in that Also includes: If there are multiple electronic shelf labels displayed in a preset display mode within the preset electronic shelf label target area corresponding to the commodity with the preliminary recognition result in the same shelf layer, the electronic shelf label displayed in the preset display mode closest to the commodity with the preliminary recognition result is obtained; The product bound to the electronic shelf label displayed in a preset display mode closest to the product with the preliminary recognition result is used as the final product recognition result.

11. The method of claim 3, further comprising: When a final commodity recognition result is obtained according to the distance that the commodity with the preliminary recognition result is the commodity bound to the electronic shelf label displayed in a preset display mode, the confidence of the candidate commodity in the preset commodity feature library is improved to optimize the preset commodity feature library.

12. A method for commodity identification, characterized in that: The commodity identification method is applied to the second system, and the method includes: The second robot collects the shelf image to be identified; performs commodity segmentation and identification on the shelf image to be identified to obtain a preliminary identification result of each commodity, and extracts the features of each commodity with the preliminary identification result from the shelf image to be identified; matches each extracted feature with the commodity feature in the preset commodity feature library; for commodities with a matching degree greater than a first matching degree threshold and less than a second matching degree threshold, generates a candidate commodity list with a matching degree value in the top N, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; generates a search command for searching for commodities with consistent candidate commodity information and their bound electronic shelf labels in the first supermarket according to the candidate commodity list, and sends the search command to the second supermarket end; When the second supermarket terminal receives the search command, for each candidate product in the candidate product list, the second supermarket terminal performs the steps of determining whether there is a product with the same information as the candidate product and its bound electronic shelf label in the second supermarket, and if so, sends a command indicating a preset display mode to the corresponding bound electronic shelf label; and sends a command to re-shoot the image of the shelf to be identified to the second robot; If the electronic shelf label receives a command indicating a preset display mode, it displays in the preset display mode; When the second robot receives the re-collection command, it re-collects the shelf image to be identified, identifies whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified, and if an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, obtains the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified; determines the position of the commodity with a preliminary recognition result corresponding to the current candidate commodity in the initially collected shelf image to be identified; normalizes the initially collected shelf image to be identified and the re-collected shelf image to be identified to the same coordinate system to obtain a normalized shelf image; determines the distance between the commodity with a preliminary recognition result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified; and obtains a final commodity recognition result of whether the commodity with a preliminary recognition result is the commodity bound to the electronic shelf label according to the distance.

13. A method for commodity identification, characterized in that: The commodity identification method is applied to the second robot, and the method includes: Collect images of shelves to be identified; Segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, and extract features of each commodity with a preliminary identification result from the shelf image to be identified; Match each extracted feature with the product features in the preset product feature library; For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; Generate a search command for searching for products that are consistent with the candidate product information and their bound electronic shelf labels in the second supermarket according to the candidate product list, and send the search command to the second supermarket end; the second supermarket end is used to perform the steps of determining whether there is a product that is consistent with the candidate product information and its bound electronic shelf label in the second supermarket for each candidate product in the candidate product list upon receiving the search command, and if so, send a command indicating a preset display mode to the corresponding bound electronic shelf label; send a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used to display in the preset display mode if receiving the command indicating a preset display mode; When receiving a re-capture command, re-capture the image of the shelf to be identified; Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified; If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified is obtained; the position of the commodity with a preliminary recognition result corresponding to the current candidate commodity in the initially collected shelf image to be identified is determined; the initially collected shelf image to be identified and the re-collected shelf image to be identified are normalized to the same coordinate system to obtain a normalized shelf image; according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified, the distance between the commodity with a preliminary recognition result and the electronic shelf label in the normalized shelf image is determined; and according to the distance, a final commodity recognition result of whether the commodity with a preliminary recognition result is the commodity bound to the electronic shelf label is obtained.

14. A first system for commodity identification, characterized in that: include: An image acquisition device is used to acquire images of shelves to be identified and send them to an image recognition server; When receiving the re-capture command, re-capture the shelf image to be identified, and send the re-captured shelf image to be identified to the image recognition server; The image recognition server is used to segment and identify commodities in the shelf image to be identified, obtain a preliminary identification result for each commodity, extract features of each commodity with a preliminary identification result from the shelf image to be identified, and match each extracted feature with a commodity feature in a preset commodity feature library; For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; Generate a search command for searching for products with the same information as the candidate products and their bound electronic shelf labels in the first supermarket according to the candidate product list, and send the search command to the first supermarket end; Identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified. If an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified, obtain the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified; determine the position of the commodity with a preliminary identification result corresponding to the current candidate commodity in the initially collected shelf image to be identified; normalize the initially collected shelf image to be identified and the re-collected shelf image to be identified to the same coordinate system to obtain a normalized shelf image; determine the distance between the commodity with a preliminary identification result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified; obtain a final commodity identification result of whether the commodity with a preliminary identification result is the commodity bound to the electronic shelf label according to the distance; The first supermarket terminal is used for, upon receiving the search command, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a re-capture command for re-shooting the shelf image to be identified to the image acquisition device through the image recognition server; The electronic shelf label is used to display in the preset display mode if a command indicating the preset display mode is received.

15. An image recognition server for commodity identification, characterized in that: include: The first feature extraction unit is used to segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with the preliminary identification result from the shelf image to be identified; the shelf image to be identified is sent by the image acquisition device; A first matching processing unit, used to match each extracted feature with a commodity feature in a preset commodity feature library; A first candidate product list determination unit, for generating a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; A first generating unit is used to generate a search command for searching for a product consistent with the candidate product information and its bound electronic shelf label in the first supermarket according to the candidate product list, and send the search command to the first supermarket end; The first supermarket terminal is used for, upon receiving the search command, executing the step of determining whether there is a product in the first supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; the electronic shelf label is used for displaying in the preset display mode upon receiving the command indicating a preset display mode; Sending a recapture command to the image acquisition device to retake the image of the shelf to be identified; The image acquisition device is used to re-acquire the image of the shelf to be identified upon receiving the re-acquisition command, and send the re-acquired image of the shelf to be identified to the image recognition server; A first label detection unit, used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified; The first commodity identification unit is used for obtaining the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified if an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified; determining the position of the commodity with a preliminary identification result corresponding to the current candidate commodity in the initially collected shelf image to be identified; normalizing the initially collected shelf image to be identified and the re-collected shelf image to be identified to the same coordinate system to obtain a normalized shelf image; determining the distance between the commodity with the preliminary identification result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified; and obtaining a final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label according to the distance.

16. A second system for commodity identification, characterized in that: include: The second robot is used to collect images of shelves to be identified; Segment and identify commodities on the shelf image to be identified, obtain a preliminary identification result for each commodity, extract features of each commodity with a preliminary identification result from the shelf image to be identified; match each extracted feature with a commodity feature in a preset commodity feature library; For products whose matching degree is greater than the first matching degree threshold and less than the second matching degree threshold, a candidate product list with matching degree values ​​in the top N is generated, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; Generate a search command for searching for products with the same information as the candidate products and their bound electronic shelf labels in the second supermarket according to the candidate product list, and send the search command to the second supermarket end; Upon receiving the re-capture command, the shelf image to be identified is re-captured, and whether there is an electronic shelf label displayed in a preset display mode in the re-captured shelf image to be identified; if an electronic shelf label displayed in a preset display mode is identified in the re-captured shelf image to be identified, the position of the electronic shelf label displayed in the preset display mode in the re-captured shelf image to be identified is obtained; the position of the commodity with a preliminary identification result corresponding to the current candidate commodity in the initially collected shelf image to be identified is determined; the initially collected shelf image to be identified and the re-captured shelf image to be identified are normalized to the same coordinate system to obtain a normalized shelf image; according to the position of the electronic shelf label in the re-captured shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified, the distance between the commodity with a preliminary identification result and the electronic shelf label in the normalized shelf image is determined; according to the distance, a final commodity identification result is obtained as to whether the commodity with a preliminary identification result is the commodity bound to the electronic shelf label; The second supermarket terminal is used for, upon receiving the search command, executing the step of determining whether there is a product in the second supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; and sending a command to retake the image of the shelf to be identified to the second robot; The electronic shelf label is used to display in the preset display mode if a command indicating the preset display mode is received.

17. A second robot for commodity recognition, characterized in that: include: A collection unit, used for collecting images of shelves to be identified; When receiving a re-capture command, re-capture the image of the shelf to be identified; The second feature extraction unit is used to segment and identify commodities on the initially collected shelf image to be identified, obtain a preliminary identification result of each commodity, and extract the features of each commodity with a preliminary identification result from the shelf image to be identified; A second matching processing unit, used to match each extracted feature with a commodity feature in a preset commodity feature library; A second candidate product list determination unit, for generating a candidate product list with top N matching values ​​for products whose matching degrees are greater than a first matching degree threshold and less than a second matching degree threshold, where N is a positive integer greater than 1, and the second matching degree threshold is greater than the first matching degree threshold; A second generating unit is used to generate a search command for searching for a product consistent with the candidate product information and its bound electronic shelf label in the second supermarket according to the candidate product list, and send the search command to the second supermarket end; The second supermarket terminal is used for, upon receiving the search command, executing the steps of determining whether there is a product in the second supermarket that is consistent with the candidate product information and its bound electronic shelf label for each candidate product in the candidate product list, and if so, sending a command indicating a preset display mode to the corresponding bound electronic shelf label; sending a command to re-shoot the shelf image to be identified to the second robot; the electronic shelf label is used for displaying in the preset display mode if receiving the command indicating a preset display mode; A second label detection unit is used to identify whether there is an electronic shelf label displayed in a preset display mode in the re-collected shelf image to be identified; The second commodity identification unit is used for obtaining the position of the electronic shelf label displayed in the preset display mode in the re-collected shelf image to be identified if an electronic shelf label displayed in a preset display mode is identified in the re-collected shelf image to be identified; determining the position of the commodity with a preliminary identification result corresponding to the current candidate commodity in the initially collected shelf image to be identified; normalizing the initially collected shelf image to be identified and the re-collected shelf image to be identified to the same coordinate system to obtain a normalized shelf image; determining the distance between the commodity with the preliminary identification result and the electronic shelf label in the normalized shelf image according to the position of the electronic shelf label in the re-collected shelf image to be identified and the position of the commodity in the initially collected shelf image to be identified; and obtaining a final commodity identification result of whether the commodity with the preliminary identification result is the commodity bound to the electronic shelf label according to the distance.

18. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

20. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.