New Product Data Acquisition Method and Device Based on Shopping Cart

Abstract

The present invention discloses a method and device for collecting new product data based on a shopping cart, which are applied to the field of communication technologies. The method includes: obtaining barcode information of a product; searching for eigenvalue data corresponding to the barcode information in a database; when the number of eigenvalue data corresponding to the searched barcode information is less than a quantity threshold, determining that the product is a new product and sending a shooting and collection signal to a visual sensor; receiving multiple frame video images of the new product sent by the visual sensor and screening out candidate images of the new product; matching the eigenvalue data of the candidate images with the eigenvalue data corresponding to the barcode information in the database, and storing them in the database when the matching is successful; when the matching fails, comparing the eigenvalue data of the candidate images of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information, and storing them in the database when the comparison difference is greater than a difference threshold. The present invention can automatically collect new product images during the use of the shopping cart.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular, to a method and device for collecting new product data based on a shopping cart. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention described in the claims. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] With the rapid development of technologies such as the Internet of Things, artificial intelligence, big data analysis, mobile payment, and smart hardware, the supermarket industry is undergoing a wave of transformation towards intelligence and digitization. Among them, intelligent shopping and settlement are important links. For intelligent shopping and settlement, it is necessary to identify the products selected by users. When identifying products, generally, it is necessary to collect, label, train, etc. the picture data of the products to be identified in advance, and then deploy and apply the trained identification model. When encountering newly launched products, a new round of collection, labeling, training, etc. work is required, with cumbersome workload and low efficiency. Summary of the Invention

[0004] An embodiment of the present invention provides a method for collecting new product data based on a shopping cart, which is applied to the shopping cart. When a new product is encountered during the use of the shopping cart, it can automatically evaluate, collect, and save the image of the product, and can automatically correct and update when the product image is mis-collected, and finally obtain a better set of new product images, improving the efficiency of new product image collection. The method includes:

[0005] Obtain the barcode information of the product, where the barcode information is obtained by the shopper scanning with a barcode scanner;

[0006] Search for the eigenvalue data corresponding to the barcode information in the database;

[0007] When the number of eigenvalue data corresponding to the searched barcode information is less than the number threshold, determine that the product is a new product, and send a shooting and collection signal to the visual sensor;

[0008] Receive multiple frame video images of the new product sent by the visual sensor, and screen out candidate images of the new product from the multiple frame video images;

[0009] Match the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database. When the match is successful, store the candidate image of the new product and the corresponding eigenvalue data in the database;

[0010] When the matching fails, the eigenvalue data of the candidate image of the new product is compared with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding eigenvalue data are stored in the database.

[0011] An embodiment of the present invention provides a new product data acquisition device based on a shopping cart, which is applied to a shopping cart. When encountering a new product during the use of the shopping cart, it can automatically evaluate, collect, and save the image of the product, and can automatically correct and update when the product image is miscollected, and finally obtain a better set of new product images, improving the efficiency of new product image acquisition. The device includes:

[0012] A barcode information acquisition module, configured to acquire the barcode information of the product, where the barcode information is obtained by the shopper scanning with a barcode scanner;

[0013] An eigenvalue data search module, configured to search for the eigenvalue data corresponding to the barcode information from the database;

[0014] A new product shooting module, configured to determine that the product is a new product when the number of eigenvalue data corresponding to the searched barcode information is less than the number threshold, and send a shooting and acquisition signal to the visual sensor;

[0015] An image screening module, configured to receive multiple frames of video images of the new product sent by the visual sensor, and screen out the candidate images of the new product from the multiple frames of video images;

[0016] A first eigenvalue data comparison module, configured to match the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database. When the matching is successful, store the candidate image of the new product and the corresponding eigenvalue data in the database;

[0017] A second eigenvalue data comparison module, configured to, when the matching fails, compare the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0018] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned new product data acquisition method based on a shopping cart is implemented.

[0019] An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned new product data acquisition method based on a shopping cart is implemented.

[0020] An embodiment of the present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned method for collecting new product data based on a shopping cart is implemented.

[0021] In an embodiment of the present invention, barcode information of a product is obtained, and the barcode information is obtained by a shopper scanning with a barcode scanner; eigenvalue data corresponding to the barcode information is searched from a database; when the number of eigenvalue data corresponding to the searched barcode information is less than a quantity threshold, it is determined that the product is a new product, and a shooting and collection signal is sent to a vision sensor; multiple frame video images of the new product sent by the vision sensor are received, and candidate images of the new product are screened from the multiple frame video images; the eigenvalue data of the candidate images of the new product is matched with the eigenvalue data corresponding to the barcode information in the database. When the matching is successful, the candidate images of the new product and the corresponding eigenvalue data are stored in the database; when the matching fails, the eigenvalue data of the candidate images of the new product is compared with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than a difference threshold, the candidate images of the new product and the corresponding eigenvalue data are stored in the database. Compared with the prior art, the present invention can automatically evaluate, collect, and save images of new products when new products are encountered during the use of a shopping cart, and during the above-mentioned collection process, through two key processes of image screening and secondary eigenvalue data comparison, the efficiency of judging new product images can be greatly improved, thereby improving the efficiency of the entire new product image collection. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0023] Figure 1 is a flowchart of the method for collecting new product data based on a shopping cart in an embodiment of the present invention;

[0024] Figure 2 is a flowchart of screening candidate images of a new product from multiple frame video images in an embodiment of the present invention;

[0025] Figure 3 is a flowchart of non-new products in the process of corrective verification in an embodiment of the present invention;

[0026] Figure 4Schematic diagram of the new product data acquisition device based on the shopping cart in the embodiment of the present invention;

[0027] Figure 5 Schematic diagram of the computer device in the embodiment of the present invention. Detailed implementation manners

[0028] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer and more understandable, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0029] The inventors found that the intelligent shopping cart is a representative tool and device for intelligence in the supermarket environment. The intelligent shopping cart has a series of advanced functions, including automatic scanning and settlement, navigation and product positioning, item identification and weighing, personalized recommendation and advertisement display, etc. For example, in terms of self-service scanning and settlement, the intelligent shopping cart allows customers to scan product barcodes instantaneously during the shopping process, view the total shopping price in real time, and complete the payment through the self-service settlement system on the shopping cart. This simplified and brand-new shopping process enables customers to complete the checkout without interacting with the cashier, improving the shopping experience. At the same time, for merchants, it reduces the dependence on cashiers and effectively reduces the operating costs.

[0030] When a user uses the shopping cart, new products are put in or taken out. When a new product is encountered during the use of the shopping cart, the image of the product is automatically evaluated, collected, and saved, and can be automatically corrected and updated when the product image is wrongly collected, and finally a set of good new product images is obtained, improving the efficiency of new product image acquisition. Therefore, the embodiments of the present invention propose a new product data acquisition method and device based on the shopping cart.

[0031] Figure 1 Flowchart of the new product data acquisition method based on the shopping cart in the embodiment of the present invention, applied to the shopping cart, including:

[0032] Step 101: Obtain the barcode information of the product, where the barcode information is obtained by the shopper scanning with a barcode scanner;

[0033] Step 102: Search for the eigenvalue data corresponding to the barcode information in the database;

[0034] Step 103: When the number of the eigenvalue data corresponding to the searched barcode information is less than the number threshold, determine that the product is a new product, and send a shooting and acquisition signal to the visual sensor;

[0035] Step 104: Receive multiple frames of video images of the new product sent by the visual sensor, and screen out the candidate images of the new product from the multiple frames of video images;

[0036] Step 105: Match the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database. When the match is successful, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0037] Step 106: When the match fails, compare the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0038] Compared with the prior art, when a new product is encountered during the use of the shopping cart, the present invention can automatically evaluate, collect, and save the image of the new product. During the above acquisition process, through the two key processes of image screening and secondary eigenvalue data comparison, the efficiency of judging the image of the new product can be greatly improved, and thus the efficiency of the entire acquisition of the new product image can be improved.

[0039] In the embodiment of the present invention, for the products that have been learned or trained by the product recognition algorithm model, the corresponding barcode information (including at least the product ean code) and multiple eigenvalue data (feature vectors extracted from multiple collected product pictures) are stored in at least one database of the shopping cart in a one-to-one correspondence. Therefore, the information of each product can be processed through the "add, delete, modify, and query" operations of the database. In addition to barcode information, images, and eigenvalue data, the database can also store other product information such as product names.

[0040] In step 101, obtain the barcode information of the product, and the barcode information is obtained by the shopper scanning with a barcode scanner.

[0041] Specifically, in the embodiment of the present invention, the shopping cart is an intelligent shopping cart, and the barcode scanner can be integrated on the shopping cart. Before putting the product into the shopping cart, the shopper can scan the barcode of the product with the barcode scanner to obtain the barcode information. In the embodiment of the present invention, the barcode is not limited to barcodes, two-dimensional codes, and other identifiers that can be scanned by various barcode scanners.

[0042] In step 102, search for the eigenvalue data corresponding to the barcode information in the database. Specifically, the barcode information can be set as the key value in the database.

[0043] In step 103, when the number of the eigenvalue data corresponding to the searched barcode information is less than the number threshold, determine that the product is a new product, and send a shooting and acquisition signal to the visual sensor.

[0044] Specifically, the number of the searched eigenvalue data can be 0 or a specific value. However, as long as it is less than the quantity threshold, it can be confirmed as a new product. At this time, a shooting acquisition signal instruction is sent to the vision sensor, and the shooting acquisition signal is to take a picture of the product. The vision sensor obtains continuous n-frame video images at a certain frame rate. The vision sensor can be a camera, and what it shoots is the continuous action of the shopper putting the product into the shopping cart.

[0045] In step 104, receive multiple-frame video images of the new product sent by the vision sensor, and screen out candidate images of the new product from the multiple-frame video images.

[0046] See Figure 2 For the flowchart of screening candidate images of the new product from multiple-frame video images in the embodiment of the present invention. In one embodiment, screening candidate images of the new product from multiple-frame video images includes:

[0047] Step 201, crop multiple-frame product images from the multiple-frame video images. This step specifically includes:

[0048] Input each frame of the video image into the moving product detection model. The moving product detection model is used to detect and crop the products in motion in each frame of the video image to obtain product images. The moving product detection model is an object detection model constructed based on a deep neural network.

[0049] Step 202, perform image quality evaluation on each frame of the product image to obtain the image quality of each frame of the product image.

[0050] Specifically, performing image quality evaluation on each frame of the product image includes: inputting each frame of the product image into the product image quality evaluation model to obtain the image quality of each frame of the product image. The product image quality evaluation model is a classification model constructed based on a deep neural network.

[0051] The product image quality evaluation model can divide the image quality into at least 5 categories, such as 20%, 40%, 60%, 80%, 100%. The larger the value, the higher the image quality of the captured product image, that is, it is clear enough, not blurred, has no hand occlusion area, and has sufficient texture details. When the product image quality evaluation model is trained, after the data set is prepared, a deep learning method is used to train a deep neural network model.

[0052] Step 203, screen out the product image with the highest image quality from the multiple-frame product images.

[0053] Step 204, if the image quality of the product image with the highest image quality is greater than the quality threshold, determine the product image with the highest image quality as the candidate image of the new product.

[0054] In one embodiment, the method further includes:

[0055] If the image quality of the product image with the highest image quality is not greater than the quality threshold, discard the multi-frame product images, continue to send a shooting and acquisition signal to the vision sensor, and after obtaining a new multi-frame video image, screen the candidate images of the new product.

[0056] Since there are two situations during the product collection stage: 1) The shopper scanned A (A is a new product), but deliberately put in B; 2) The moving product detection model makes a mistake when detecting a moving target product. At this time, for the candidate images, if no judgment is made and they are directly entered, incorrect product images will be entered. Therefore, a verification judgment needs to be made. The embodiments of the present invention perform two eigenvalue data comparisons.

[0057] In step 105, match the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database. When the match is successful, store the candidate image of the new product and the corresponding eigenvalue data in the database;

[0058] In one embodiment, matching the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database and, when the match is successful, storing the candidate image of the new product and the corresponding eigenvalue data in the database includes:

[0059] Calculate the distances between the eigenvalue data of the candidate image and the eigenvalue data corresponding to the barcode information in the database respectively, where the distance is the cosine distance or the Euclidean distance;

[0060] When the minimum value of all distances is less than the first distance threshold a, it is determined that the match is successful.

[0061] Specifically, if the number of eigenvalue data corresponding to the barcode information in the database is zero, it means that no image of the new product has been stored yet, and the corresponding barcode information has not been registered in the database either. It is necessary to enter the barcode information of the new product, the candidate image, and the corresponding eigenvalue data in the database. If the number of eigenvalue data corresponding to the barcode information in the database is not zero, it means that the barcode information of the new product has been registered in the database, and only the candidate image of the new product and the corresponding eigenvalue data need to be entered. Each candidate image corresponds to one eigenvalue data.

[0062] In step 106, when the match fails, compare the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0063] In one embodiment, the eigenvalue data of the candidate image of the new product is compared with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding eigenvalue data are stored in the database, including:

[0064] Calculate the distance between the eigenvalue data of the candidate image and each eigenvalue data corresponding to other barcode information except the barcode information. The distance is the cosine distance or the Euclidean distance;

[0065] Determine the minimum value of all distances as the comparison difference;

[0066] When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0067] Specifically, if the comparison difference is greater than the second distance threshold b, it means that the candidate image corresponds to a product with non-scanned barcode information, so the subsequent saving steps are not performed. When the comparison difference is greater than the difference threshold, it is also considered that the candidate image can be saved.

[0068] Regarding the above first distance threshold a, second distance threshold b, and difference threshold c, it should be supplemented that assuming a = 0.2, b = 0.4, c = 0.8, when calculating the eigenvalue distance for the first time, if the minimum eigenvalue distance is in [0, 0.2], or when calculating the eigenvalue distance for the second time, if the minimum eigenvalue distance is between [0.8, 1], then it can be considered that the candidate product image can be saved as a new product image.

[0069] For the candidate image and eigenvalue data of the newly entered product, there may still be a chance of error, and correction is required when the candidate image and eigenvalue data of the new product are used again. The specific method is as follows: For each new product, when the number of eigenvalue data in the database is greater than a certain threshold, it can be determined that the product can be used for normal product identification. However, if the product has been considered "not a new product" for a short time, for example, less than one month, during this one-month period, the product is still in the stage of correction and verification and needs to be verified. The following gives a verification scheme.

[0070] See Figure 3 For the flowchart of the correction and verification of non-new products in the embodiments of the present invention. In one embodiment, the method further includes:

[0071] Step 301, when the number of eigenvalue data corresponding to the searched barcode information is not less than the quantity threshold, determine that the product is a non-new product, and judge whether the non-new product is in the stage of correction and verification;

[0072] Step 302, if so, send a shooting acquisition signal to the visual sensor, receive multiple frames of video images of non-new products sent by the visual sensor, and screen out the candidate images for inspection from the multiple frames of video images;

[0073] Step 303, increment the matching count of the non-new product by 1, and the matching count of the non-new product is initialized to zero when the product is first identified as a non-new product;

[0074] Step 304, search in the database for the eigenvalue data corresponding to the barcode information that matches the eigenvalue data of the candidate image for inspection. After the shopper confirms the eigenvalue data corresponding to the matching barcode information through the interactive screen, increment the frequency of the eigenvalue data corresponding to the matching barcode information by 1, and the frequency of each eigenvalue data is initialized to zero when the product is first identified as a non-new product;

[0075] Step 305, when the matching count of the non-new product exceeds the matching count threshold, determine whether the eigenvalue data under the barcode information of the non-new product belongs to other products according to the frequencies of all the eigenvalue data of the non-new product, and delete the eigenvalue data belonging to other products from under the barcode information of the non-new product;

[0076] Step 306, when the number of eigenvalue data corresponding to the barcode information of the non-new product is less than the quantity threshold, re-identify the non-new product as a new product.

[0077] Through the above steps, after each new product is purchased multiple times, the previously saved eigenvalue data can be corrected. When the matching count of the non-new product exceeds the matching count threshold, that is, the non-new product has been purchased multiple times, there is a record of the usage frequency of the previously saved eigenvalue data, and this record can be used to analyze the validity of the eigenvalue data in the subsequent database.

[0078] In order to improve the efficiency of determining that the product is in the correction verification stage, a label scheme is proposed below.

[0079] In an embodiment, after storing the candidate image and the corresponding eigenvalue data of the new product in the database, it further includes:

[0080] If the number of eigenvalue data corresponding to the barcode information of the new product is not less than the quantity threshold, mark a non-new product label and the matching count on the barcode information, and the non-new product label contains the marking time;

[0081] Initialize the matching count to 0;

[0082] Determining whether the non-new product is in the correction verification stage includes:

[0083] Query the marking time of the non-new product label marked on the barcode information of the non-new product;

[0084] When the time distance from the marked time to the current time is less than a preset duration, it is determined that the non-new product is in the corrective verification stage;

[0085] Re-identifying the non-new product as a new product includes: deleting the non-new product label on the barcode information of the non-new product.

[0086] In one embodiment, searching in the database for the eigenvalue data corresponding to the barcode information that matches the eigenvalue data of the inspection candidate image, and after the shopper confirms the eigenvalue data corresponding to the matched barcode information through the interactive screen, incrementing the frequency of the eigenvalue data corresponding to the matched barcode information by 1, including:

[0087] Calculating the distances between the eigenvalue data of the inspection candidate image and the eigenvalue data corresponding to the barcode information in the database respectively;

[0088] When the minimum value of all distances is less than the first distance threshold, it is confirmed that the match is successful, and the eigenvalue data corresponding to the minimum value of all distances in the database is used as the eigenvalue data corresponding to the matched barcode information;

[0089] Displaying the match success information on the interactive screen of the shopping cart for the shopper to confirm;

[0090] After the shopper confirms that the match success information is correct, incrementing the frequency of the eigenvalue data corresponding to the matched barcode information by 1.

[0091] Specifically, the eigenvalue data corresponding to the matched barcode information is the stored eigenvalue data corresponding to the barcode information of the non-new product in the database. When the match success information is displayed on the interactive screen of the shopping cart, if the shopper confirms it correctly and the credibility of the eigenvalue data corresponding to the matchable barcode information increases, then the frequency of the eigenvalue data corresponding to the matched barcode information is incremented by 1.

[0092] In one embodiment, according to the frequencies of all the eigenvalue data of the non-new product, determining whether the eigenvalue data under the barcode information of the non-new product belongs to other products, and deleting the eigenvalue data belonging to other products from under the barcode information of the non-new product, including:

[0093] Screening out from the database the eigenvalue data of the non-new product with a frequency less than the frequency threshold as the eigenvalue data to be verified;

[0094] For each eigenvalue data to be verified, calculating the distances between the eigenvalue data to be verified and each eigenvalue data corresponding to other products in the database respectively. If the minimum value of all distances is less than the first distance threshold, it is determined that the eigenvalue data to be verified belongs to other products, and the eigenvalue data to be verified is deleted from under the barcode information of the non-new product.

[0095] Specifically, if the frequency of use of some eigenvalue data is less than the frequency threshold, the frequency threshold can be equal to 0 or greater than zero; in this case, it is suspected that these eigenvalue data (for example, there are m pieces, called the eigenvalue data to be verified) are mis-entered data. At this time, for these m pieces of eigenvalue data belonging to non-new products, the distance is calculated one by one with the eigenvalue data entered for other products in the database. If the minimum value of all distances is less than the first distance threshold, it is considered that this piece of eigenvalue data to be verified is mis-recorded and belongs to the eigenvalue data of other products. At this time, this piece of eigenvalue data to be verified is deleted, that is, this piece of eigenvalue data to be verified is deleted from the barcode information of non-new products.

[0096] The embodiment of the present invention also proposes a new product data acquisition device based on a shopping cart, the principle of which is similar to the new product data acquisition method based on a shopping cart, and will not be elaborated here.

[0097] Figure 4 FIG. is a schematic diagram of a new product data acquisition device based on a shopping cart in an embodiment of the present invention, applied to a shopping cart, including:

[0098] A barcode information acquisition module 401, configured to acquire barcode information of a product, where the barcode information is obtained by a shopper scanning with a barcode scanner;

[0099] An eigenvalue data search module 402, configured to search for eigenvalue data corresponding to the barcode information from a database;

[0100] A new product photographing module 403, configured to determine that the product is a new product when the number of eigenvalue data corresponding to the searched barcode information is less than a quantity threshold, and send a photographing acquisition signal to a vision sensor;

[0101] An image screening module 404, configured to receive multiple frames of video images of a new product sent by a vision sensor, and screen out candidate images of the new product from the multiple frames of video images;

[0102] A first eigenvalue data comparison module 405, configured to match the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database, and when the matching is successful, store the candidate image of the new product and the corresponding eigenvalue data in the database;

[0103] A second eigenvalue data comparison module 406, configured to, when the matching fails, compare the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information, and when the comparison difference is greater than a difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0104] In one embodiment, the image screening module is used to:

[0105] Crop multiple frames of commodity images from multiple frames of video images;

[0106] Perform image quality assessment on each frame of commodity image to obtain the image quality of each frame of commodity image;

[0107] Select the commodity image with the highest image quality from multiple frames of commodity images;

[0108] If the image quality of the commodity image with the highest image quality is greater than the quality threshold, determine the commodity image with the highest image quality as the candidate image of the new commodity.

[0109] In one embodiment, the image screening module is used for:

[0110] Input each frame of video image into a moving commodity detection model, which is used to detect and crop the commodities in motion in each frame of video image to obtain commodity images. The moving commodity detection model is an object detection model constructed based on a deep neural network;

[0111] Input each frame of commodity image into a commodity image quality assessment model to obtain the image quality of each frame of commodity image. The commodity image quality assessment model is a classification model constructed based on a deep neural network.

[0112] In one embodiment, the image screening module is used for:

[0113] If the image quality of the commodity image with the highest image quality is not greater than the quality threshold, discard the multiple frames of commodity images, continue to send a shooting and acquisition signal to the vision sensor, and re-obtain multiple frames of video images to screen the candidate images of the new commodity.

[0114] In one embodiment, the first eigenvalue data comparison module is used for:

[0115] Calculate the distances between the eigenvalue data of the candidate image and the eigenvalue data corresponding to the barcode information in the database respectively. The distance is the cosine distance or the Euclidean distance;

[0116] When the minimum value of all distances is less than the first distance threshold, determine that the matching is successful.

[0117] In one embodiment, the second eigenvalue data comparison module is used for:

[0118] Calculate the distances between the eigenvalue data of the candidate image and each eigenvalue data corresponding to other barcode information except the barcode information respectively. The distance is the cosine distance or the Euclidean distance;

[0119] Determine the minimum value of all distances as the comparison difference;

[0120] When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

[0121] In one embodiment, the device further includes a correction module for:

[0122] When the number of eigenvalue data corresponding to the bar code information found is not less than the quantity threshold, determine that the product is not a new product, and judge whether the non-new product is in the correction verification stage;

[0123] If so, send a shooting acquisition signal to the vision sensor, receive multiple frame video images of the non-new product sent by the vision sensor, and screen out the inspection candidate images from the multiple frame video images;

[0124] Increment the matching times of the non-new product by 1. The matching times of the non-new product are initialized to zero when the product is first identified as a non-new product;

[0125] Search in the database for the eigenvalue data corresponding to the bar code information that matches the eigenvalue data of the inspection candidate image. After the shopper confirms the eigenvalue data corresponding to the matching bar code information through the interactive screen, increment the frequency of the eigenvalue data corresponding to the matching bar code information by 1. The frequency of each eigenvalue data is initialized to zero when the product is first identified as a non-new product;

[0126] When the matching times of the non-new product exceed the matching times threshold, judge whether the eigenvalue data under the bar code information of the non-new product belongs to other products according to the frequencies of all the eigenvalue data of the non-new product, and delete the eigenvalue data belonging to other products from under the bar code information of the non-new product;

[0127] When the number of eigenvalue data corresponding to the bar code information of the non-new product is less than the quantity threshold, re-identify the non-new product as a new product.

[0128] In one embodiment, the correction module is used for:

[0129] After storing the candidate image of the new product and the corresponding eigenvalue data in the database, if the number of eigenvalue data corresponding to the bar code information of the new product is not less than the quantity threshold, mark a non-new product label and the matching times on the bar code information. The non-new product label contains the marking time;

[0130] Initialize the initial value of the matching times to 0;

[0131] When judging whether the non-new product is in the correction verification stage, query the marking time of the non-new product label marked on the bar code information of the non-new product;

[0132] When the marking time is less than the preset duration from the current time, determine that the non-new product is in the correction verification stage;

[0133] Delete the non-new product label on the barcode information of non-new products.

[0134] In one embodiment, the correction module is configured to:

[0135] Calculate the distances between the eigenvalue data of the candidate inspection image and the eigenvalue data corresponding to the barcode information in the database respectively;

[0136] When the minimum value of all the distances is less than the first distance threshold, confirm that the matching is successful, and use the eigenvalue data corresponding to the minimum value of all the distances in the database as the eigenvalue data corresponding to the matched barcode information;

[0137] Display the matching success information on the interactive screen of the shopping cart for the shopper to confirm;

[0138] After the shopper confirms that the matching success information is correct, increment the frequency of the eigenvalue data corresponding to the matched barcode information by 1.

[0139] In one embodiment, the correction module is configured to:

[0140] Screen out the eigenvalue data of non-new products with a frequency less than the frequency threshold from the database as the eigenvalue data to be verified;

[0141] For each eigenvalue data to be verified, calculate the distances between the eigenvalue data to be verified and each eigenvalue data corresponding to other products in the database respectively. If the minimum value of all the distances is less than the first distance threshold, determine that the eigenvalue data to be verified belongs to other products, and delete the eigenvalue data to be verified from the barcode information of non-new products.

[0142] In summary, the method and device proposed in the embodiments of the present invention have the following beneficial effects:

[0143] It can automatically determine whether a product is a new product and whether it needs to be collected, can evaluate the image quality standard for collecting new products, and only enter product images with higher quality; it can check the similarity (eigenvalue distance) of candidate images of new products to avoid incorrect collection; in the time period shortly after the new product is collected, a verification process through shopping interaction is required, and the previously incorrectly saved data is eliminated according to certain rules. Through the above process, when a new product is encountered during the use of the shopping cart, the images of new products are automatically evaluated, collected, saved, and corrected. It can automatically correct and update when the product images are incorrectly collected, and finally obtain a better set of new product images. When it is subsequently widely spread to more markets, the shopping cart product can be applied more quickly and ensure more accurate product recognition accuracy.

[0144] The embodiments of the present invention also provide a computer device,Figure 5 It is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored on the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, the above-mentioned new product data collection method based on the shopping cart is implemented.

[0145] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned new product data collection method based on the shopping cart is implemented.

[0146] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned new product data collection method based on the shopping cart is implemented.

[0147] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0149] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps for realizing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 in one block or a plurality of blocks.

[0151] The specific embodiments described above have further elaborated on the object, technical solution and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for collecting new product data based on a shopping cart, characterized in that, Applied to a shopping cart, including: Obtain the barcode information of the commodity, which is obtained by the shopper scanning with a barcode scanner; Search for the eigenvalue data corresponding to the barcode information in the database; When the number of eigenvalue data corresponding to the searched barcode information is less than the quantity threshold, determine that the commodity is a new commodity and send a shooting and acquisition signal to the visual sensor; Receive multiple frames of video images of the new commodity sent by the visual sensor, and screen out candidate images of the new commodity from the multiple frames of video images; Match the eigenvalue data of the candidate image of the new commodity with the eigenvalue data corresponding to the barcode information in the database. When the matching is successful, store the candidate image of the new commodity and the corresponding eigenvalue data in the database; When the matching fails, compare the eigenvalue data of the candidate image of the new commodity with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, store the candidate image of the new commodity and the corresponding eigenvalue data in the database.

2. The method according to claim 1, characterized in that, Screen out candidate images of the new commodity from multiple frames of video images, including: Crop multiple frames of commodity images from multiple frames of video images; Perform image quality evaluation on each frame of commodity image to obtain the image quality of each frame of commodity image; Screen out the commodity image with the highest image quality from multiple frames of commodity images; If the image quality of the commodity image with the highest image quality is greater than the quality threshold, determine that the commodity image with the highest image quality is the candidate image of the new commodity.

3. The method according to claim 2, wherein Crop multiple frames of commodity images from multiple frames of video images, including: Input each frame of video image into a moving commodity detection model, which is used to detect and crop the commodities in motion in each frame of video image to obtain commodity images. The moving commodity detection model is an object detection model constructed based on a deep neural network; Perform image quality evaluation on each frame of commodity image, including: Input each frame of commodity image into a commodity image quality evaluation model to obtain the image quality of each frame of commodity image. The commodity image quality evaluation model is a classification model constructed based on a deep neural network.

4. The method according to claim 2, wherein It also includes: If the image quality of the commodity image with the highest image quality is not greater than the quality threshold, discard the multiple frames of commodity images, continue to send a shooting and acquisition signal to the visual sensor, and re-obtain multiple frames of video images to screen out candidate images of the new commodity.

5. The method according to claim 1, wherein Match the eigenvalue data of the candidate image of the new commodity with the eigenvalue data corresponding to the barcode information in the database, including: Calculate the distances between the eigenvalue data of the candidate image and the eigenvalue data corresponding to the barcode information in the database respectively. The distance is the cosine distance or the Euclidean distance; When the minimum value of all distances is less than the first distance threshold, determine that the matching is successful.

6. The method according to claim 1, wherein Compare the eigenvalue data of the candidate image of the new commodity with the eigenvalue data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, store the candidate image of the new commodity and the corresponding eigenvalue data in the database, including: Calculate the distances between the eigenvalue data of the candidate image and each eigenvalue data corresponding to other barcode information except the said barcode information respectively, where the distances are cosine distances or Euclidean distances; Determine the minimum value of all the distances as the comparison difference; When the comparison difference is greater than the difference threshold, store the candidate image of the new product and the corresponding eigenvalue data in the database.

7. The method according to claim 1, wherein It also includes: When the number of eigenvalue data corresponding to the searched barcode information is not less than the quantity threshold, determine that the product is a non-new product, and judge whether the non-new product is in the correction verification stage; If so, send a shooting and acquisition signal to the vision sensor, receive multiple frame video images of the non-new product sent by the vision sensor, and screen out the inspection candidate images from the multiple frame video images; Increment the matching times of the non-new product by 1, and the matching times of the non-new product are initialized to zero when the product is first identified as a non-new product; Search in the database for the eigenvalue data corresponding to the barcode information that matches the eigenvalue data of the inspection candidate image. After the shopper confirms the eigenvalue data corresponding to the matched barcode information through the interactive screen, increment the frequency of the eigenvalue data corresponding to the matched barcode information by 1, and the frequency of each eigenvalue data is initialized to zero when the product is first identified as a non-new product; When the matching times of the non-new product exceed the matching times threshold, judge whether the eigenvalue data under the barcode information of the non-new product belongs to other products according to the frequencies of all the eigenvalue data of the non-new product, and delete the eigenvalue data belonging to other products from under the barcode information of the non-new product; When the number of eigenvalue data corresponding to the barcode information of the non-new product is less than the quantity threshold, re-identify the non-new product as a new product; After storing the candidate image of the new product and the corresponding eigenvalue data in the database, it also includes: If the number of eigenvalue data corresponding to the barcode information of the new product is not less than the quantity threshold, mark a non-new product label and the matching times on the said barcode information, and the non-new product label contains the marking time; Initialize the matching times to 0; Judging whether the non-new product is in the correction verification stage includes: Query the marking time of the non-new product label marked on the barcode information of the non-new product; When the marking time is less than the preset duration from the current time, determine that the non-new product is in the correction verification stage; Re-identifying the non-new product as a new product includes: deleting the non-new product label on the barcode information of the non-new product.

8. The method according to claim 7, wherein Search in the database for the eigenvalue data corresponding to the barcode information that matches the eigenvalue data of the inspection candidate image. After the shopper confirms the eigenvalue data corresponding to the matched barcode information through the interactive screen, increment the frequency of the eigenvalue data corresponding to the matched barcode information by 1, including: Calculate the distances between the eigenvalue data of the inspection candidate image and the eigenvalue data corresponding to the barcode information in the database respectively; When the minimum value of all the distances is less than the first distance threshold, confirm that the matching is successful, and use the eigenvalue data corresponding to the minimum value of all the distances in the database as the eigenvalue data corresponding to the matched barcode information; Display the matching success information on the interactive screen of the shopping cart for the shopper to confirm; After the shopper confirms that the matching success information is correct, increment the frequency of the eigenvalue data corresponding to the matched barcode information by 1.

9. The method according to claim 7, wherein Based on the frequencies of all eigenvalue data of non-new products, determine whether the eigenvalue data under the barcode information of non-new products belongs to other products, and delete the eigenvalue data belonging to other products from under the barcode information of non-new products, including: Filter out the eigenvalue data of non-new products with frequencies less than the frequency threshold from the database as the eigenvalue data to be verified; For each eigenvalue data to be verified, calculate the distance between the eigenvalue data to be verified and each eigenvalue data corresponding to other products in the database respectively. If the minimum value of all distances is less than the first distance threshold, determine that the eigenvalue data to be verified belongs to other products, and delete the eigenvalue data to be verified from under the barcode information of non-new products.

10. A new product data acquisition device based on a shopping cart, characterized in that, Applied to the shopping cart, including: A barcode information acquisition module for acquiring the barcode information of a product, where the barcode information is obtained by the shopper scanning with a barcode scanner; An eigenvalue data search module for searching the eigenvalue data corresponding to the barcode information from the database; A new product shooting module for determining that the product is a new product when the number of eigenvalue data corresponding to the searched barcode information is less than the quantity threshold, and sending a shooting and acquisition signal to the visual sensor; An image screening module for receiving multiple frame video images of the new product sent by the visual sensor and screening out candidate images of the new product from the multiple frame video images; A first eigenvalue data comparison module for matching the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to the barcode information in the database, and storing the candidate image of the new product and the corresponding eigenvalue data in the database when the matching is successful; A second eigenvalue data comparison module for, when the matching fails, comparing the eigenvalue data of the candidate image of the new product with the eigenvalue data corresponding to other barcode information in the database except the barcode information, and storing the candidate image of the new product and the corresponding eigenvalue data in the database when the comparison difference is greater than the difference threshold.

11. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 9.

Images