New commodity data acquisition method and device based on shopping cart

Abstract

The invention discloses a new commodity data acquisition method and device based on a shopping cart, which are applied to the technical field of communication, and the method comprises the steps: obtaining the bar code information of a commodity; searching characteristic value data corresponding to the bar code information from a database; when the quantity of the characteristic value data corresponding to the searched bar code information is smaller than a quantity threshold value, determining that the commodity is a new commodity, and sending a shooting acquisition signal to a visual sensor; receiving a multi-frame video image of a new commodity sent by a visual sensor, and screening out candidate images of the new commodity; matching the feature value data of the candidate image with feature value data corresponding to the bar code information in a database, and storing the feature value data in the database when matching succeeds; and when the matching fails, comparing the feature value data of the candidate image of the new commodity with feature value data corresponding to other bar code information except the bar code information in the database, and when the comparison difference is greater than a difference threshold value, storing the feature value data into the database. According to the invention, new commodity images can be automatically collected in the use process of the shopping cart.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method and device for collecting new commodity data based on a shopping cart. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] With the rapid development of technologies such as the Internet of Things, artificial intelligence, big data analysis, mobile payments, and smart hardware, the supermarket industry is experiencing a wave of transformation towards intelligence and digitalization. Among them, smart shopping and settlement is an important link. Smart shopping and settlement require the identification of the goods selected by the user. When identifying goods, it is generally necessary to collect, annotate, and train the image data of the goods to be identified in advance, and then deploy and apply the trained recognition model. When encountering newly listed goods, a new round of collection, annotation, training, etc. is required, which is cumbersome and inefficient. Summary of the invention

[0004] The embodiment of the present invention provides a method for collecting new product data based on a shopping cart. The method is applied to the shopping cart. When a new product is encountered during the use of the shopping cart, the method can automatically evaluate, collect, and save the image of the product. When the product image is collected incorrectly, the method can automatically correct and update the image, and finally obtain a better set of new product images, thereby improving the efficiency of collecting new product images. The method includes: Obtaining barcode information of the product, where the barcode information is obtained by a shopper scanning with a barcode scanner; Searching for characteristic value data corresponding to the barcode information from a database; When the number of the characteristic value data corresponding to the searched barcode information is less than the number threshold, the commodity is determined to be a new commodity, and a shooting acquisition signal is sent to the visual sensor; receiving multiple frames of video images of new products sent by a visual sensor, and screening out candidate images of the new products from the multiple frames of video images; Matching the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, storing the candidate image of the new product and the corresponding feature value data in the database; When the match fails, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0005] The embodiment of the present invention provides a new product data collection device based on a shopping cart, which is applied to the shopping cart. When a new product is encountered during the use of the shopping cart, the device can automatically evaluate, collect, and save the image of the product, and can automatically correct and update the product image when it is collected incorrectly, so as to finally obtain a better set of new product images and improve the efficiency of new product image collection. The device includes: A barcode information acquisition module is used to obtain the barcode information of the product, where the barcode information is obtained by the shopper through scanning with a barcode scanner; A characteristic value data search module, used to search the characteristic value data corresponding to the barcode information from a database; A new product shooting module, used for determining that the product is a new product when the number of feature value data corresponding to the searched barcode information is less than a quantity threshold, and sending a shooting acquisition signal to the visual sensor; An image screening module is used to receive multiple frames of video images of new products sent by a visual sensor, and screen candidate images of new products from the multiple frames of video images; A first feature value data comparison module is used to match the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, the candidate image of the new product and the corresponding feature value data are stored in the database; The second feature value data comparison module is used to compare the feature value data of the candidate image of the new product with the feature value data corresponding to other barcode information in the database except the barcode information when the matching fails, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0006] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned shopping cart-based new product data collection method when executing the computer program.

[0007] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned new product data collection method based on a shopping cart is implemented.

[0008] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned new product data collection method based on a shopping cart is implemented.

[0009] In an embodiment of the present invention, barcode information of a product is obtained, and the barcode information is obtained by a shopper by scanning with a barcode scanner; feature value data corresponding to the barcode information is searched from a database; when the number of feature value data corresponding to the searched barcode information is less than a quantity threshold, the product is determined to be a new product, and a shooting acquisition signal is sent to a visual sensor; multiple frames of video images of the new product sent by the visual sensor are received, and candidate images of the new product are screened out from the multiple frames of video images; the feature value data of the candidate image of the new product is matched with the feature value data corresponding to the barcode information in the database, and when the match is successful, the candidate image of the new product and the corresponding feature value data are stored in the database; when the match fails, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database. Compared with the prior art, the present invention can automatically evaluate, collect and save images of new products when encountering new products during the use of the shopping cart. In addition, in the above-mentioned collection process, the efficiency of new product image judgment can be greatly improved through the two key processes of image screening and secondary eigenvalue data comparison, thereby improving the efficiency of the entire new product image collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 Flow chart of a method for collecting new product data based on a shopping cart in an embodiment of the present invention; Figure 2 A flowchart of selecting candidate images of new products from multiple frames of video images in an embodiment of the present invention; Figure 3 This is a flow chart of a non-new product undergoing correction and verification in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a new commodity data collection device based on a shopping cart in an embodiment of the present invention; Figure 5 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0012] The inventors found that the smart shopping cart is a representative tool for intelligence in the supermarket environment. The smart shopping cart has a series of advanced functions, including automatic scanning and settlement, navigation and product positioning, item identification and weight measurement, personalized recommendations and advertising display, etc. For example, in terms of self-service code scanning and settlement, the smart shopping cart allows customers to scan product barcodes instantly during the shopping process, view the total purchase price in real time, and complete the payment through the self-service settlement system on the shopping cart. This simplified, new shopping process allows customers to complete checkout without interacting with the cashier, which improves the shopping experience. At the same time, for merchants, it reduces dependence on cashiers and effectively reduces operating costs.

[0013] When a user uses a shopping cart, he or she puts in or takes out new products. When a new product is encountered during the use of the shopping cart, the image of the product is automatically evaluated, collected, and saved. When the product image is collected incorrectly, it can be automatically corrected and updated, and finally a better set of new product images is obtained, thereby improving the efficiency of new product image collection. Therefore, an embodiment of the present invention proposes a new product data collection method and device based on a shopping cart.

[0014] Figure 1 This is a flow chart of a method for collecting new commodity data based on a shopping cart in an embodiment of the present invention, which is applied to a shopping cart and includes: Step 101, obtaining the barcode information of the product, wherein the barcode information is obtained by a shopper by scanning with a barcode scanner; Step 102, searching the database for characteristic value data corresponding to the barcode information; Step 103, when the number of the characteristic value data corresponding to the searched barcode information is less than the number threshold, it is determined that the commodity is a new commodity, and a shooting acquisition signal is sent to the visual sensor; Step 104, receiving multiple frames of video images of the new product sent by the visual sensor, and screening out candidate images of the new product from the multiple frames of video images; Step 105, matching the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, storing the candidate image of the new product and the corresponding feature value data in the database; Step 106, when the matching fails, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0015] Compared with the prior art, the present invention can automatically evaluate, collect and save images of new products when encountering new products during the use of the shopping cart. In addition, in the above-mentioned collection process, the efficiency of new product image judgment can be greatly improved through the two key processes of image screening and secondary eigenvalue data comparison, thereby improving the efficiency of the entire new product image collection.

[0016] In the embodiment of the present invention, for commodities that have been learned or trained by the commodity recognition algorithm model, the corresponding barcode information (including at least the commodity EAN code) and multiple feature value data (feature vectors extracted from multiple collected commodity images) are stored in a one-to-one correspondence in at least one database of the shopping cart. Therefore, the information of each commodity can be processed through the "add, delete, modify and query" operation of the database. In addition to the barcode information, image and feature value data, the database can also store other commodity information such as commodity name.

[0017] In step 101, the barcode information of the product is obtained, and the barcode information is obtained by the shopper through scanning with a barcode scanner; Specifically, in the embodiment of the present invention, the shopping cart is a smart shopping cart, and the barcode scanner can be integrated in the shopping cart. Before putting the goods into the shopping cart, the shopper can scan the barcode of the goods through the barcode scanner to obtain the barcode information. In the embodiment of the present invention, the barcode is not limited to barcodes, QR codes, etc., which can be scanned by various barcode scanners.

[0018] In step 102, the characteristic value data corresponding to the barcode information is searched from a database. Specifically, the barcode information can be set as a key value in the database.

[0019] In step 103, when the number of the characteristic value data corresponding to the searched barcode information is less than the number threshold, the commodity is determined to be a new commodity, and a shooting acquisition signal is sent to the visual sensor; Specifically, the number of feature value data searched can be 0 or a specific value, but as long as it is less than the quantity threshold, it can be confirmed as a new product. At this time, a shooting and acquisition signal instruction is sent to the visual sensor. The shooting and acquisition signal is to take a photo of the product, wherein the visual sensor shoots at a certain frame rate to obtain continuous n frames of video images. The visual sensor can be a camera, which captures the continuous action of the shopper putting the product into the shopping cart.

[0020] In step 104, multiple frames of video images of the new product sent by the visual sensor are received, and candidate images of the new product are screened out from the multiple frames of video images; See also Figure 2 The flowchart of selecting candidate images of new commodities from multiple frames of video images in an embodiment of the present invention is as follows. In one embodiment, selecting candidate images of new commodities from multiple frames of video images includes: Step 201, cutting out multiple frames of product images from multiple frames of video images, specifically includes: Input each frame of video image into a moving commodity detection model, wherein the moving commodity detection model is used to detect and crop moving commodities in each frame of video image to obtain commodity images, wherein the moving commodity detection model is a target detection model built based on a deep neural network; Step 202, performing image quality evaluation on each frame of commodity image to obtain the image quality of each frame of commodity image; Specifically, performing image quality assessment on each frame of commodity image includes: inputting each frame of commodity image into a commodity image quality assessment model to obtain the image quality of each frame of commodity image, wherein the commodity image quality assessment model is a classification model constructed based on a deep neural network.

[0021] The product image quality assessment model can classify image quality into at least five categories, such as 20%, 40%, 60%, 80%, and 100%. The larger the value, the higher the quality of the captured product image, that is, it is clear enough, not blurred, has no hand occlusion area, and has sufficient texture details. When training the product image quality assessment model, after the data set is prepared, a deep learning method is used to train a deep neural network model.

[0022] Step 203, selecting the product image with the highest image quality from the multiple frames of product images; Step 204: If the image quality of the product image with the highest image quality is greater than the quality threshold, the product image with the highest image quality is determined as a candidate image of a new product.

[0023] In one embodiment, the method further comprises: If the image quality of the product image with the highest image quality is not greater than the quality threshold, the multiple frames of product images are abandoned, and the shooting acquisition signal continues to be sent to the visual sensor. After re-obtaining multiple frames of video images, candidate images of new products are screened.

[0024] In the commodity collection stage, there are two situations: 1) the shopper scans A (A is a new commodity), but deliberately puts in B; 2) the moving commodity detection model makes an error when detecting the moving target commodity. At this time, for the candidate image, if it is directly entered without making a judgment, the wrong commodity image will be entered, so a check judgment must be made. The embodiment of the present invention performs two feature value data comparisons.

[0025] In step 105, the feature value data of the candidate image of the new product is matched with the feature value data corresponding to the barcode information in the database. When the match is successful, the candidate image of the new product and the corresponding feature value data are stored in the database; In one embodiment, matching the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and storing the candidate image of the new product and the corresponding feature value data in the database when the match is successful, includes: Respectively calculating the distance between the feature value data of the candidate image and the feature value data corresponding to the barcode information in the database, wherein the distance is a cosine distance or a Euclidean distance; When the minimum value of all distances is less than the first distance threshold a, it is determined that the match is successful.

[0026] Specifically, if the number of feature value data corresponding to the barcode information in the database is zero, it means that the image of the new product has not been stored, and the corresponding barcode information has not been registered in the database. It is necessary to enter the barcode information and candidate images of the new product and the corresponding feature value data in the database. If the number of feature value data corresponding to the barcode information in the database is not zero, it means that the barcode information of the new product has been registered in the database, and it is only necessary to enter the candidate images of the new product and the corresponding feature value data. Each candidate image corresponds to one feature value data.

[0027] In step 106, when the matching fails, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0028] In one embodiment, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information other than the barcode information in the database, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database, including: Respectively calculating the distance between the feature value data of the candidate image and each feature value data corresponding to other barcode information except the barcode information, wherein the distance is a cosine distance or a Euclidean distance; Determine the minimum value of all distances as the alignment difference c; When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0029] Specifically, if the comparison difference is greater than the second distance threshold b, it means that the candidate image is not a product corresponding to the scanned barcode information, so the subsequent saving step is not performed. When the comparison difference is greater than the difference threshold, it is also considered that the candidate image can be saved.

[0030] With respect to the above-mentioned first distance threshold a, second distance threshold b and comparison difference c, it should be additionally explained that, assuming a=0.2, b=0.4, c=0.8, when the first eigenvalue distance is calculated, the minimum eigenvalue distance is between [0,0.2], or when the second eigenvalue distance is calculated, the minimum eigenvalue distance is between [0.8,1], then it can be considered that the candidate product image can be saved as a new product image.

[0031] There may still be a chance of error in the candidate images and feature value data of new products that are entered, and they need to be corrected when they are used again. The specific approach is: for each new product, when the number of feature value data in the database is greater than a certain threshold, it can be determined that the product can be used for normal product identification. However, if the product has not been considered a "new product" for a long time, for example, less than one month, during this one month, the product is still in the correction and verification stage and needs to be verified. A verification plan is given below.

[0032] See also Figure 3 This is a flow chart of a non-new product undergoing correction check in an embodiment of the present invention. In one embodiment, the method further includes: Step 301, when the number of the characteristic value data corresponding to the searched barcode information is not less than the number threshold, determining that the commodity is a non-new commodity, and determining whether the non-new commodity is in the correction and verification stage; Step 302: If yes, send a capture signal to the visual sensor, receive multiple frames of video images of non-new products sent by the visual sensor, and select candidate images for inspection from the multiple frames of video images; Step 303, adding 1 to the matching number of the non-new product, wherein the matching number of the non-new product is initialized to zero when the product is first identified as a non-new product; Step 304, searching the database for feature value data corresponding to the barcode information that matches the feature value data of the picket candidate image, and after the shopper confirms the feature value data corresponding to the matched barcode information through the interactive screen, the frequency of the feature value data corresponding to the matched barcode information is increased by 1, and the frequency of each feature value data is initialized to zero when the product is first identified as a non-new product; Step 305: when the number of matches of the non-new product exceeds the matching number threshold, determine whether the feature value data under the barcode information of the non-new product belongs to other products based on the frequency of all feature value data of the non-new product, and delete the feature value data belonging to other products from the barcode information of the non-new product; Step 306: When the quantity of the characteristic value data corresponding to the barcode information of the non-new commodity is less than the quantity threshold, the non-new commodity is re-identified as a new commodity.

[0033] Through the above steps, after each new product has been purchased multiple times, the previously saved feature value data can be corrected. If the number of matches of a non-new product exceeds the matching number threshold, that is, the non-new product has been purchased multiple times, the previously saved feature value data has a record of the frequency of use, which can be used to analyze the validity of the feature value data in the subsequent database.

[0034] In order to improve the efficiency of determining that goods are in the correction verification stage, a labeling scheme is proposed below.

[0035] In one embodiment, after storing the candidate images of the new product and the corresponding feature value data in the database, the method further includes: If the number of feature value data corresponding to the barcode information of the new product is not less than the quantity threshold, a non-new product label and the number of matches are marked on the barcode information, and the non-new product label contains the marking time; Initialize the number of matches to 0; Determine whether a non-new product is in the correction and verification stage, including: Check the marking time of the non-new product label marked on the barcode information of the non-new product; When the distance between the marked time and the current time is less than a preset time, it is determined that the non-new product is in the correction and verification stage; Re-identify the non-new product as a new product, including: deleting the non-new product label on the barcode information of the non-new product.

[0036] In one embodiment, searching a database for feature value data corresponding to the barcode information that matches the feature value data of the picket candidate image, and after the shopper confirms the feature value data corresponding to the matched barcode information through an interactive screen, adding 1 to the frequency of the feature value data corresponding to the matched barcode information comprises: Respectively calculating the distance between the feature value data of the candidate picket image and the feature value data corresponding to the barcode information in the database; When the minimum value of all distances is less than the first distance threshold, the match is confirmed to be successful, and the characteristic value data corresponding to the minimum value of all distances in the database is used as the characteristic value data corresponding to the matched barcode information; Display the successful matching information on the interactive screen of the shopping cart for the shopper to confirm; After the shopper confirms that the matching success information is correct, the frequency of the feature value data corresponding to the matched barcode information is increased by 1.

[0037] Specifically, the characteristic value data corresponding to the matched barcode information is the already stored characteristic value data corresponding to the barcode information of the non-new product in the database. When the successful matching information is displayed on the interactive screen of the shopping cart, if the shopper confirms that it is correct, the credibility of the characteristic value data corresponding to the matched barcode information increases, and the frequency of the characteristic value data corresponding to the marked matched barcode information is increased.

[0038] In one embodiment, judging whether the characteristic value data under the barcode information of the non-new product belongs to other products according to the frequency of all characteristic value data of the non-new product, and deleting the characteristic value data belonging to other products from the barcode information of the non-new product includes: Filter out the feature value data of non-new products whose frequency is less than the frequency threshold from the database as the feature value data to be verified; For each characteristic value data to be verified, the distance between the characteristic value data to be verified and each characteristic value data corresponding to other commodities in the database is calculated respectively. If the minimum value of all distances is less than the first distance threshold, it is determined that the characteristic value data to be verified belongs to other commodities, and the data value to be verified is deleted from the barcode information of non-new commodities.

[0039] Specifically, if the frequency of use of some feature value data is less than the frequency threshold, the frequency threshold may be equal to 0 or greater than zero; at this time, it is suspected that these feature value data (for example, there are m pieces, called feature value data to be verified) are incorrectly entered. At this time, the distance between the m feature value data belonging to the non-new product and the feature value data entered for other products in the database is calculated one by one. If the minimum value of all distances is less than the first distance threshold, it is considered that the feature value data to be verified is incorrectly recorded and belongs to other products. At this time, the feature value data to be verified is deleted, that is, the feature value data to be verified is deleted from the barcode information of the non-new product.

[0040] The embodiment of the present invention further proposes a new product data collection device based on a shopping cart, the principle of which is similar to the new product data collection method based on a shopping cart, and will not be described in detail here.

[0041] Figure 4 This is a schematic diagram of a new commodity data collection device based on a shopping cart in an embodiment of the present invention, which is applied to a shopping cart and includes: The barcode information obtaining module 401 is used to obtain the barcode information of the commodity, and the barcode information is obtained by the shopper through scanning with a barcode scanner; A characteristic value data search module 402 is used to search the characteristic value data corresponding to the barcode information from a database; A new product shooting module 403 is used to determine that the product is a new product when the number of feature value data corresponding to the searched barcode information is less than a quantity threshold, and send a shooting acquisition signal to the visual sensor; The image screening module 404 is used to receive multiple frames of video images of new products sent by the visual sensor, and screen out candidate images of new products from the multiple frames of video images; The first feature value data comparison module 405 is used to match the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, the candidate image of the new product and the corresponding feature value data are stored in the database; The second feature value data comparison module 406 is used to compare the feature value data of the candidate image of the new product with the feature value data corresponding to other barcode information in the database except the barcode information when the matching fails, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0042] In one embodiment, the image screening module is used to: Crop multiple frames of product images from multiple frames of video images; Perform image quality evaluation on each frame of product image to obtain the image quality of each frame of product image; Filter out the product image with the highest image quality from multiple frames of product images; If the image quality of the product image with the highest image quality is greater than the quality threshold, the product image with the highest image quality is determined as a candidate image of the new product.

[0043] In one embodiment, the image screening module is used to: Input each frame of video image into a moving commodity detection model, wherein the moving commodity detection model is used to detect and crop moving commodities in each frame of video image to obtain commodity images, wherein the moving commodity detection model is a target detection model built based on a deep neural network; Each frame of commodity image is input into a commodity image quality assessment model to obtain the image quality of each frame of commodity image. The commodity image quality assessment model is a classification model built based on a deep neural network.

[0044] In one embodiment, the image screening module is used to: If the image quality of the product image with the highest image quality is not greater than the quality threshold, the multiple frames of product images are abandoned, and the shooting acquisition signal continues to be sent to the visual sensor. After re-obtaining multiple frames of video images, candidate images of new products are screened.

[0045] In one embodiment, the first feature value data comparison module is used to: Respectively calculating the distance between the feature value data of the candidate image and the feature value data corresponding to the barcode information in the database, wherein the distance is a cosine distance or a Euclidean distance; When the minimum value of all distances is less than the first distance threshold, it is determined that the match is successful.

[0046] In one embodiment, the second characteristic value data comparison module is used to: Respectively calculating the distance between the feature value data of the candidate image and each feature value data corresponding to other barcode information except the barcode information, wherein the distance is a cosine distance or a Euclidean distance; Determine the minimum of all distances as the alignment difference; When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

[0047] In one embodiment, the apparatus further comprises a correction module, configured to: When the number of the characteristic value data corresponding to the searched barcode information is not less than the quantity threshold, determining that the commodity is a non-new commodity, and judging whether the non-new commodity is in the correction and verification stage; If yes, send a shooting acquisition signal to the visual sensor, receive multiple frames of video images of non-new products sent by the visual sensor, and select picket candidate images from the multiple frames of video images; The number of matches for a non-new product is increased by 1, where the number of matches for the non-new product is initialized to zero when the product is first identified as a non-new product; Searching for feature value data corresponding to the barcode information that matches the feature value data of the picket candidate image from a database, and after the shopper confirms the feature value data corresponding to the matched barcode information through an interactive screen, adding 1 to the frequency of the feature value data corresponding to the matched barcode information, and the frequency of each feature value data is initialized to zero when a product is first identified as a non-new product; When the number of matches of a non-new product exceeds the matching number threshold, judging whether the feature value data under the barcode information of the non-new product belongs to other products based on the frequency of all feature value data of the non-new product, and deleting the feature value data belonging to other products from the barcode information of the non-new product; When the quantity of the characteristic value data corresponding to the barcode information of the non-new product is less than the quantity threshold, the non-new product is re-identified as a new product.

[0048] In one embodiment, the correction module is used to: After the candidate image of the new product and the corresponding feature value data are stored in the database, if the number of feature value data corresponding to the barcode information of the new product is not less than the quantity threshold, a non-new product label and the number of matches are marked on the barcode information, and the non-new product label contains the marking time; Initialize the number of matches to 0; When determining whether a non-new product is in the correction and verification stage, query the marking time of the non-new product label marked on the barcode information of the non-new product; When the distance between the marked time and the current time is less than a preset time, it is determined that the non-new product is in the correction and verification stage; Delete the non-new product label on the barcode information of the non-new product.

[0049] In one embodiment, the correction module is used to: Respectively calculating the distance between the feature value data of the candidate picket image and the feature value data corresponding to the barcode information in the database; When the minimum value of all distances is less than the first distance threshold, the match is confirmed to be successful, and the characteristic value data corresponding to the minimum value of all distances in the database is used as the characteristic value data corresponding to the matched barcode information; Display the successful matching information on the interactive screen of the shopping cart for the shopper to confirm; After the shopper confirms that the matching success information is correct, the frequency of the feature value data corresponding to the matched barcode information is increased by 1.

[0050] In one embodiment, the correction module is used to: Filter out the feature value data of non-new products whose frequency is less than the frequency threshold from the database as the feature value data to be verified; For each characteristic value data to be verified, the distance between the characteristic value data to be verified and each characteristic value data corresponding to other commodities in the database is calculated respectively. If the minimum value of all distances is less than the first distance threshold, it is determined that the characteristic value data to be verified belongs to other commodities, and the data value to be verified is deleted from the barcode information of non-new commodities.

[0051] In summary, the method and device proposed in the embodiments of the present invention have the following beneficial effects: It can automatically determine whether a product is new and whether it needs to be collected. It can evaluate the image quality standards of new product collection and only record product images with higher quality. It can check the similarity (feature value distance) of candidate images of new products to avoid incorrect collection. Shortly after the collection of new products, it needs to go through the shopping interaction verification process, and the previously incorrectly saved data will be eliminated according to certain rules. Through the above process, when new products are encountered during the use of the shopping cart, the images of new products are automatically evaluated, collected, saved, and corrected. When product images are collected incorrectly, they can be automatically corrected and updated, and finally a better collection of new product images can be obtained. When it is rolled out on a large scale in more markets in the future, shopping cart products can be applied more quickly and more accurate product recognition accuracy can be guaranteed.

[0052] An embodiment of the present invention further provides a computer device, Figure 5 It is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, the above-mentioned new product data collection method based on the shopping cart is implemented.

[0053] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned new product data collection method based on a shopping cart is implemented.

[0054] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned new product data collection method based on a shopping cart is implemented.

[0055] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for collecting new commodity data based on a shopping cart, characterized in that: Applied to shopping cart, including: Obtaining barcode information of the product, where the barcode information is obtained by a shopper scanning with a barcode scanner; Searching for characteristic value data corresponding to the barcode information from a database; When the number of the characteristic value data corresponding to the searched barcode information is less than the number threshold, the commodity is determined to be a new commodity, and a shooting acquisition signal is sent to the visual sensor; receiving multiple frames of video images of new products sent by a visual sensor, and screening out candidate images of the new products from the multiple frames of video images; Matching the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, storing the candidate image of the new product and the corresponding feature value data in the database; When the match fails, the feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information. When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

2. The method according to claim 1, characterized in that Filter candidate images of new products from multiple video frames, including: Crop multiple frames of product images from multiple frames of video images; Perform image quality evaluation on each frame of product image to obtain the image quality of each frame of product image; Filter out the product image with the highest image quality from multiple frames of product images; If the image quality of the product image with the highest image quality is greater than the quality threshold, the product image with the highest image quality is determined as a candidate image of the new product.

3. The method according to claim 2, characterized in that Cropping multiple frames of product images from multiple frames of video images, including: Input each frame of video image into a moving commodity detection model, wherein the moving commodity detection model is used to detect and crop moving commodities in each frame of video image to obtain commodity images, wherein the moving commodity detection model is a target detection model built based on a deep neural network; Perform image quality assessment on each frame of product image, including: Each frame of commodity image is input into a commodity image quality assessment model to obtain the image quality of each frame of commodity image. The commodity image quality assessment model is a classification model built based on a deep neural network.

4. The method according to claim 2, characterized in that Also includes: If the image quality of the product image with the highest image quality is not greater than the quality threshold, the multiple frames of product images are abandoned, and the shooting acquisition signal continues to be sent to the visual sensor. After re-obtaining multiple frames of video images, candidate images of new products are screened.

5. The method according to claim 1, characterized in that Matching the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database includes: Respectively calculating the distance between the feature value data of the candidate image and the feature value data corresponding to the barcode information in the database, wherein the distance is a cosine distance or a Euclidean distance; When the minimum value of all distances is less than the first distance threshold, it is determined that the match is successful.

6. The method according to claim 1, characterized in that The feature value data of the candidate image of the new product is compared with the feature value data corresponding to other barcode information in the database except the barcode information, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database, including: Respectively calculating the distance between the feature value data of the candidate image and each feature value data corresponding to other barcode information except the barcode information, wherein the distance is a cosine distance or a Euclidean distance; Determine the minimum of all distances as the alignment difference; When the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

7. The method according to claim 1, characterized in that Also includes: When the number of the characteristic value data corresponding to the searched barcode information is not less than the quantity threshold, determining that the commodity is a non-new commodity, and judging whether the non-new commodity is in the correction and verification stage; If yes, send a shooting acquisition signal to the visual sensor, receive multiple frames of video images of non-new products sent by the visual sensor, and select picket candidate images from the multiple frames of video images; The number of matches for a non-new product is increased by 1, where the number of matches for the non-new product is initialized to zero when the product is first identified as a non-new product; Searching for feature value data corresponding to the barcode information that matches the feature value data of the picket candidate image from a database, and after the shopper confirms the feature value data corresponding to the matched barcode information through an interactive screen, adding 1 to the frequency of the feature value data corresponding to the matched barcode information, and the frequency of each feature value data is initialized to zero when a product is first identified as a non-new product; When the number of matches of a non-new product exceeds the matching number threshold, judging whether the feature value data under the barcode information of the non-new product belongs to other products based on the frequency of all feature value data of the non-new product, and deleting the feature value data belonging to other products from the barcode information of the non-new product; When the quantity of the characteristic value data corresponding to the barcode information of the non-new product is less than the quantity threshold, the non-new product is re-identified as a new product.

8. The method according to claim 7, characterized in that After storing the candidate images of the new product and the corresponding feature value data in the database, the following steps are also included: If the number of feature value data corresponding to the barcode information of the new product is not less than the quantity threshold, a non-new product label and the number of matches are marked on the barcode information, and the non-new product label contains the marking time; Initialize the number of matches to 0; Determine whether a non-new product is in the correction and verification stage, including: Check the marking time of the non-new product label marked on the barcode information of the non-new product; When the distance between the marked time and the current time is less than a preset time, it is determined that the non-new product is in the correction and verification stage; Re-identify the non-new product as a new product, including: deleting the non-new product label on the barcode information of the non-new product.

9. The method according to claim 7, characterized in that Searching for feature value data corresponding to the barcode information that matches the feature value data of the picket candidate image from a database, and after the shopper confirms the feature value data corresponding to the matched barcode information through an interactive screen, adding 1 to the frequency of the feature value data corresponding to the matched barcode information, including: Respectively calculating the distance between the feature value data of the candidate picket image and the feature value data corresponding to the barcode information in the database; When the minimum value of all distances is less than the first distance threshold, the match is confirmed to be successful, and the characteristic value data corresponding to the minimum value of all distances in the database is used as the characteristic value data corresponding to the matched barcode information; Display the successful matching information on the interactive screen of the shopping cart for the shopper to confirm; After the shopper confirms that the matching success information is correct, the frequency of the feature value data corresponding to the matched barcode information is increased by 1.

10. The method according to claim 7, characterized in that According to the frequency of all characteristic value data of the non-new product, it is determined whether the characteristic value data under the barcode information of the non-new product belongs to other products, and the characteristic value data belonging to other products is deleted from the barcode information of the non-new product, including: Filter out the feature value data of non-new products whose frequency is less than the frequency threshold from the database as the feature value data to be verified; For each characteristic value data to be verified, the distance between the characteristic value data to be verified and each characteristic value data corresponding to other commodities in the database is calculated respectively. If the minimum value of all distances is less than the first distance threshold, it is determined that the characteristic value data to be verified belongs to other commodities, and the data value to be verified is deleted from the barcode information of non-new commodities.

11. A new commodity data collection device based on a shopping cart, characterized in that: Applied to shopping cart, including: A barcode information acquisition module is used to obtain the barcode information of the product, where the barcode information is obtained by the shopper through scanning with a barcode scanner; A characteristic value data search module, used to search the characteristic value data corresponding to the barcode information from a database; A new product shooting module, used for determining that the product is a new product when the number of feature value data corresponding to the searched barcode information is less than a quantity threshold, and sending a shooting acquisition signal to the visual sensor; An image screening module is used to receive multiple frames of video images of new products sent by a visual sensor, and screen candidate images of new products from the multiple frames of video images; A first feature value data comparison module is used to match the feature value data of the candidate image of the new product with the feature value data corresponding to the barcode information in the database, and when the match is successful, the candidate image of the new product and the corresponding feature value data are stored in the database; The second feature value data comparison module is used to compare the feature value data of the candidate image of the new product with the feature value data corresponding to other barcode information in the database except the barcode information when the matching fails, and when the comparison difference is greater than the difference threshold, the candidate image of the new product and the corresponding feature value data are stored in the database.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images