Multi-commodity Automatic Calibration and Recognition Method and Intelligent Self-service Cashier

Abstract

The present invention relates to the technical field of image recognition, and discloses a multi-commodity automatic calibration recognition method and an intelligent self-service cash register. The steps of the method include: extracting the front image of a commodity from the commodity image; identifying the label area and selecting associated feature points for each label area; segmenting the commodity area from the front image of the commodity; determining the associated commodity area of each label area; extracting the side image of the commodity and determining the associated commodity area of each label area in the side image of the commodity; reconstructing the three-dimensional model of the commodity; scanning the label area to obtain commodity information, and verifying the commodity information based on the gravity sensor and the three-dimensional model of the commodity. The present invention not only improves the accuracy and settlement efficiency of commodity recognition, but also improves the user experience, enhances the security and reliability of the system, and provides a more efficient, convenient and reliable self-service cash register solution.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a multi-commodity automatic calibration recognition method and an intelligent self-checkout cashier. Background Art

[0002] Existing self-checkout technologies have met basic needs to a certain extent, but there are still obvious deficiencies in handling the simultaneous settlement of multiple commodities. Most current self-checkout systems settle accounts based on the method of scanning each commodity one by one. Once multiple commodities are placed in the settlement area at the same time, it is very difficult for the system to quickly and accurately identify each commodity. Since each commodity needs to be scanned one by one, the multi-commodity settlement process often takes a long time. Especially during peak hours, it may lead to customers waiting in line for too long. In actual use, the labels of some commodities are also prone to mislabeling, that is, the label of commodity B is pasted on commodity A, resulting in the wrong settlement of commodity B during settlement, causing losses to users and the inability to match the commodity inventory information.

[0003] For example, the Chinese patent application with the authorization announcement number CN116308327B discloses a self-checkout system and method based on RFID technology. The system includes an information extraction module, a commodity verification module, a verification failure module, a verification success module, a commodity settlement module, and a settlement success module. It can obtain the commodity weight, commodity label, and identification information of the commodities to be settled by using a shopping cart, verify the commodities to be settled placed in the shopping cart by using the commodity weight, commodity label, and identification information, store the settlement information of the successfully verified commodities in a first-level RFID component, and perform settlement uniformly in the settlement workshop. Thus, the steps of verifying and settling each commodity one by one are split into the shopping process, reducing the waiting time in the settlement workshop and improving the cashier efficiency during self-checkout, but it does not solve the problem of mislabeled commodity labels.

[0004] As the patent application with the publication number CN118134474A discloses a self-checkout method and system, which includes the following steps: Based on the cameras in the self-checkout area, using image capture technology, the selected goods of the customer are captured to generate a set of product images; Based on the set of product images, using the convolutional neural network algorithm, product image recognition is performed to generate product list information. In this technical solution, by using image capture technology and convolutional neural network algorithm for product recognition, the efficiency and accuracy of the checkout process are improved. The personalized recommendation list generated through data analysis and collaborative filtering algorithm enhances the shopping experience of customers and provides more accurate product recommendations. Combining symmetric encryption algorithm and multi-factor authentication technology significantly enhances the security of the payment process, reduces the fraud risk, and also provides more diverse payment methods, making the entire shopping process more convenient and personalized, and improving the operation efficiency of retailers and customer satisfaction. However, its authentication process is complex, the processing efficiency is low, and the recognition difficulty of multiple products is large, resulting in deficiencies.

[0005] The information disclosed in this background art section is only intended to enhance the overall understanding of the present invention and should not be regarded as an admission or any form of implication that this information constitutes prior art already known to those of ordinary skill in the art. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art, and provide a multi-product automatic correction and recognition method and an intelligent self-checkout machine, which improve the accuracy of product recognition and the settlement efficiency, and enhance the security and reliability of the checkout system.

[0007] To solve the above technical problems, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a multi-product automatic correction and recognition method, including the following steps:

[0009] Collect the product images and the recognition information of the product radio frequency tags in the intelligent self-checkout machine, and the product images include multiple product patterns;

[0010] According to the product images and the recognition information, combined with the preset images of each product, judge whether the radio frequency tag of each product is corresponding and bound to the product;

[0011] If all are bound, perform product settlement operations based on the recognition information of each product.

[0012] As a preferred solution of the multi-product automatic correction and recognition method of the present invention, the step of judging whether the radio frequency tag of each product is corresponding and bound to the product according to the product images and the recognition information, combined with the preset images of each product, includes:

[0013] Identify the label areas from the front image of the commodity, number each label area, and select associated feature points for each label area;

[0014] Respectively segment the commodity areas from the front image of the commodity based on each associated feature point;

[0015] Align the label areas with the commodity areas to determine the associated commodity areas of each label area;

[0016] Extract the side image of the commodity from the commodity image; mark the positions of each label area in the side image of the commodity, and determine the associated commodity areas of each label area in the side image of the commodity;

[0017] Reconstruct the three-dimensional model of the commodity based on the associated commodity areas of each label area in the front image and the side image of the commodity;

[0018] Scan the label areas to obtain commodity information, and verify the commodity information based on the gravity sensor and the three-dimensional model of the commodity.

[0019] As a preferred solution of the multi-commodity automatic calibration and recognition method of the present invention, wherein: the commodity image is an image of the cash settlement area collected by a camera, including images collected by cameras arranged directly above and laterally above the cash settlement area; the front image of the commodity is extracted based on the commodity image collected directly above the cash settlement area, and the extraction method is background subtraction.

[0020] As a preferred solution of the multi-commodity automatic calibration and recognition method of the present invention, wherein: the method for selecting associated feature points for each label area is as follows: calculate the coordinate range of the label area in the front image of the commodity; set a feature area for the label area in the front image of the commodity, and uniformly select n pixel points as the associated feature points of the label area, where n is a positive integer; wherein, the feature area of the label area consists of all pixel points whose distance from the label area is not less than and not greater than of all pixel points, is a preset first distance threshold, is a preset second distance threshold.

[0021] As a preferred solution of the multi-commodity automatic calibration and recognition method of the present invention, wherein: the segmenting the commodity areas from the front image of the commodity based on each associated feature point includes using the associated feature points as seed points for region growing, and the steps are as follows:

[0022] S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If it meets, mark the neighbor pixel as a seed point and add it to the current region;

[0023] The similarity criterion is specifically as follows: If the straight-line distance between the seed point and the neighboring pixel in the RGB space is less than the preset similarity threshold, then the neighboring pixel meets the similarity criterion; otherwise, the neighboring pixel does not meet the similarity criterion.

[0024] S302: Mark the seed point that has completed the similarity criterion check for all neighboring pixels as processed;

[0025] S303: Repeat S301 to S302. The seed points marked as processed will no longer participate in the check until all seed points are marked as processed, and then stop region growing; record the pixel range of the current region as the segmented commodity region.

[0026] As a preferred embodiment of the multi-commodity automatic calibration and recognition method of the present invention, wherein: the associated commodity region is the commodity region of the commodity corresponding to the label region; the method for determining the associated commodity region of each label region is as follows:

[0027] Extract h commodity regions segmented by i associated feature points based on the label region; remove duplicates from the h commodity regions and delete the completely overlapping commodity regions; if only one commodity region remains after deduplication, the remaining commodity region is the associated commodity region of the label region; if m commodity regions remain after deduplication, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity region, and the commodity region containing the most associated feature points is the associated commodity region of the label region.

[0028] As a preferred embodiment of the multi-commodity automatic calibration and recognition method of the present invention, wherein: the commodity side image is extracted based on the commodity image collected from the upper side of the cashier settlement area; the method for marking the position of each label region in the commodity side image is as follows: Detect and recognize each label region in the commodity side image, and extract the character information of the label region; based on the character information, find the corresponding label region in the commodity front image for each label region in the commodity side image; the corresponding label regions in the commodity side image and the commodity front image have the same character information; obtain the number of each label region and mark the position of each label region in the commodity side image.

[0029] As a preferred embodiment of the multi-commodity automatic calibration and recognition method of the present invention, wherein: after determining the associated commodity region of each label region in the commodity side image, perform a secondary alignment verification on each label region and the corresponding associated commodity region, including the following steps:

[0030] Segment the associated commodity region of the label region from the commodity front image and mark it as the first associated image; segment the associated commodity region of the label region from the commodity side image and mark it as the second associated image;

[0031] Based on feature point matching, determine whether the first associated image and the second associated image belong to the same set of product images; if so, perform secondary alignment verification; if not, re-determine the associated product area of the label area; the fact that they belong to the same set of product images means that the first associated image and the second associated image are images of the same product from different angles.

[0032] As a preferred solution of the multi-product automatic calibration and recognition method of the present invention, the method for re-determining the associated product area of the label area is as follows:

[0033] Segment p product areas after removing duplicates from the front product image, and all are marked as front product areas; segment q product areas after removing duplicates from the side product image, and all are marked as side product areas; both p and q are positive integers;

[0034] Based on feature point matching, group the front product areas and the side product areas; any group contains one front product area belonging to the same set of product images and the corresponding side product area; delete the front product areas and the side product areas that are not included in the group;

[0035] If there is only one group, both the front product area and the side product area in the group are the associated product areas of the label area; if there are more than one group, count the total number of associated feature points contained in each group, and both the front product area and the side product area in the group with the most associated feature points are the associated product areas of the label area; mark the front product area in the associated product area as the first associated image, and mark the side product area in the associated product area as the second associated image.

[0036] As a preferred solution of the multi-product automatic calibration and recognition method of the present invention, the product information includes product type and product weight; the verification of the product information includes product type verification and product weight verification;

[0037] The method for product type verification is as follows: Compare and match the three-dimensional model of the product with the product models in the preset three-dimensional product database to identify the product type; if the identified product type is consistent with the product type obtained by scanning the label area, the product type verification is passed; otherwise, the product type verification fails, prompt to mark the first associated image corresponding to the product with failed product type verification in the front product image, and prompt the user to re-place the product or temporarily remove the product from the cashier settlement area;

[0038] The method for verifying the weight of the goods is as follows: Scan all label areas, obtain the weight of the goods, and calculate the total weight of all goods, which is recorded as the scanned total weight; Calculate the total weight of all goods in the cash settlement area through a gravity sensor, which is recorded as the weighed total weight; Calculate the weight difference between the weighed total weight and the scanned total weight. If the weight difference is less than a preset weight difference threshold, the weight of the goods is verified; Otherwise, the weight verification of the goods fails, and the user is prompted to remove all goods from the cash settlement area and place them one by one into the cash settlement area for self-checkout.

[0039] In a second aspect, the present invention provides an intelligent self-checkout machine for automatic correction and recognition of multiple goods, including a cash register, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; Among them:

[0040] The cash register includes a cash settlement area and a gravity sensor. The cash settlement area is used to place goods, and the gravity sensor is used to detect the weight of the goods;

[0041] The camera unit includes cameras arranged above and above the side of the cash settlement area, which are used to collect images of goods;

[0042] The label reading device is used to scan the label area and obtain product information;

[0043] The display device includes a user interface screen, which is used to display product information, prices, payment options to the user, and provide operation guides;

[0044] The storage unit is used to store a three-dimensional product database, user account data, and transaction records;

[0045] The processing unit is configured with an image processor, which is used to align the label area with the product area, and perform secondary alignment verification on the label area and the associated product area, and is used to reconstruct the three-dimensional model of the product; The processing unit is also used to verify product information.

[0046] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0047] By introducing advanced image processing technology and three-dimensional reconstruction algorithms, the present invention can improve the accuracy and efficiency of product recognition. By accurately numbering the label area and selecting associated feature points, the accurate matching between each label and its corresponding product area is ensured. Based on the region growing algorithm, the product area can be reliably segmented, and a high recognition accuracy can be maintained even when the products are mutually occluded.

[0048] The present invention can efficiently identify multiple commodities in one operation. On the one hand, it combines radio frequency identification for fast identification, and on the other hand, it combines image processing. The image processing process is based on radio frequency identification, thus greatly reducing the amount of identification processing and improving the efficiency. In addition, the image is further used as a calibration scheme for radio frequency information. By aligning the tags with the commodity images one by one, it ensures that all commodities can be correctly identified and settled, reducing the risks of missed scanning, mis-scanning, or incorrect pasting of commodity labels.

[0049] The commodity weight verification function is introduced. By measuring the total weight of the commodity with a gravity sensor and comparing it with the weight data in the tag information, it effectively prevents the behavior of commodity substitution or replacement. Comparing the three-dimensional model with the commodity models in the preset database further enhances the reliability of commodity type verification and reduces the recognition errors caused by packaging changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0051] Figure 1 It is a flowchart of the multi-commodity automatic calibration and recognition method provided by the present invention;

[0052] Figure 2 It is a flowchart of the method for segmenting the commodity area from the front image of the commodity provided by the present invention;

[0053] Figure 3 It is a schematic structural diagram of the intelligent self-service cash register for multi-commodity automatic calibration and recognition provided by the present invention.

[0054] Reference numerals: 1, cash desk; 101, cash settlement area; 2, tag reading device; 301, first camera; 302, second camera; 4, display device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following will specifically describe the technical solutions of the present invention through the drawings and specific embodiments. It should be understood that the specific features in the embodiments of the present invention are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. Without conflict, the technical features in the embodiments of the present invention and the embodiments can be combined with each other.

[0056] Embodiment 1

[0057] This embodiment introduces a multi-commodity automatic calibration and recognition method, including the following steps:

[0058] Collect the product images and the identification information of the product radio frequency tags inside the intelligent self-service cash register. The product images include multiple product patterns;

[0059] Based on the product images and the identification information, and in combination with the preset images of each product, determine whether the radio frequency tag of each product is bound to the corresponding product;

[0060] If all are bound, perform the product settlement operation based on the identification information of each product.

[0061] The present invention can efficiently identify multiple products in one operation. On the one hand, it combines radio frequency identification for fast identification, and on the other hand, it combines image processing. The image processing process is based on radio frequency identification, thus greatly reducing the amount of identification processing and improving the efficiency. In addition, the image is further used as a calibration scheme for radio frequency information. By aligning the tags with the product images one by one, it ensures that all products can be correctly identified and settled, reducing the risks of missed scanning, mis-scanning, or incorrect pasting of product tags.

[0062] Specifically, in the embodiment of the present application, the product images and the radio frequency tag information are collected. When the customer places the product in the scanning area of the intelligent self-service cash register, the machine will automatically collect the product images and the radio frequency tag information. The product images are captured by a camera and contain the patterns of multiple products; the radio frequency tag information is obtained through an RFID reader, and each product has a unique radio frequency tag. When determining whether the radio frequency tag is bound to the product, the system will perform a matching judgment based on the collected product images and radio frequency tag information, in combination with the preset images of each product.

[0063] It should be noted that the preset images are stored in the system in advance and are used to identify different types of products. By comparing the collected product images with the preset images, the system can determine whether the radio frequency tag of each product is correctly bound to its corresponding product. When performing the product settlement operation, if the radio frequency tags of all products are correctly bound to their corresponding products, the system will perform the product settlement operation based on the identification information of each product, that is, the system will calculate the total amount that the customer needs to pay according to the identified product information and display it on the screen for the customer to confirm.

[0064] The core concept of the present application is that it can detect whether there is a label error situation during the user's self-service settlement, that is, the label originally for product A is placed on product B, resulting in incorrect product information, thus avoiding unnecessary losses.

[0065] The following details the core concept of the present application.

[0066] This embodiment introduces a multi-product automatic calibration and identification method. Refer to Figure 1 , this method includes the following steps:

[0067] S1: Collect the product image and extract the front image of the product from the product image.

[0068] The product image is an image of the cashier settlement area collected by a camera, including images collected by cameras set directly above and laterally above the cashier settlement area; the front product image is extracted based on the product image collected directly above the cashier settlement area, and the extraction method is background subtraction. The steps are as follows: First, when there is no product placed in the cashier settlement area, take a background image with the camera directly above as the benchmark for subsequent background subtraction; when the product is placed in the cashier settlement area, collect the product image.

[0069] Use a background subtraction algorithm (such as frame difference method, Gaussian mixture model, etc.) to subtract the background reference image from the product image to separate and obtain the front product image. By background subtraction, the product image can be separated from the complex background, which can improve the efficiency and accuracy of subsequent object detection and reduce recognition errors when there are various items mixed in the cashier settlement area.

[0070] S2: Identify the label area from the front product image, number each label area, and select associated feature points for each label area.

[0071] The label area is the image area corresponding to the label of the product; extract the character information of the label area and assign a number to each label area; the extracted character information is used to identify the label area. Labels of products such as barcodes, QR codes, RFID tags, etc. contain the necessary product information for cashier settlement.

[0072] The method for selecting associated feature points for each label area is as follows: Calculate the coordinate range of the label area in the front product image; set a feature area for the label area in the front product image, and uniformly select n pixel points as the associated feature points of the label area in the feature area, where n is a positive integer; among them, the feature area of the label area consists of all pixel points whose distance from the label area is not less than and not greater than where is a preset first distance threshold, is a preset second distance threshold.

[0073] S3: Segment the product area from the front product image respectively based on each associated feature point.

[0074] The segmentation of the product area from the front product image based on each associated feature point includes using the associated feature point as a seed point for region growing, referring to Figure 2 , and the steps are as follows:

[0075] S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If it meets, mark the neighbor pixel as a seed point and add it to the current region.

[0076] The similarity criterion is as follows: If the straight-line distance between the seed point and the neighbor pixel in the RGB space is less than the preset similarity threshold, the neighbor pixel meets the similarity criterion; otherwise, the neighbor pixel does not meet the similarity criterion.

[0077] S302: Mark the seed point that has completed the similarity criterion check for all neighbor pixels as processed.

[0078] S303: Repeat S301 to S302. The seed points marked as processed will no longer participate in the check until all seed points are marked as processed, and then stop region growing. Record the pixel range of the current region as the segmented commodity region.

[0079] S4: Align the label region with the commodity region to determine the associated commodity region of each label region.

[0080] The associated commodity region is the commodity region of the commodity corresponding to the label region, that is, the label in the label region is the label of the corresponding commodity. The method for determining the associated commodity region of each label region is as follows:

[0081] Extract the h commodity regions segmented by i associated feature points based on the label region. Remove duplicates from the h commodity regions and delete the completely overlapping commodity regions. If only one commodity region remains after deduplication, the remaining commodity region is the associated commodity region of the label region. If m commodity regions remain after deduplication, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity region, and the commodity region containing the most associated feature points is the associated commodity region of the label region.

[0082] Based on region growing, taking the i associated feature points as seed points respectively, h commodity regions can be obtained. Ideally, the pixel points contained in the h commodity regions completely overlap, that is to say, the h commodity regions are actually the same commodity region, and at this time the commodity represented by this commodity region is the commodity corresponding to the label. However, due to the existence of situations such as label angle skew and mutual occlusion between commodities, some associated feature points may be located on other commodities, so the commodity region segmented based on this associated feature point is not the commodity region corresponding to this label. Based on the number of associated feature points contained, the commodity regions mis-segmented due to improper positions of the associated feature points can be removed.

[0083] S5: Extract the side image of the commodity from the commodity image. Mark the position of each label region in the side image of the commodity, and determine the associated commodity region of each label region in the side image of the commodity.

[0084] The side image of the commodity is extracted based on the commodity image collected above the side of the cash settlement area; the method for marking the position of each label area in the side image of the commodity is as follows: Detect and recognize each label area in the side image of the commodity, and extract the character information of the label area; Based on the character information, find the corresponding label area in the front image of the commodity for each label area in the side image of the commodity; The corresponding label areas in the side image of the commodity and the front image of the commodity have the same character information; Obtain the number of each label area and mark the position of each label area in the side image of the commodity. Through the character information, it can be judged which label areas in the side image of the commodity and the front image of the commodity are the same label area, so as to realize the one-to-one correspondence between the label areas in the side image of the commodity and the label areas in the front image of the commodity.

[0085] The method for determining the associated commodity area of each label area in the side image of the commodity is as follows: Select associated feature points for each label area in the side image of the commodity; Respectively segment the commodity area from the side image of the commodity based on each associated feature point; Remove duplicates from the segmented commodity areas, and determine the associated commodity area of the label area from the de-duplicated commodity areas. For the specific method details, refer to the method for determining the associated commodity area of each label area in the front image of the commodity, which will not be elaborated here.

[0086] After determining the associated commodity area of each label area in the side image of the commodity, perform secondary alignment verification on each label area and its corresponding associated commodity area, including the following steps:

[0087] Segment the associated commodity area of the label area from the front image of the commodity and mark it as the first associated image; Segment the associated commodity area of the label area from the side image of the commodity and mark it as the second associated image;

[0088] Based on feature point matching, judge whether the first associated image and the second associated image belong to the same set of commodity images; If so, pass the secondary alignment verification; If not, re-determine the associated commodity area of the label area; The belonging to the same set of commodity images means that the first associated image and the second associated image are images of the same commodity from different angles; Through the method of feature point detection and matching, it can be determined whether two images are images of the same commodity. For example, detect feature points, match feature points, and count the number of successfully matched feature points to judge whether the first associated image and the second associated image belong to the same set of commodity images.

[0089] The method for re-determining the associated commodity area of the label area is as follows:

[0090] p product regions after deduplication are segmented from the front image of the product and are all marked as front product regions; q product regions after deduplication are segmented from the side image of the product and are all marked as side product regions; both p and q are positive integers;

[0091] Based on feature point matching, the front product regions and the side product regions are grouped; any group contains one front product region belonging to the same group of product images and the corresponding side product region; the front product regions and the side product regions not included in the group are deleted; each product region corresponds to one product. Based on the feature point matching algorithm, the product regions that do not match the product regions at other angles will not be included in the group. Not being included in the group indicates that the label is not bound to the product region in both the front image and the side image of the product, which means that the label is not the label of the product corresponding to the product region.

[0092] If there is only one group, the front product region and the side product region in the group are both the associated product regions of the label region; if there are more than one group, the total number of associated feature points included in each group is counted, and the front product region and the side product region in the group with the most associated feature points are both the associated product regions of the label region; the front product region in the associated product region is marked as the first associated image, and the side product region in the associated product region is marked as the second associated image.

[0093] S6: Reconstruct the 3D model of the product based on the associated product regions of each label region in the front image and the side image of the product;

[0094] For any label region, 3D reconstruction is performed through its first associated image and second associated image to obtain the 3D model of the product corresponding to the label region. By executing S1 to S6, the alignment of the labels of multiple products in the cash settlement area with the product images is achieved, and the 3D models of the products corresponding to the labels are obtained, providing a rigorous data basis for the verification of product information.

[0095] S7: Scan the label region, obtain product information, and verify the product information based on the gravity sensor and the 3D model of the product.

[0096] The product information includes product type and product weight; the verification of the product information includes product type verification and product weight verification;

[0097] The method for verifying the commodity type is as follows: Compare and match the three-dimensional model of the commodity with the commodity models in the preset three-dimensional commodity database to identify the commodity type; if the identified commodity type is consistent with the commodity type obtained by scanning the label area, the commodity type verification is passed; otherwise, the commodity type verification fails, and it is prompted to mark the first associated image corresponding to the commodity with failed commodity type verification in the front image of the commodity, and prompt the user to reposition the commodity or temporarily remove the commodity from the cash settlement area;

[0098] Through matching with the three-dimensional database, the true shape of the commodity can be effectively identified. Even when the commodity is partially blocked or the label is misaligned, it can be accurately identified, reducing substitution and cashier errors. Preferably, multiple cameras can be set to collect side images of the commodity from multiple angles, and then obtain the third associated image, the fourth associated image,..., the Nth associated image of any label area from multiple side images of the commodity, and perform three-dimensional reconstruction of the commodity based on all associated images, thereby improving the accuracy of three-dimensional reconstruction of the commodity.

[0099] The method for verifying the commodity weight is as follows: Scan all label areas to obtain the commodity weight, and calculate the total weight of all commodities, denoted as the scanned total weight; Calculate the total weight of all commodities in the cash settlement area through the gravity sensor, denoted as the weighed total weight; Calculate the weight difference between the weighed total weight and the scanned total weight. If the weight difference is less than the preset weight difference threshold, the commodity weight verification is passed; otherwise, the commodity weight verification fails, and it is prompted that the user moves all commodities out of the cash settlement area and puts them into the cash settlement area one by one for self-checkout. By weighing, identifying, and scanning each commodity for checkout, the commodity with weighing error can be found.

[0100] Embodiment 2

[0101] This embodiment is the second embodiment of the present invention; based on the same inventive concept as Embodiment 1, this embodiment introduces an intelligent self-checkout machine for automatic correction and recognition of multiple commodities, including a cashier desk, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; where:

[0102] The cashier desk includes a cash settlement area and a gravity sensor. The cash settlement area is used to place commodities, and the gravity sensor is used to detect the weight of commodities;

[0103] The camera unit includes cameras arranged above and above the side of the cash settlement area, which are used to collect commodity images; based on cameras at different angles, commodity images at different angles can be collected, which helps in image verification and recognition of commodities and three-dimensional reconstruction.

[0104] The label reading device is used to scan the label area and obtain product information; the obtained product information includes product weight, product type, and product price; the label reading device includes a barcode scanner, a QR code reader, an RFID reader / writer, etc., for reading the information contained in the labels on the products.

[0105] The display device includes a user interface screen for presenting product information, prices, payment options to the user, and providing operation guides; the display device also includes an input interface that supports various payment methods (such as credit cards, mobile payments), and an output interface for printing shopping receipts, supporting multiple payment methods to complete transactions and automatically generating shopping receipts.

[0106] The storage unit is used to save a three-dimensional product database, user account data, and transaction records; among them, the three-dimensional product database contains the product models of each product. By matching the three-dimensional models of the products on the cash register with the product models in the product database, the types of products on the cash register can be identified.

[0107] The processing unit is configured with an image processor for aligning the label area with the product area, and for performing secondary alignment verification on the label area and the associated product area, and for reconstructing the three-dimensional model of the product; the processing unit is also used for verifying product information. The image processor is configured with a variety of image processing algorithms, including background subtraction, feature point extraction, region growing, graphic segmentation, three-dimensional reconstruction, etc.

[0108] For the specific functional implementation of the above units, refer to the relevant content in the multi-product automatic calibration and recognition method described in Embodiment 1, which will not be elaborated here.

[0109] Embodiment 3

[0110] Based on the same inventive concept as other embodiments, refer to Figure 3 , this embodiment provides an application example of an intelligent self-service cash register for multi-product automatic calibration and recognition, specifically as follows: The intelligent self-service cash register includes a cash register 1; the cash register 1 includes a cash settlement area 101 for placing products; a gravity sensor is provided under the cash settlement area 101 for monitoring the product weight;

[0111] The intelligent self-service cash register also includes a label reading device 2, which is fixed to the cash register 1 through a bracket and can respectively read the RF tags of multiple products placed in the cash settlement area 101 to obtain the product information of multiple products;

[0112] The intelligent self-service cash register further includes a first camera 301 and a second camera 302. Among them, the first camera 301 is fixed on the cash register 1 through a bracket and is located directly above the cash settlement area 101 for collecting commodity settlement images of multiple commodities from directly above; the second camera 302 is fixed on the cash register 1 through a bracket and is located obliquely above the cash settlement area 101 for collecting commodity settlement images of multiple commodities from obliquely above.

[0113] The intelligent self-service cash register further includes a display device 4; the display device 4 is fixed on the cash register 1 through a bracket and is used to list and display the commodity information and prices of multiple commodities to customers, and provide input interfaces for payment options and various payment methods.

[0114] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The above describes the embodiments of the present invention in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the purpose and scope of the present invention. These all fall within the protection scope of the present invention.

Claims

1. A multi-commodity automatic correction and identification method, characterized in that: The following steps are involved: Collecting commodity images and identification information of commodity radio frequency tags in the intelligent self-service cash register, wherein the commodity images include a plurality of commodity patterns; According to the product image and identification information, combined with the preset image of each product, determine whether the radio frequency tag of each product is bound to the corresponding product; If all are bound, commodity settlement operations are performed based on the identification information of each commodity; According to the product image and identification information, combined with the preset image of each product, it is determined whether the radio frequency tag of each product is bound to the corresponding product, specifically including: Identify label regions from the product front image, number each label region, and select associated feature points for each label region; The method for selecting associated feature points for each label area is as follows: calculating the coordinate range of the label area in the product front image; setting a feature area for the label area in the product front image, and uniformly selecting n pixels in the feature area as associated feature points of the label area, where n is a positive integer; wherein the feature area of ​​the label area is determined by the distance from the label area to the label area being not less than and not greater than All the pixels of is the preset first distance threshold, is a preset second distance threshold; Segmenting a commodity area from the commodity front image based on each associated feature point; Align the label area with the product area and determine the associated product area of ​​each label area; Extracting a product side image from the product image; marking the position of each label area in the product side image, and determining the associated product area of ​​each label area in the product side image; Reconstructing a three-dimensional model of the product based on the associated product area of ​​each label area in the product front image and the product side image; The label area is scanned to obtain product information, and the product information is verified based on the gravity sensor and the three-dimensional model of the product.

2. The method for automatic correction and identification of multiple commodities according to claim 1, characterized in that: The product image is an image of the cashier settlement area captured by a camera, including images captured by cameras arranged directly above and above the cashier settlement area; the product front image is extracted based on the product image captured directly above the cashier settlement area, and the extraction method is background subtraction.

3. The multi-commodity automatic correction and identification method according to claim 2, characterized in that: The steps of segmenting the commodity area from the commodity front image based on each associated feature point include using the associated feature point as a seed point for region growing, and the steps are as follows: S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If so, mark the neighbor pixel as a seed point and add it to the current area; The similarity criterion is specifically as follows: if the straight-line distance between the seed point and the neighboring pixel in the RGB space is less than a preset similarity threshold, the neighboring pixel satisfies the similarity criterion; otherwise, the neighboring pixel does not satisfy the similarity criterion; S302: marking the seed point that has completed the similarity criterion check of all neighboring pixels as processed; S303: Repeat S301 to S302, and the seed points marked as processed are no longer involved in the inspection until all seed points are marked as processed and the region growth is stopped; the pixel range of the current region is recorded as the segmented product region.

4. The method for automatic correction and identification of multiple commodities according to claim 3, characterized in that: The associated product area is the product area of ​​the product corresponding to the label area; the method for determining the associated product area of ​​each label area is as follows: Extract h commodity areas segmented based on i associated feature points of the label area; remove duplicates from the h commodity areas and delete completely overlapping commodity areas; if only one commodity area remains after removing duplicates, the remaining commodity area is the associated commodity area of ​​the label area; if m commodity areas remain after removing duplicates, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity area, and the commodity area containing the most associated feature points is the associated commodity area of ​​the label area.

5. The method for automatic correction and identification of multiple commodities according to claim 4, characterized in that: The side image of the product is extracted based on the product image collected on the upper side of the cashier settlement area; the method for marking the position of each label area in the side image of the product is as follows: detect and identify each label area in the side image of the product, and extract the character information of the label area; based on the character information, find the label area corresponding to each label area in the side image of the product in the front image of the product; the corresponding label areas in the side image of the product and the front image of the product have the same character information; obtain the number of each label area and mark the position of each label area in the side image of the product.

6. The multi-commodity automatic correction and identification method according to claim 5, characterized in that: After determining the associated product area of ​​each label area in the product side image, a secondary alignment verification is performed on each label area and the corresponding associated product area, including the following steps: Segment the associated product area of ​​the label area from the product front image, and mark it as the first associated image; segment the associated product area of ​​the label area from the product side image, and mark it as the second associated image; Based on feature point matching, determine whether the first associated image and the second associated image belong to the same group of product images; if so, perform secondary alignment verification; if not, re-determine the associated product area of ​​the label area; belonging to the same group of product images means that the first associated image and the second associated image are images of the same product at different angles.

7. The method for automatic correction and identification of multiple commodities according to claim 6, characterized in that: The method for re-determining the associated product area of ​​the label area is as follows: The p product regions after deduplication is segmented from the product front image are all marked as the front product region; the q product regions after deduplication is segmented from the product side image are all marked as the side product region; p, q are both positive integers; The front product area and the side product area are grouped based on feature point matching; any group contains a front product area and a corresponding side product area belonging to the same group of product images; the front product area and the side product area that are not grouped are deleted; If there is only one group, the front product area and the side product area in the group are both associated product areas of the label area; if there is more than one group, the total number of associated feature points contained in each group is counted, and the front product area and the side product area in the group containing the most associated feature points are both associated product areas of the label area; the front product area in the associated product area is marked as the first associated image, and the side product area in the associated product area is marked as the second associated image.

8. The method for automatic correction and identification of multiple commodities according to claim 7, characterized in that: The commodity information includes commodity type and commodity weight; the commodity information verification includes commodity type verification and commodity weight verification; The method for verifying the commodity type is as follows: comparing and matching the three-dimensional model of the commodity with the commodity model in a preset three-dimensional commodity database to identify the commodity type; if the identified commodity type is consistent with the commodity type obtained by scanning the label area, the commodity type verification is passed; otherwise, the commodity type verification fails, and a prompt is given to mark the first associated image corresponding to the commodity for which the commodity type verification fails in the commodity front image, and the user is prompted to rearrange the commodity or temporarily move the commodity out of the cashier settlement area; The method for verifying the weight of the goods is as follows: scan all label areas, obtain the weight of the goods, and calculate the total weight of all the goods, which is recorded as the scanned total weight; calculate the total weight of all the goods in the cashier settlement area through a gravity sensor, which is recorded as the weighed total weight; calculate the weight difference between the weighed total weight and the scanned total weight, if the weight difference is less than a preset weight difference threshold, the goods weight verification is passed; otherwise, the goods weight verification fails, and the user is prompted to move all the goods out of the cashier settlement area, and put them one by one into the cashier settlement area for self-service checkout.

9. An intelligent self-service cash register for automatic correction and identification of multiple commodities, used to implement the automatic correction and identification method of multiple commodities according to any one of claims 1 to 8, characterized in that: It includes a cashier counter, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; wherein: The cashier counter includes a cashier settlement area and a gravity sensor. The cashier settlement area is used to place goods, and the gravity sensor is used to detect the weight of goods. The camera unit includes cameras arranged above and above the cashier settlement area, and is used to collect commodity images; The label reading device is used to scan the label area to obtain product information; The display device includes a user interface screen for displaying product information, prices, payment options, and providing operation instructions to the user; The storage unit is used to store the three-dimensional commodity database, user account data, and transaction records; The processing unit is configured with an image processor for aligning the label area with the product area, performing secondary alignment verification on the label area and the associated product area, and for reconstructing a three-dimensional model of the product; the processing unit is also used to verify product information.

Images