Multi-commodity automatic correction and identification method and intelligent self-service cash register

Abstract

The invention relates to the technical field of image recognition, and discloses a multi-commodity automatic correction and recognition method and an intelligent self-service cash register, and the method comprises the steps: extracting a commodity front image from a commodity image; identifying label areas, and selecting associated feature points for each label area; segmenting a commodity area from the front image of the commodity; determining an associated commodity area of each label area; extracting a commodity side image, and determining an associated commodity area of each label area in the commodity side image; reconstructing a three-dimensional model of the commodity; scanning a label area to obtain commodity information, and verifying the commodity information based on a gravity sensor and the three-dimensional model of the commodity; according to the invention, the accuracy of commodity identification and the settlement efficiency are improved, the user experience is improved, the security and reliability of the system are enhanced, and a more efficient, convenient and reliable self-service cashier solution is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for automatically correcting and recognizing multiple commodities and an intelligent self-service cash register. Background Art

[0002] The existing self-service checkout technology has met basic needs to a certain extent, but it still has obvious shortcomings when dealing with the simultaneous settlement of multiple items. Most of the current self-service checkout systems are based on the method of scanning individual items one by one for settlement. Once multiple items are placed in the settlement area at the same time, it is difficult for the system to quickly and accurately identify each item. Since each item needs to be scanned one by one, the settlement process of multiple items often takes a long time, especially during peak hours, which may cause customers to wait in line for a long time. In actual use, the labels of some products are also prone to mislabeling, that is, product A is labeled with the product label of product B, which results in the incorrect settlement of product B during settlement, resulting in user losses and the inability to match product inventory information.

[0003] For example, a Chinese patent application with authorization announcement number CN116308327B discloses a self-service checkout system and method based on RFID technology, which includes an information extraction module, a product verification module, a verification failure module, a verification success module, a product settlement module and a settlement success module. The shopping cart can be used to obtain the product weight, product label and identification information of the product to be settled, and the product weight, product label and identification information are used to verify the product to be settled placed in the shopping cart, and the settlement information of the successfully verified products is stored in the first-level RFID component, and the settlement is performed uniformly in the settlement workshop, thereby splitting the steps of verifying and settling each product one by one into the shopping process, reducing the waiting time in the settlement workshop, and improving the checkout efficiency during self-service checkout, but it does not solve the problem of mislabeling of products.

[0004] For example, the patent application with publication number CN118134474A discloses a self-service checkout method and system, which includes the following steps: based on the self-service checkout area camera, using image capture technology to capture the goods selected by the customer and generate a set of goods images; based on the set of goods images, using a convolutional neural network algorithm to perform product image recognition and generate product list information. In this technical solution, by using image capture technology and convolutional neural network algorithm to identify goods, the efficiency and accuracy of the checkout process are improved. The personalized recommendation list generated by data analysis and collaborative filtering algorithm enhances the customer's shopping experience and provides more accurate product recommendations. Combining symmetric encryption algorithm and multi-factor authentication technology, the security of the payment process is significantly enhanced, the risk of fraud is reduced, and more diversified payment methods are provided, making the entire shopping process more convenient and personalized, and improving the retailer's operating efficiency and customer satisfaction. However, its authentication process is complicated, the processing efficiency is low, and it is difficult to identify multiple goods, resulting in shortcomings.

[0005] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgement or any form of suggestion that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art, provide a method for automatic correction and identification of multiple commodities and an intelligent self-service cash register, improve the accuracy of commodity identification and settlement efficiency, and enhance the security and reliability of the cash register system.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: In one aspect, the present invention provides a method for automatic correction and identification of multiple commodities, comprising the following steps: Collecting commodity images and identification information of commodity radio frequency tags in the intelligent self-service cash register, wherein the commodity images include a plurality of commodity patterns; According to the product image and identification information, combined with the preset image of each product, determine whether the radio frequency tag of each product is bound to the corresponding product; If all are bound, commodity settlement operations are performed based on the identification information of each commodity.

[0008] As a preferred solution of the method for automatic correction and identification of multiple commodities of the present invention, the method of judging whether the radio frequency tag of each commodity is bound to the corresponding commodity according to the commodity image and identification information and the preset image of each commodity includes: Identify label regions from the product front image, number each label region, and select associated feature points for each label region; Segmenting a commodity area from the commodity front image based on each associated feature point; Align the label area with the product area and determine the associated product area of ​​each label area; Extracting a product side image from the product image; marking the position of each label area in the product side image, and determining the associated product area of ​​each label area in the product side image; Reconstructing a three-dimensional model of the product based on the associated product area of ​​each label area in the product front image and the product side image; The label area is scanned to obtain product information, and the product information is verified based on the gravity sensor and the three-dimensional model of the product.

[0009] As a preferred solution of the multi-commodity automatic correction and identification method described in the present invention, wherein: the commodity image is an image of the cashier settlement area captured by a camera, including images captured by cameras arranged directly above and above the cashier settlement area; the commodity front image is extracted based on the commodity image captured directly above the cashier settlement area, and the extraction method is background subtraction.

[0010] As a preferred solution of the multi-product automatic correction and identification method of the present invention, the method of selecting associated feature points for each label area is as follows: calculating the coordinate range of the label area in the product front image; setting a feature area for the label area in the product front image, and uniformly selecting n pixel points in the feature area as associated feature points of the label area, where n is a positive integer; wherein the feature area of ​​the label area is composed of a pixel point that is not less than the distance from the label area. and not greater than All the pixels of is the preset first distance threshold, is a preset second distance threshold.

[0011] As a preferred solution of the method for automatic correction and identification of multiple commodities of the present invention, the steps of segmenting the commodity area from the commodity front image based on each associated feature point include using the associated feature point as a seed point for region growing, and the steps are as follows: S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If so, mark the neighbor pixel as a seed point and add it to the current area; The similarity criterion is specifically as follows: if the straight-line distance between the seed point and the neighboring pixel in the RGB space is less than a preset similarity threshold, the neighboring pixel satisfies the similarity criterion; otherwise, the neighboring pixel does not satisfy the similarity criterion.

[0012] S302: marking the seed point that has completed the similarity criterion check of all neighboring pixels as processed; S303: Repeat S301 to S302, and the seed points marked as processed are no longer involved in the inspection until all seed points are marked as processed and the region growth is stopped; the pixel range of the current region is recorded as the segmented product region.

[0013] As a preferred solution of the multi-commodity automatic correction and identification method of the present invention, the associated commodity area is the commodity area of ​​the commodity corresponding to the label area; the method for determining the associated commodity area of ​​each label area is as follows: Extract h commodity areas segmented based on i associated feature points of the label area; remove duplicates from the h commodity areas and delete completely overlapping commodity areas; if only one commodity area remains after removing duplicates, the remaining commodity area is the associated commodity area of ​​the label area; if m commodity areas remain after removing duplicates, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity area, and the commodity area containing the most associated feature points is the associated commodity area of ​​the label area.

[0014] As a preferred solution of the method for automatic correction and identification of multiple commodities described in the present invention, wherein: the side image of the commodity is extracted based on the commodity image collected on the upper side of the cashier settlement area; the method for marking the position of each label area in the side image of the commodity is as follows: detect and identify each label area in the side image of the commodity, and extract the character information of the label area; based on the character information, find the label area corresponding to each label area in the side image of the commodity in the front image of the commodity; the corresponding label areas in the side image of the commodity and the front image of the commodity have the same character information; obtain the number of each label area and mark the position of each label area in the side image of the commodity.

[0015] As a preferred solution of the multi-product automatic correction and identification method of the present invention, after determining the associated product area of ​​each label area in the product side image, a secondary alignment verification is performed on each label area and the corresponding associated product area, including the following steps: Segment the associated product area of ​​the label area from the product front image, and mark it as the first associated image; segment the associated product area of ​​the label area from the product side image, and mark it as the second associated image; Based on feature point matching, determine whether the first associated image and the second associated image belong to the same group of product images; if so, perform secondary alignment verification; if not, re-determine the associated product area of ​​the label area; belonging to the same group of product images means that the first associated image and the second associated image are images of the same product at different angles.

[0016] As a preferred solution of the method for automatic correction and identification of multiple commodities of the present invention, the method for re-determining the associated commodity area of ​​the label area is as follows: The p product regions after deduplication from the product front image are marked as the front product region; the q product regions after deduplication from the product side image are marked as the side product region; p and q are both positive integers; The front product area and the side product area are grouped based on feature point matching; any group contains a front product area and a corresponding side product area belonging to the same group of product images; the front product area and the side product area that are not grouped are deleted; If there is only one group, the front product area and the side product area in the group are both associated product areas of the label area; if there is more than one group, the total number of associated feature points contained in each group is counted, and the front product area and the side product area in the group containing the most associated feature points are both associated product areas of the label area; the front product area in the associated product area is marked as the first associated image, and the side product area in the associated product area is marked as the second associated image.

[0017] As a preferred solution of the multi-commodity automatic correction and identification method of the present invention, wherein: the commodity information includes commodity type and commodity weight; the verification of commodity information includes commodity type verification and commodity weight verification; The method for verifying the commodity type is as follows: comparing and matching the three-dimensional model of the commodity with the commodity model in a preset three-dimensional commodity database to identify the commodity type; if the identified commodity type is consistent with the commodity type obtained by scanning the label area, the commodity type verification is passed; otherwise, the commodity type verification fails, and a prompt is given to mark the first associated image corresponding to the commodity for which the commodity type verification fails in the commodity front image, and the user is prompted to rearrange the commodity or temporarily move the commodity out of the cashier settlement area; The method for verifying the weight of the goods is as follows: scan all label areas, obtain the weight of the goods, and calculate the total weight of all the goods, which is recorded as the scanned total weight; calculate the total weight of all the goods in the cashier settlement area through a gravity sensor, which is recorded as the weighed total weight; calculate the weight difference between the weighed total weight and the scanned total weight, if the weight difference is less than a preset weight difference threshold, the goods weight verification is passed; otherwise, the goods weight verification fails, and the user is prompted to move all the goods out of the cashier settlement area, and put them one by one into the cashier settlement area for self-service checkout.

[0018] In a second aspect, the present invention provides an intelligent self-service cash register for automatic correction and identification of multiple commodities, including a cash register, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; wherein: The cashier counter includes a cashier settlement area and a gravity sensor. The cashier settlement area is used to place goods, and the gravity sensor is used to detect the weight of goods. The camera unit includes cameras arranged above and above the cashier settlement area, and is used to collect commodity images; The label reading device is used to scan the label area to obtain product information; The display device includes a user interface screen for displaying product information, prices, payment options, and providing operation instructions to the user; The storage unit is used to store the three-dimensional commodity database, user account data, and transaction records; The processing unit is configured with an image processor for aligning the label area with the product area, performing secondary alignment verification on the label area and the associated product area, and for reconstructing a three-dimensional model of the product; the processing unit is also used to verify product information.

[0019] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: The present invention can improve the accuracy and efficiency of commodity recognition by introducing advanced image processing technology and three-dimensional reconstruction algorithm. By accurately numbering the label area and selecting the associated feature points, the accurate match between each label and its corresponding commodity area is ensured. Based on the region growing algorithm, the commodity area can be reliably segmented, and a high recognition accuracy can be maintained even when the commodities are mutually blocked.

[0020] The present invention can efficiently identify multiple commodities in one operation. On the one hand, it combines radio frequency identification for rapid identification, and on the other hand, it combines image processing. The image processing process is based on radio frequency identification, which greatly reduces the recognition processing volume and improves efficiency. In addition, the image is further used as a calibration scheme for radio frequency information. The label and the commodity image can be aligned one by one to ensure that all commodities can be correctly identified and settled, reducing the risk of missed scanning, wrong scanning or wrong labeling of commodities.

[0021] The product weight verification function has been introduced. The total weight of the product is measured by a gravity sensor and compared with the weight data in the label information, effectively preventing product swapping or replacement. The three-dimensional model is compared with the product model in the preset database to further enhance the reliability of product type verification and reduce recognition errors caused by packaging changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them: Figure 1 A flow chart of the multi-commodity automatic correction and identification method provided by the present invention; Figure 2 A flow chart of a method for segmenting a commodity area from a commodity front image provided by the present invention; Figure 3 A schematic diagram of the structure of an intelligent self-service cash register with automatic correction and identification of multiple commodities provided by the present invention.

[0023] Figure numerals: 1. cashier counter; 101. cashier settlement area; 2. label reading device; 301. first camera; 302. second camera; 4. display device. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0025] Example 1 This embodiment introduces a method for automatic correction and identification of multiple commodities, including the following steps: Collecting commodity images and identification information of commodity radio frequency tags in the intelligent self-service cash register, wherein the commodity images include a plurality of commodity patterns; According to the product image and identification information, combined with the preset image of each product, determine whether the radio frequency tag of each product is bound to the corresponding product; If all are bound, commodity settlement operations are performed based on the identification information of each commodity.

[0026] The present invention can efficiently identify multiple commodities in one operation. On the one hand, it combines radio frequency identification for rapid identification, and on the other hand, it combines image processing. The image processing process is based on radio frequency identification, which greatly reduces the recognition processing volume and improves efficiency. In addition, the image is further used as a calibration scheme for radio frequency information. The label and the commodity image can be aligned one by one to ensure that all commodities can be correctly identified and settled, reducing the risk of missed scanning, wrong scanning or wrong labeling of commodities.

[0027] Specifically, in the embodiment of the present application, the product image and radio frequency tag information are collected. When the customer puts the product into the scanning area of ​​the intelligent self-service cash register, the machine will automatically collect the product image and radio frequency tag information. The product image is taken by a camera and contains patterns of multiple products; the radio frequency tag information is obtained through an RFID reader, and each product has a unique radio frequency tag. When judging whether the radio frequency tag is bound to the product, the system will make a matching judgment based on the collected product image and radio frequency tag information, combined with the preset images of each product.

[0028] It should be noted that the preset images are stored in the system in advance and are used to identify different types of goods. By comparing the collected product images with the preset images, the system can determine whether the RFID tag of each product is correctly bound to its corresponding product. When performing the product settlement operation, if the RFID tags of all products are correctly bound to their corresponding products, the system will perform the product settlement operation based on the identification information of each product, that is, the system will calculate the total amount that the customer needs to pay based on the identified product information and display it on the screen for the customer to confirm.

[0029] The core idea of ​​this application is to be able to check whether there is a label error when the user self-checks out, that is, the label originally for product A is placed on product B, resulting in incorrect product information, thereby avoiding unnecessary losses.

[0030] The details of the core concept of this application are described in detail below.

[0031] This embodiment introduces a method for automatic correction and identification of multiple commodities. Figure 1 , the method comprises the following steps: S1: Collect product images and extract product front images from the product images; The product image is an image of the cashier settlement area captured by a camera, including images captured by cameras set just above and above the cashier settlement area; the product front image is extracted based on the product image captured just above the cashier settlement area, and the extraction method is background subtraction. The steps are as follows: first, when there is no product placed in the cashier settlement area, a background image is taken by the camera just above, which serves as the basis for subsequent background subtraction; when the product is placed in the cashier settlement area, the product image is captured; Use background subtraction algorithms (such as frame difference method, mixed Gaussian model, etc.) to subtract the background reference image from the product image to separate the product front image. By separating the product image from the complex background through background subtraction, the efficiency and accuracy of subsequent target detection can be improved, and recognition errors can be reduced when there are multiple items mixed in the checkout settlement area.

[0032] S2: Identify the label area from the product front image, number each label area, and select associated feature points for each label area; The label area is the image area corresponding to the label of the product; the character information of the label area is extracted, and a number is assigned to each label area; the extracted character information is used to identify the label area. The product label, such as a barcode, QR code, RFID tag, etc., contains the product information necessary for cashier settlement; The method for selecting associated feature points for each label region is as follows: calculating the coordinate range of the label region in the product front image; setting a feature region for the label region in the product front image, and uniformly selecting n pixels in the feature region as associated feature points of the label region, where n is a positive integer; wherein the feature region of the label region is determined by the distance from the label region being not less than and not greater than All the pixels of is the preset first distance threshold, is a preset second distance threshold.

[0033] S3: Segmenting a commodity area from the commodity front image based on each associated feature point; The method of segmenting the commodity area from the commodity front image based on each associated feature point includes using the associated feature point as a seed point for region growth, referring to Figure 2 , the steps are as follows: S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If so, mark the neighbor pixel as a seed point and add it to the current area; The similarity criterion is specifically as follows: if the straight-line distance between the seed point and the neighboring pixel in the RGB space is less than a preset similarity threshold, the neighboring pixel satisfies the similarity criterion; otherwise, the neighboring pixel does not satisfy the similarity criterion.

[0034] S302: marking the seed point that has completed the similarity criterion check of all neighboring pixels as processed; S303: Repeat S301 to S302, and the seed points marked as processed are no longer involved in the inspection until all seed points are marked as processed and the region growth is stopped; the pixel range of the current region is recorded as the segmented product region.

[0035] S4: aligning the label area with the product area to determine the associated product area of ​​each label area; The associated product area is the product area of ​​the product corresponding to the label area, that is, the label in the label area is the label of the corresponding product. The method for determining the associated product area of ​​each label area is as follows: Extract h commodity areas segmented based on i associated feature points of the label area; remove duplicates from the h commodity areas and delete completely overlapping commodity areas; if only one commodity area remains after removing duplicates, the remaining commodity area is the associated commodity area of ​​the label area; if m commodity areas remain after removing duplicates, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity area, and the commodity area containing the most associated feature points is the associated commodity area of ​​the label area.

[0036] Based on region growing, h product regions can be obtained by taking i associated feature points as seed points. Ideally, the pixels contained in the h product regions completely overlap, that is, the h product regions are actually the same product region. In this case, the product represented by the product region is the product corresponding to the label. However, due to the existence of label angle deviation, mutual occlusion between products, etc., some associated feature points may be located on other products. In this way, the product region segmented based on the associated feature points is not the product region corresponding to the label. Based on the number of associated feature points, the product region that is incorrectly segmented due to improper location of the associated feature points can be eliminated.

[0037] S5: extracting a product side image from the product image; marking the position of each label area in the product side image, and determining an associated product area of ​​each label area in the product side image; The side image of the product is extracted based on the product image collected on the upper side of the cashier settlement area; the method for marking the position of each label area in the side image of the product is as follows: detect and identify each label area in the side image of the product, and extract the character information of the label area; find the label area corresponding to each label area in the side image of the product in the front image of the product based on the character information; the corresponding label areas in the side image of the product and the front image of the product have the same character information; obtain the number of each label area and mark the position of each label area in the side image of the product. Through the character information, it can be determined which label areas in the side image of the product and the front image of the product are the same label area, thereby achieving a one-to-one correspondence between the label areas in the side image of the product and the label areas in the front image of the product.

[0038] The method for determining the associated product area of ​​each label area in the product side image is as follows: selecting associated feature points for each label area in the product side image; segmenting the product area from the product side image based on each associated feature point; deduplicating the segmented product areas, and determining the associated product area of ​​the label area from the deduplicated product areas. The specific method details refer to the method for determining the associated product area of ​​each label area in the product front image, and will not be repeated here.

[0039] After determining the associated product area of ​​each label area in the product side image, a secondary alignment verification is performed on each label area and the corresponding associated product area, including the following steps: Segment the associated product area of ​​the label area from the product front image, and mark it as the first associated image; segment the associated product area of ​​the label area from the product side image, and mark it as the second associated image; Based on feature point matching, determine whether the first associated image and the second associated image belong to the same group of product images; if so, perform secondary alignment verification; if not, re-determine the associated product area of ​​the label area; belonging to the same group of product images means that the first associated image and the second associated image are images of the same product at different angles; through the method of feature point detection and matching, it can be determined whether the two images are images of the same product, for example, by detecting feature points, matching feature points, and counting the number of successfully matched feature points, it can be determined whether the first associated image and the second associated image belong to the same group of product images.

[0040] The method for re-determining the associated product area of ​​the label area is as follows: The p product regions after deduplication from the product front image are marked as the front product region; the q product regions after deduplication from the product side image are marked as the side product region; p and q are both positive integers; The front product area and the side product area are grouped based on feature point matching; any group contains a front product area and the corresponding side product area belonging to the same group of product images; the front product area and the side product area that are not grouped are deleted; each product area corresponds to a product, and based on the feature point matching algorithm, product areas that are not matched to other angles will not be grouped. Not being grouped means that the label is not bound to the product area in both the front product image and the side product image, indicating that the label is not the label of the product corresponding to the product area.

[0041] If there is only one group, the front product area and the side product area in the group are both associated product areas of the label area; if there is more than one group, the total number of associated feature points contained in each group is counted, and the front product area and the side product area in the group containing the most associated feature points are both associated product areas of the label area; the front product area in the associated product area is marked as the first associated image, and the side product area in the associated product area is marked as the second associated image.

[0042] S6: reconstructing a three-dimensional model of the product based on the associated product area of ​​each label area in the product front image and the product side image; For any label area, three-dimensional reconstruction is performed through its first associated image and second associated image to obtain a three-dimensional model of the product corresponding to the label area. By executing S1 to S6, the labels of multiple products in the cashier settlement area are aligned with the product images, and the three-dimensional model of the product corresponding to the label is obtained, providing a rigorous data basis for the verification of product information.

[0043] S7: Scan the label area to obtain product information, and verify the product information based on the gravity sensor and the three-dimensional model of the product.

[0044] The commodity information includes commodity type and commodity weight; the commodity information verification includes commodity type verification and commodity weight verification; The method for verifying the commodity type is as follows: comparing and matching the three-dimensional model of the commodity with the commodity model in a preset three-dimensional commodity database to identify the commodity type; if the identified commodity type is consistent with the commodity type obtained by scanning the label area, the commodity type verification is passed; otherwise, the commodity type verification fails, and a prompt is given to mark the first associated image corresponding to the commodity for which the commodity type verification fails in the commodity front image, and the user is prompted to rearrange the commodity or temporarily move the commodity out of the cashier settlement area; Through 3D database matching, the real shape of the product can be effectively identified, and even when the product is partially blocked or the label is misplaced, it can be accurately identified, reducing package switching and cashier errors. Preferably, multiple cameras can be set to collect side images of the product from multiple angles, and then the third associated image, fourth associated image, ..., Nth associated image of any label area can be obtained from multiple side images of the product, and the three-dimensional reconstruction of the product is performed based on all the associated images, thereby improving the accuracy of the three-dimensional reconstruction of the product.

[0045] The method of commodity weight verification is as follows: scan all label areas, obtain commodity weight, and calculate the total weight of all commodities, which is recorded as the scanned total weight; calculate the total weight of all commodities in the cashier settlement area through a gravity sensor, which is recorded as the weighed total weight; calculate the weight difference between the weighed total weight and the scanned total weight, and if the weight difference is less than a preset weight difference threshold, the commodity weight verification is passed; otherwise, the commodity weight verification fails, and the user is prompted to move all commodities out of the cashier settlement area and put them one by one into the cashier settlement area for self-service checkout. By weighing, identifying, and scanning the cashier one by one, the mis-weighed commodities can be found.

[0046] Example 2 This embodiment is the second embodiment of the present invention; based on the same inventive concept as the first embodiment, this embodiment introduces an intelligent self-service cash register for automatic correction and identification of multiple commodities, including a cash register, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; wherein: The cashier counter includes a cashier settlement area and a gravity sensor. The cashier settlement area is used to place goods, and the gravity sensor is used to detect the weight of goods. The camera unit includes cameras arranged above and above the cashier settlement area, which are used to collect product images; cameras based on different angles can collect product images at different angles, which is helpful for image verification, recognition and three-dimensional reconstruction of the products.

[0047] The label reading device is used to scan the label area to obtain product information; the obtained product information includes product weight, product type, and product price; the label reading device includes a barcode scanner, a QR code reader, an RFID reader, etc., which is used to read the information contained in the label on the product.

[0048] The display device includes a user interface screen for displaying product information, prices, payment options, and providing operating guides to users; the display device also includes an input interface that supports various payment methods (such as credit cards, mobile payments), and an output interface for printing shopping receipts, supporting multiple payment methods to complete transactions and automatically generating shopping receipts.

[0049] The storage unit is used to save a three-dimensional commodity database, user account data, and transaction records; wherein the three-dimensional commodity database contains a commodity model for each commodity, and the type of commodity on the checkout counter can be identified by matching the three-dimensional model of the commodity on the checkout counter with the commodity model in the commodity database.

[0050] The processing unit is equipped with an image processor, which is used to align the label area with the product area, perform secondary alignment verification on the label area and the associated product area, and reconstruct the three-dimensional model of the product; the processing unit is also used to verify the product information. The image processor is equipped with a variety of image processing algorithms, including background subtraction, feature point extraction, region growing, graphic segmentation, three-dimensional reconstruction, etc.

[0051] The specific functions of the above-mentioned units are realized by referring to the relevant contents of the multi-commodity automatic correction and identification method described in Example 1, which will not be described in detail.

[0052] Example 3 Based on the same inventive concept as other embodiments, refer to Figure 3 , this embodiment provides an application example of an intelligent self-service cash register with automatic correction and identification of multiple commodities, which is as follows: the intelligent self-service cash register includes a cash register 1; the cash register 1 includes a cash register settlement area 101 for placing commodities; a gravity sensor is provided under the cash register settlement area 101 for monitoring the weight of commodities; The intelligent self-service cash register also includes a label reading device 2, which is fixed to the cash register 1 through a bracket and can read the radio frequency labels of multiple commodities placed in the cash register settlement area 101 respectively to obtain commodity information of the multiple commodities; The intelligent self-service cash register further includes a first camera 301 and a second camera 302, wherein the first camera 301 is fixed on the cash register 1 through a bracket and is located directly above the cash register settlement area 101, and is used to collect commodity settlement images of multiple commodities from directly above; the second camera 302 is fixed on the cash register 1 through a bracket and is located above the side of the cash register settlement area 101, and is used to collect commodity settlement images of multiple commodities from above the side; The intelligent self-service cash register also includes a display device 4; the display device 4 is fixed to the cash register 1 through a bracket, and is used to list and display product information and prices of multiple products to customers, and provide payment options and input interfaces for multiple payment methods.

[0053] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the purpose and scope of protection of the present invention, which are all within the protection of the present invention.

Claims

1. A multi-commodity automatic correction and identification method, characterized in that: The following steps are involved: Collecting commodity images and identification information of commodity radio frequency tags in the intelligent self-service cash register, wherein the commodity images include a plurality of commodity patterns; According to the product image and identification information, combined with the preset image of each product, determine whether the radio frequency tag of each product is bound to the corresponding product; If all are bound, commodity settlement operations are performed based on the identification information of each commodity; According to the product image and identification information, combined with the preset image of each product, it is determined whether the radio frequency tag of each product is bound to the corresponding product, specifically including: Identify label regions from the product front image, number each label region, and select associated feature points for each label region; Segmenting a commodity area from the commodity front image based on each associated feature point; Align the label area with the product area and determine the associated product area of ​​each label area; Extracting a product side image from the product image; marking the position of each label area in the product side image, and determining the associated product area of ​​each label area in the product side image; Reconstructing a three-dimensional model of the product based on the associated product area of ​​each label area in the product front image and the product side image; The label area is scanned to obtain product information, and the product information is verified based on the gravity sensor and the three-dimensional model of the product.

2. The method for automatic correction and identification of multiple commodities according to claim 1, characterized in that: The product image is an image of the cashier settlement area captured by a camera, including images captured by cameras arranged directly above and above the cashier settlement area; the product front image is extracted based on the product image captured directly above the cashier settlement area, and the extraction method is background subtraction.

3. The multi-commodity automatic correction and identification method according to claim 2, characterized in that: The method for selecting associated feature points for each label area is as follows: calculating the coordinate range of the label area in the product front image; setting a feature area for the label area in the product front image, and uniformly selecting n pixels in the feature area as associated feature points of the label area, where n is a positive integer; wherein the feature area of ​​the label area is determined by the distance from the label area to the label area being not less than and not greater than All the pixels of is the preset first distance threshold, is a preset second distance threshold.

4. The multi-commodity automatic correction and identification method according to claim 3, characterized in that: The steps of segmenting the commodity area from the commodity front image based on each associated feature point include using the associated feature point as a seed point for region growing, and the steps are as follows: S301: Check whether each neighbor pixel of the seed point meets the similarity criterion. If so, mark the neighbor pixel as a seed point and add it to the current area; The similarity criterion is specifically as follows: if the straight-line distance between the seed point and the neighboring pixel in the RGB space is less than a preset similarity threshold, the neighboring pixel satisfies the similarity criterion; otherwise, the neighboring pixel does not satisfy the similarity criterion; S302: marking the seed point that has completed the similarity criterion check of all neighboring pixels as processed; S303: Repeat S301 to S302, and the seed points marked as processed are no longer involved in the inspection until all seed points are marked as processed and the region growth is stopped; the pixel range of the current region is recorded as the segmented product region.

5. The method for automatic correction and identification of multiple commodities according to claim 4, characterized in that: The associated product area is the product area of ​​the product corresponding to the label area; the method for determining the associated product area of ​​each label area is as follows: Extract h commodity areas segmented based on i associated feature points of the label area; remove duplicates from the h commodity areas and delete completely overlapping commodity areas; if only one commodity area remains after removing duplicates, the remaining commodity area is the associated commodity area of ​​the label area; if m commodity areas remain after removing duplicates, where m is a positive integer greater than 1, count the number of associated feature points contained in each commodity area, and the commodity area containing the most associated feature points is the associated commodity area of ​​the label area.

6. The multi-commodity automatic correction and identification method according to claim 5, characterized in that: The side image of the product is extracted based on the product image collected on the upper side of the cashier settlement area; the method for marking the position of each label area in the side image of the product is as follows: detect and identify each label area in the side image of the product, and extract the character information of the label area; based on the character information, find the label area corresponding to each label area in the side image of the product in the front image of the product; the corresponding label areas in the side image of the product and the front image of the product have the same character information; obtain the number of each label area and mark the position of each label area in the side image of the product.

7. The method for automatic correction and identification of multiple commodities according to claim 6, characterized in that: After determining the associated product area of ​​each label area in the product side image, a secondary alignment verification is performed on each label area and the corresponding associated product area, including the following steps: Segment the associated product area of ​​the label area from the product front image, and mark it as the first associated image; segment the associated product area of ​​the label area from the product side image, and mark it as the second associated image; Based on feature point matching, determine whether the first associated image and the second associated image belong to the same group of product images; if so, perform secondary alignment verification; if not, re-determine the associated product area of ​​the label area; belonging to the same group of product images means that the first associated image and the second associated image are images of the same product at different angles.

8. The multi-commodity automatic correction and identification method according to claim 7, characterized in that: The method for re-determining the associated product area of ​​the label area is as follows: The p product regions after deduplication is segmented from the product front image are all marked as the front product region; the q product regions after deduplication is segmented from the product side image are all marked as the side product region; p, q are both positive integers; The front product area and the side product area are grouped based on feature point matching; any group contains a front product area and a corresponding side product area belonging to the same group of product images; the front product area and the side product area that are not grouped are deleted; If there is only one group, the front product area and the side product area in the group are both associated product areas of the label area; if there is more than one group, the total number of associated feature points contained in each group is counted, and the front product area and the side product area in the group containing the most associated feature points are both associated product areas of the label area; the front product area in the associated product area is marked as the first associated image, and the side product area in the associated product area is marked as the second associated image.

9. The multi-commodity automatic correction and identification method according to claim 8, characterized in that: The commodity information includes commodity type and commodity weight; the commodity information verification includes commodity type verification and commodity weight verification; The method for verifying the commodity type is as follows: comparing and matching the three-dimensional model of the commodity with the commodity model in a preset three-dimensional commodity database to identify the commodity type; if the identified commodity type is consistent with the commodity type obtained by scanning the label area, the commodity type verification is passed; otherwise, the commodity type verification fails, and a prompt is given to mark the first associated image corresponding to the commodity for which the commodity type verification fails in the commodity front image, and the user is prompted to rearrange the commodity or temporarily move the commodity out of the cashier settlement area; The method for verifying the weight of the goods is as follows: scan all label areas, obtain the weight of the goods, and calculate the total weight of all the goods, which is recorded as the scanned total weight; calculate the total weight of all the goods in the cashier settlement area through a gravity sensor, which is recorded as the weighed total weight; calculate the weight difference between the weighed total weight and the scanned total weight, if the weight difference is less than a preset weight difference threshold, the goods weight verification is passed; otherwise, the goods weight verification fails, and the user is prompted to move all the goods out of the cashier settlement area, and put them one by one into the cashier settlement area for self-service checkout.

10. An intelligent self-service cash register for automatic correction and identification of multiple commodities, used to implement the automatic correction and identification method for multiple commodities according to any one of claims 1 to 9, characterized in that: It includes a cashier counter, a camera unit, a label reading device, a display device, a processing unit, and a storage unit; wherein: The cashier counter includes a cashier settlement area and a gravity sensor. The cashier settlement area is used to place goods, and the gravity sensor is used to detect the weight of goods. The camera unit includes cameras arranged above and above the cashier settlement area, and is used to collect commodity images; The label reading device is used to scan the label area to obtain product information; The display device includes a user interface screen for displaying product information, prices, payment options, and providing operation instructions to the user; The storage unit is used to store the three-dimensional commodity database, user account data, and transaction records; The processing unit is configured with an image processor for aligning the label area with the product area, performing secondary alignment verification on the label area and the associated product area, and for reconstructing a three-dimensional model of the product; the processing unit is also used to verify product information.

Images