Self-service cashier method and system based on multi-modal interaction and storage medium

Abstract

The invention discloses a self-service cashier method and system based on multi-modal interaction and a storage medium, and belongs to the technical field of data processing, and the method comprises the steps: shooting an initial image of a predetermined region, splitting the initial image into a plurality of commodity images, and recognizing each commodity image to obtain commodity information and first confidence in the commodity image; generating a first commodity list based on the commodity information of which the first confidence is greater than a first threshold value, and performing joint reasoning based on all commodity images and weight information to obtain commodity information of unidentified commodities and second confidence; if the second confidence coefficient is smaller than the first threshold value, the commodity information is verified based on the picking and placing information of the selling shelf; and if the verification is successful, adding the commodity information of the unidentified commodities into the first commodity list to obtain a second commodity list, generating amount information based on the second commodity list, and displaying a collection interface. According to the invention, the to-be-settled commodity information is obtained in combination with multiple methods, so that the commodity identification accuracy is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a self-service checkout method, system and storage medium based on multimodal interaction. Background Art

[0002] Self-service checkout systems have become core facilities for improving operational efficiency in supermarkets, convenience stores and other scenarios. Self-service checkout currently relies mainly on barcode recognition. However, this process requires users to find the barcode on the product packaging by themselves. When the product packaging is more complicated or the checkout staff is unskilled, the settlement time may be longer, affecting the settlement efficiency.

[0003] To solve this problem, the following methods have been proposed in the prior art. For example, a Chinese patent document with publication number CN109214806A discloses a self-service settlement method, device and storage medium. The method uses a variety of image recognition technologies to identify monitoring images, identify the category and quantity of goods to be settled from the monitoring images, and then obtain the actual weight of the goods to be settled in combination with a weight sensor. The standard weight of the goods is calculated based on the recognition result, and the actual weight is compared with the standard weight to verify the accuracy of the recognition result. The method can complete the settlement identification of the goods without scanning the barcode, thereby greatly improving the settlement efficiency.

[0004] However, if the product images in the settlement area are only identified in the above manner, there may be a low product identification accuracy rate. Although the above technical solution introduces the method of obtaining the weight of the product to verify the identification result, this method can only determine whether the identification result is correct. If it is incorrect, it still needs manual confirmation, and it is impossible to improve the recognition accuracy from the source. Summary of the invention

[0005] To solve the above problems, the present invention provides a self-service checkout method, device and storage medium based on multimodal interaction to solve the problems existing in the prior art.

[0006] In order to achieve the above-mentioned invention object, the present invention proposes a self-service checkout method based on multimodal interaction, comprising: After receiving the start instruction, an initial image of a predetermined area is captured, the initial image is split into a plurality of product images, and each of the product images is recognized based on a deep recognition model to obtain product information and a first confidence level included therein; generating a first commodity list based on the commodity information whose first confidence level is greater than a first threshold, and counting a first number and a second number of recognized commodities and unrecognized commodities in the initial image; If the first quantity and the second quantity meet a preset condition, obtaining weight information of the predetermined area, performing joint reasoning based on all the commodity images and the weight information, and obtaining the commodity information and a second confidence level of the unrecognized commodity; If the second confidence level is less than the first threshold, locating a corresponding sales shelf based on the product information of the unidentified product, and verifying the product information based on the pick-and-place information of the sales shelf; If the verification is successful, the product information of the unidentified product is added to the first product list to obtain the second product list; if the verification fails, a manual entry reminder is generated; After receiving the completion information, the amount information is generated based on the second commodity list, and the payment interface is displayed until the settlement completion information or the payment cancellation information is received.

[0007] Furthermore, identifying the product image includes the following steps: If a barcode area is detected in the product image, the product information is generated based on the barcode, and the corresponding first confidence is set to 100%; if the barcode area is not identified, the product image is identified based on the deep recognition model to obtain the product information and the first confidence; the product type is determined based on the product information; if the product type belongs to a preset type and the first confidence is greater than the first threshold, the product quality continues to be identified; if damaged products are identified, a product quality reminder is generated.

[0008] Furthermore, identifying the quality of the product includes the following steps: The product image is divided into grids to obtain a plurality of grid areas, an original average pixel value of each of the grid areas is obtained, a plurality of interval ranges are set, each of the interval ranges corresponds to a mapping pixel, and based on the interval range where the original average pixel value is located, it is mapped to the corresponding mapping pixel; The grid area is divided into multiple types of sub-areas, and statistical features of the sub-areas are obtained, wherein the statistical features include the mapped pixels existing in the sub-areas and their occupation ratios; a standard feature library is established, wherein the standard feature library includes standard features of each sub-area of ​​each product type in a damaged state; the statistical features of the sub-areas in the product image are compared with the standard features, and the quality of the product is determined based on the comparison results.

[0009] Furthermore, verifying the commodity information based on the pick-and-place information of the selling shelf includes the following steps: Acquire a shelf image of the selling shelf, and when a product is removed from the shelf in the shelf image, determine the product information of the removed product based on the shelf data, track the user who removed the product to obtain a user movement image, capture a facial image and a body image in the user movement image for pre-storage, and mark the product information of the selling shelf in the facial image and the body image as an image tag; when a verification requirement arises, perform a search based on the product information of the unrecognized product and the image tag; if the facial image and the body image containing the product information of the unrecognized product are retrieved, compare them with the current user image; if the comparison is successful, determine that the currently recognized product information is correct.

[0010] Furthermore, performing joint reasoning based on all the product images and the weight information includes the following steps: Establishing a weight database, the weight database including a standard weight and a floating range of each commodity, setting an error distribution of a weight sensor, obtaining a first weight of an identified commodity based on the weight database, calculating a second weight of an unidentified commodity based on a total weight of commodities in the predetermined area and the first weight, and correcting the second weight based on the floating range and the error distribution to obtain a weight distribution range thereof; Acquire candidate information, where the candidate information is the commodity information whose first confidence is greater than a second threshold in a single commodity image recognition result, acquire the standard weight and the floating range of the candidate information based on the weight database, generate a candidate combination that satisfies the weight distribution range based on the standard weight and the floating range, and calculate a combination score for each candidate combination based on the weight distribution range of the candidate combination; The candidate score of each candidate combination is calculated based on the first confidence and the combination score, and the candidate combination corresponding to the maximum candidate score is screened as the target combination, wherein the candidate information included therein is used as the inference result of the unidentified commodity, and the corresponding candidate score is used as the second confidence.

[0011] Further, calculating the candidate score of the candidate combination includes the following steps: The purchase probability of each type of candidate information is obtained based on historical purchase records, and the first confidence, the combination score and the purchase probability are weighted and summed based on preset weights to obtain a joint score of the candidate combination, and the joint score is normalized to obtain a normalized score, a penalty value is generated based on the number of candidate combinations currently existing, and all the normalized scores are corrected based on the penalty value to obtain the candidate score.

[0012] Furthermore, the preset condition includes that the first number and the second number are both smaller than a third threshold.

[0013] Furthermore, if it is detected in the shelf image that the user moves the product back to the shelf, or the user leaves the shopping mall area, the facial image and the body image of the user are deleted.

[0014] The present invention also provides a self-service cashier system based on multimodal interaction, which is used to implement the self-service cashier method based on multimodal interaction described above, and the system includes: The recognition module, after receiving a start instruction, captures an initial image of a predetermined area, splits the initial image into multiple product images, recognizes each of the product images based on a deep recognition model to obtain the product information and a first confidence level included therein, and generates a first product list based on the product information whose first confidence level is greater than a first threshold.

[0015] The first verification module counts the first number and the second number of identified commodities and unidentified commodities in the initial image, obtains the weight information of the predetermined area if the first number and the second number meet the preset conditions, performs joint reasoning based on all the commodity images and the weight information, and obtains the commodity information and the second confidence level of the unidentified commodities.

[0016] The second verification module locates the corresponding sales shelf based on the product information of the unidentified product if the second confidence level is less than the first threshold value, and verifies the product information based on the pick-and-place information of the sales shelf. If the verification is successful, the product information of the unidentified product is added to the first product list to obtain the second product list. If the verification fails, a manual entry reminder is generated.

[0017] The settlement module, after receiving the completion information, generates amount information based on the second commodity list and displays a payment collection interface until receiving the completion information or the payment cancellation information.

[0018] The present invention also discloses a computer storage medium, wherein the computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned method.

[0019] Beneficial effects: After receiving the start instruction, the present invention takes an initial image of a predetermined area, splits it into multiple product images based on an edge detection algorithm, and then uses a depth recognition model to perform recognition, thereby quickly and accurately obtaining product information in each product image. This method effectively solves the problem of low efficiency in manually scanning product barcodes one by one, and improves cashier efficiency.

[0020] By counting the number of identified and unidentified goods in the initial image, the system can clearly sort out the list of goods to be settled after the preliminary identification of the goods. If the number of identified and unidentified goods meets the preset conditions, the weight information of the predetermined area continues to be obtained, and joint reasoning is performed based on the weight information to obtain the product information of the unidentified goods. By adding weight information for joint reasoning, the accuracy and comprehensiveness of product identification are improved. The present invention also locates the sales shelf based on the product information of the unidentified goods, and verifies the product information according to the shelf's pick-and-place information. The correctness of the inferred product information is ensured by verification, which solves the problem that the product information is still inaccurate after joint reasoning, or the problem that joint reasoning cannot be performed; in summary, the present invention combines multiple methods to obtain the product information to be settled, thereby greatly improving the accuracy of product identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of the steps of the self-service checkout method based on multimodal interaction of the present invention; Figure 2 It is a structural schematic diagram of the self-service cashier device of the present invention; Figure 3 This is a schematic diagram of the principle of dividing commodity images in the present invention; Figure 4 It is a structural schematic diagram of the self-service checkout system based on multimodal interaction of the present invention.

[0022] In the figure, 1, placement area; 2, display area; 3, wireless transmission module; 4, code scanning area. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] It should be noted that all acquisition and processing of information such as images and data in the present invention are carried out in compliance with relevant laws, regulations and policies.

[0025] like Figure 1 As shown, the present invention provides a self-service checkout method based on multimodal interaction, comprising: S1: After receiving a start instruction, an initial image of a predetermined area is captured, the initial image is split into multiple product images, and each product image is identified based on a deep recognition model to obtain product information and a first confidence level included therein.

[0026] like Figure 2The self-service cash register device for implementing the present invention is shown, which includes a placement area 1 and a display area 2. The placement area 1 is used to place goods. A weight sensor is provided in the placement area 1. The display area 2 is used to display a list of goods and amount information. The device also includes a wireless transmission module 3 for sending and receiving data of the self-service cash register device. When the method based on the present invention cannot automatically identify the goods, the barcode scanning area 4 can also be used to identify the goods based on the barcode. After the self-service cash register device receives the start instruction, the start instruction is input by clicking on the screen, and the camera is called to shoot the initial image of the predetermined area, that is, the placement area 1. A fixed camera can be arranged above the cash register device to shoot. In particular, when placing items, the device reminds the user through voice, and the reminder information includes the following content: "Do not stack the goods when placing", "Please place and identify fresh fruit products separately".

[0027] After the shooting is completed, the initial image is obtained, and the object contours in the initial image are detected based on the Canny edge detection algorithm. Then, according to the object contours, the initial image is cropped into multiple product images containing only one object contour. Then, each product image is identified using a CNN-based deep recognition model. The deep recognition model outputs the product information included in the product image and the corresponding first confidence. For example, if the deep recognition model outputs the product information as brand A potato chips, the first confidence is 98%. The higher the first confidence, the greater the possibility that the deep recognition model believes that the product image includes brand A potato chips. In particular, if the image includes a whole bag of fruit, each contour identified therein is split into a product image based on the contour edge. If all product images are identified as the same kind of fruit, the product price is calculated based on the unit price and weight of the fruit.

[0028] S2: Generate a first commodity list based on commodity information with a first confidence level greater than a first threshold, and count a first number and a second number of recognized commodities and unrecognized commodities in the initial image.

[0029] S3: If the first quantity and the second quantity meet the preset conditions, obtain the weight information of the predetermined area, perform joint reasoning based on all product images and weight information, and obtain the product information and the second confidence level of the unidentified product.

[0030] In this embodiment, the preset condition includes that both the first number and the second number are smaller than a third threshold.

[0031] Here, the first threshold is set to 95%. If the first confidence level of the product information is greater than the first threshold, the judgment result is directly considered correct and added to the first product list. At the same time, the corresponding selling price is obtained from the price database. For fruits, the unit price of the fruit is multiplied by the weight to obtain the selling price. The first product list is a list of products to be settled. Then, the first quantity and the second quantity in the initial image are counted. If multiple products are placed in the predetermined area at the same time, 5 of them are identified and added to the first product list, and 3 are not identified, then the first quantity is 5 and the second quantity is 4. The third threshold is set to 3. When the number of identified products and unidentified products exceeds 3, the error of joint reasoning by weight will increase greatly. In this case, the subsequent steps are used to determine the product information.

[0032] When it is determined that joint reasoning can be performed, combined reasoning is performed based on the weight information of the identified goods and the unidentified goods. The specific reasoning method will be introduced later. After the reasoning is completed, the product information and the second confidence level of the unidentified goods can be obtained.

[0033] S4: If the second confidence level is less than the first threshold, locate the corresponding sales shelf based on the product information of the unidentified product, and verify the product information based on the pick-and-place information of the sales shelf.

[0034] S5: If the verification is successful, the product information of the unidentified product is added to the first product list to obtain the second product list. If the verification fails, a manual entry reminder is generated.

[0035] S6: After receiving the completion information, generate amount information based on the second commodity list and display the payment interface until receiving the completion settlement information or the payment cancellation information.

[0036] If the second confidence level is still less than the first threshold, or joint reasoning cannot be performed, it indicates that the product information still cannot be accurately determined. At this time, based on the currently inferred product information, obtain the shelf surveillance image of the product, and determine whether the user currently checking out has purchased the inferred product based on the surveillance image, that is, verify the product information. If the verification passes, add the inferred product information to the first product list and update it to the second product list. If the verification fails, it is recommended that the user manually scan the barcode to enter the unentered products.

[0037] After the user clicks to complete the entry on the display interface, the system receives the completion information and generates the amount information based on the second product list. After confirmation, the payment interface is displayed until it is detected that the user has completed the payment or canceled the payment.

[0038] After receiving the start instruction, the present invention takes an initial image of a predetermined area, splits it into multiple product images based on an edge detection algorithm, and then uses a depth recognition model to perform recognition, thereby quickly and accurately obtaining product information in each product image. This method effectively solves the problem of low efficiency in manually scanning product barcodes one by one, and improves cashier efficiency.

[0039] By counting the number of identified and unidentified goods in the initial image, the system can clearly sort out the list of goods to be settled after the preliminary identification of the goods. If the number of identified and unidentified goods meets the preset conditions, the weight information of the predetermined area continues to be obtained, and joint reasoning is performed based on the weight information to obtain the product information of the unidentified goods. By adding weight information for joint reasoning, the accuracy and comprehensiveness of product identification are improved. The present invention also locates the sales shelf based on the product information of the unidentified goods, and verifies the product information according to the shelf's pick-and-place information. The correctness of the inferred product information is ensured by verification, which solves the problem that the product information is still inaccurate after joint reasoning, or the problem that joint reasoning cannot be performed; in summary, the present invention combines multiple methods to obtain the product information to be settled, thereby greatly improving the accuracy of product identification.

[0040] Specifically, the present invention recognizes the commodity image and includes the following steps: If a barcode area is detected in the product image, the product information is generated based on the barcode and the corresponding first confidence level is set to 100%. If the barcode area is not identified, the product image is identified based on the deep recognition model to obtain the product information and the first confidence level. The product type is determined based on the product information. If the product type belongs to the preset type and the first confidence level is greater than the first threshold, the product quality continues to be identified. If damaged products are identified, a product quality reminder is generated.

[0041] First, the barcode area in the product image is detected. Since the barcode area usually shows high gradient changes in the horizontal direction (strip edge) and small gradient changes in the vertical direction, a detection method based on gradient direction can be used. Specifically, the Sobel operator is used to calculate the X and Y gradients of the image, and then the gradient amplitude is calculated. A threshold is set to filter out low-gradient areas and retain high-gradient areas. Then, the gradient direction is counted to screen areas with high directional consistency. If the proportion of horizontal or vertical gradient directions in the area exceeds a preset threshold (such as 80%), it is determined to be a valid barcode area.

[0042] After detecting the presence of a barcode area in the image, the product information is identified based on the barcode, and the first confidence level is set to 100%, indicating that the product information obtained through the barcode must be correct. If the barcode area is not identified, the deep recognition model is used to identify the product image to obtain the product information and the corresponding first confidence level.

[0043] The preset type of this embodiment is fruit or vegetable. The present invention introduces a quality judgment function in the self-service cash register. When quality problems are detected in fruits or vegetables, a reminder can be automatically generated, thereby improving the user's purchase satisfaction. For example, when the presence of Red Fuji apples is detected in the product image based on the deep recognition model, the quality inspection is continued. When it is found that the Red Fuji apples are damaged, a reminder is generated. In particular, fruits are usually settled in bags, and the bottom fruit is often blocked by the top fruit. In this case, when the top fruit is identified to be damaged, a reminder will still be generated.

[0044] In this embodiment, the evaluation of product quality includes the following steps: The product image is gridded to obtain multiple grid areas, the original average pixel value of each grid area is obtained, and multiple interval ranges are set. Each interval range corresponds to a mapping pixel. Based on the interval range where the original average pixel value is located, it is mapped to the corresponding mapping pixel.

[0045] For ease of description, the product image is divided into 6*6 grid areas, and each grid area includes 4*4 pixels. In particular, the finer the grid segmentation, the more accurate the final judgment result, but the judgment speed will be slower. The coarser the grid segmentation, the less accurate the final judgment result, but the judgment speed will be improved. The original average pixel value is obtained by calculating the average pixel value of all the same channels in the grid area. In this embodiment, 10 intervals are set for the three RGB channels. For example, for the three channels of R, G, and B, 0 to 25 are set as an interval unit, and the corresponding mapping pixel point is 13. When the original average pixel value of a grid area in the R channel is 19, it is in the interval range of 0 to 25, then it is mapped to 13. The processing method of other channels is similar. By mapping, the calculation complexity can be reduced and the calculation speed can be increased.

[0046] The grid area is divided into multiple types of sub-areas, and the statistical features of the sub-areas are obtained. The statistical features include the mapping pixels existing in the sub-areas and the occupancy ratio. A standard feature library is established. The standard feature library includes the standard features of each sub-area in the damaged state of each product type. The statistical features of the sub-areas in the product image are compared with the standard features, and the product quality is determined based on the comparison results.

[0047] like Figure 3As shown, with the product image as the center, multiple grid areas are divided into one area around its circumference, so that the product image is divided into three layers from the outside to the inside, wherein the area located in the outermost layer is defined as sub-area A, the area located in the middle layer is defined as sub-area B, and the area located in the innermost layer is defined as sub-area C. In other embodiments, the product image can also be divided into multiple 3*3 rectangular areas. Then, the statistical features of each sub-area are calculated. For example, in the R channel, sub-area B includes three types of mapped pixel values, 13, 38, and 63, and the ratio in sub-area B is 1:1:2, then the occupancy ratio of the mapped pixel 38 is 25%, and the mapped pixels and occupancy ratios of the G channel and the B channel are continued to be obtained in this way.

[0048] The standard database includes standard features of goods in a damaged state. For example, when the surface of the Red Fuji apple is rotten, the sub-region B should include mapping pixels 38 in the R channel, and the proportion occupied by mapping pixels 38 should be greater than or equal to 20%, the G channel should include mapping pixels 88, and the proportion occupied by mapping pixels 88 should be greater than or equal to 20%, and the B channel should include mapping pixels 138, and the proportion occupied by mapping pixels 138 should be greater than or equal to 20%. When the mapping pixels and proportions included in the sub-region meet the above conditions, it is considered that the sub-region meets the standard features. For example, in the R channel of the above-mentioned sub-region B, the proportion occupied by mapping pixels 38 is 25%, which is greater than the proportion in the standard features, that is, it meets the standard features of the R channel. When the statistical features of the three channels all meet the standard features, it is considered that the goods are damaged in the sub-region. In the product image, when at least one sub-region has a damaged feature, it is determined that there is a problem with the product quality. In other embodiments, in order to speed up the calculation speed, the product image can also be processed into a grayscale image and then processed.

[0049] Since the use of deep neural networks for recognition requires a large number of training sets for training, when it is not possible to obtain more training sets and the use of transfer learning methods cannot achieve good recognition results, the above method proposed in the present invention can be quickly deployed.

[0050] In this embodiment, verifying the commodity information based on the pick-and-place information of the selling shelf includes the following steps: A shelf image of a sales shelf is obtained. When a product is removed from the shelf in the shelf image, the product information of the removed product is determined based on the shelf data. The user who removed the product is tracked to obtain a user movement image. The facial image and body image are captured in the user movement image for pre-storage. The product information of the sales shelf is marked in the facial image and the body image as an image label. When a verification requirement arises, a search is performed based on the product information and image labels of unrecognized products. If a facial image and a body image containing unrecognized product information are retrieved, they are compared with the current user image. If the comparison is successful, it is determined that the currently recognized product information is correct.

[0051] First, a surveillance camera is set up in each shelf area. The surveillance camera captures the image of the shelf area and identifies based on the deep recognition model to determine whether there is any picking and placing behavior on the commodity shelves in the shelf area. The shelf data includes the commodity information sold on each shelf in the monitoring area of ​​each surveillance camera. When it is detected that there is a commodity removed from the selling shelf, the location of the shelf is first determined based on the monitoring image, and then the corresponding shelf data is located based on the shelf position to obtain the commodity information of the selling shelf. If it is determined that the selling shelf sells commodity A, the image of the person performing the removal operation is continued to be intercepted, and the person is tracked based on the target tracking algorithm until the facial image and body image are obtained. The features that need to be recorded in the body image include the color of the clothing. Finally, the facial image and body image are pre-stored, and the commodity information is marked as commodity A.

[0052] When the self-service cash register determines that the product information of the unrecognized product is product A, but the first confidence or the second confidence is less than the first threshold, the facial data and body data containing the image label of product A are searched in the pre-stored image data. If they are not found, the verification of the product information fails. If the facial image and body image containing product A are searched, the facial image and body image of the current settlement user are obtained and compared with the searched facial image and body image. If the comparison result shows that both are the same person, the verification of the product information passes.

[0053] In this embodiment, if it is detected in the shelf image that the user moves the product back to the shelf, or the user leaves the shopping mall area, the user's facial image and body image are deleted.

[0054] When the user moves the product back to the shelf, it means that the user has given up the purchase, so it is deleted to avoid subsequent verification errors. After the user leaves the mall, the corresponding facial and body images are removed to protect the user's privacy.

[0055] In this embodiment, joint reasoning based on all product images and weight information includes the following steps: A weight database is established, which includes the standard weight and floating range of each commodity. The error distribution of the weight sensor is set. The first weight of the identified commodity is obtained based on the weight database. The second weight of the unidentified commodity is calculated based on the total weight of the commodities in the predetermined area and the first weight. The second weight is corrected based on the floating range and the error distribution to obtain its weight distribution range.

[0056] The weight database includes the standard weight of each commodity, such as potato chips A is 200g, and the floating range of each commodity, for example, the floating range of potato chips A can be ±5g. For the weight sensor, the error distribution is set to 5g. The following example introduces the reasoning process. Assume that there are currently two identified commodities and two unidentified commodities. The identified commodities are all potato chips A. The first weight is 200g, and the total weight of the current reserved area is 1200g. The second weight is 1200-2*200=800g. Due to the presence of two potato chips A, plus the error of the sensor, the weight distribution of the second weight after correction is [800-3*5g, 800+3*5g], that is, [785g, 815g].

[0057] Acquire candidate information, which is product information with a first confidence level greater than a second threshold in a single product image recognition result; acquire the standard weight and floating range of the candidate information based on a weight database; generate candidate combinations that meet the weight distribution range based on the standard weight and floating range; and calculate a combination score for each candidate combination based on the weight distribution range of the candidate combination.

[0058] The image of the unrecognized product is identified, and the first confidence of the product A is determined to be 0.6, the first confidence of the product B is 0.55, and the first confidence of the product C is 0.4. The second threshold is set to 0.5, and the product A and product B are taken as two candidate information. In particular, if at least one of the products A and B is fresh fruit, the reasoning is abandoned and the user is reminded to settle separately, because due to the interference of other products, the weight cannot be accurately obtained during settlement. For product A, its standard weight is 400g, with a floating range of ±20g, and the standard weight of product B is 395g, with a floating range of ±25g. Then, the following candidate combinations are generated: candidate combination A: 2 products A, weight between [760g, 840g], candidate combination B: 2 products B, weight between [765g, 815g], and other candidate combinations are not given one by one.

[0059] This embodiment first defines that the weights of goods A and B obey the normal distribution. For example, goods A obeys the normal distribution with μ=400 and σ=20. Based on the probability density function of the normal distribution, the probability density value of goods A at the standard weight of 400g and the probability density value of goods B at the standard weight of 395g are calculated respectively. The probability density function of the normal distribution is common knowledge and will not be introduced here. After calculation, the probability density value of goods A at the standard weight of 400g is P(400)=0.0199, and the probability density value of goods B at the standard weight of 395g is P(400)=0.0158.

[0060] The combined score of each candidate combination is calculated based on the following first formula, which is: ,in, Score the combination. is the number of candidate information included in the candidate combination, is the probability density value of the i-th candidate information in the candidate combination under the standard weight, is a natural constant, is the penalty factor, and its value is 0.1. is the total weight of the candidate combination, which is obtained by accumulating the standard weights of the candidate information in the candidate combination. For example, the standard weight of candidate combination A is 400+400=800g. By substituting the above values ​​into the first formula, the combined score of candidate combination A can be obtained, which is Similarly, the combined score of candidate combination B can be calculated to be 0.000226. In the first formula, the larger the probability density value is, or the closer the total weight of the candidate combination is to the second weight, the larger the calculated combined score is.

[0061] The candidate score of each candidate combination is calculated based on the first confidence and the combination score, and the candidate combination corresponding to the maximum candidate score is selected as the target combination, in which the candidate information included is used as the inference result of the unidentified commodity, and the corresponding candidate score is used as the second confidence.

[0062] In this embodiment, calculating the candidate score of the candidate combination includes the following steps: The purchase probability of each candidate information is obtained based on the historical purchase records, and the first confidence, combination score and purchase probability are weighted and summed based on the preset weights to obtain the joint score of the candidate combination. The joint score is normalized to obtain the normalized score, and a penalty value is generated based on the number of candidate combinations currently existing. All normalized scores are corrected based on the penalty value to obtain the candidate score.

[0063] In this embodiment, the user needs to log in to the account before settlement. Although this method sacrifices a certain degree of convenience, it can store the user's purchase record in the user account, which is convenient for the user to check at any time and perform subsequent after-sales processing. After the user logs in to the account, the purchase probability of the candidate information is determined according to the user's purchase record. For example, the ratio of the historical purchase number of commodity A to the historical total shopping number is used as the purchase probability of commodity A.

[0064] The purchase probability, the first confidence level, and the combined score are weighted and summed according to the preset weights built into the system to obtain a joint score. For example, the preset weights of the purchase probability, the first confidence level, and the combined score are 0.1, 0.6, and 0.3, respectively. This embodiment performs a weighted summation based on the second formula. The second formula is: , where S is the joint score of the candidate combination, is the average purchase probability of candidate information (products) included in the candidate combination, is the first confidence average of the products included in the candidate combination, R is the combined score of the candidate combination, , and are the preset weights of purchase probability, first confidence and combination score respectively. For candidate combination A, the average purchase probability including candidate information is 0.16, the average first confidence including candidate information is 0.85, and the combination score is 0.000396. Substituting into the second formula, the joint score of candidate combination A is -2.02. Similarly, the joint score of candidate combination B is -2.39.

[0065] This embodiment uses the Softmax function for normalization, and the normalized score of candidate combination A is 0.592, and the normalized score of candidate combination B is 0.408. The more candidate combinations are generated, the higher the penalty value is, indicating that there are too many combinations that meet the weight distribution range, and the calculation accuracy will be greatly reduced. For example, the penalty value is 0.95 here, and the candidate score of candidate combination A is 0.562 after correction, and the candidate score of candidate combination B is 0.3876.

[0066] After the above calculation, it is determined that the candidate score of candidate combination A is the largest, then candidate combination A is the target combination, the two commodities A included in it are the reasoning results of unidentified commodities, and the candidate score of candidate combination A is the second confidence.

[0067] like Figure 4 As shown, the present invention also provides a self-service cashier system based on multimodal interaction, which is used to implement the above-mentioned self-service cashier method based on multimodal interaction, and the system includes: The recognition module, after receiving the start instruction, captures an initial image of a predetermined area, splits the initial image into multiple product images, recognizes each product image based on a deep recognition model to obtain the product information and a first confidence level included therein, and generates a first product list based on the product information whose first confidence level is greater than a first threshold.

[0068] The first verification module counts the first number and the second number of identified commodities and unidentified commodities in the initial image, obtains the weight information of the predetermined area if the first number and the second number meet the preset conditions, performs joint reasoning based on all commodity images and weight information, and obtains the commodity information and the second confidence level of the unidentified commodities.

[0069] The second verification module, if the second confidence level is less than the first threshold, locates the corresponding sales shelf based on the product information of the unidentified product, and verifies the product information based on the pick-and-place information of the sales shelf. If the verification is successful, the product information of the unidentified product is added to the first product list to obtain the second product list. If the verification fails, a manual entry reminder is generated.

[0070] The settlement module, after receiving the completion information, generates amount information based on the second commodity list and displays the payment interface until receiving the completion information or the payment cancellation information.

[0071] The present invention also discloses a computer storage medium, wherein the computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned method.

[0072] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0073] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0075] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-service checkout method based on multimodal interaction, characterized in that: include: After receiving the start instruction, an initial image of a predetermined area is captured, the initial image is split into a plurality of product images, and each of the product images is recognized based on a deep recognition model to obtain product information and a first confidence level included therein; generating a first commodity list based on the commodity information whose first confidence level is greater than a first threshold, and counting a first number and a second number of recognized commodities and unrecognized commodities in the initial image; If the first quantity and the second quantity meet a preset condition, obtaining weight information of the predetermined area, performing joint reasoning based on all the commodity images and the weight information, and obtaining the commodity information and a second confidence level of the unrecognized commodity; If the second confidence level is less than the first threshold, locating the corresponding sales shelf based on the product information of the unidentified product, and verifying the product information based on the pick-and-place information of the sales shelf; If the verification is successful, the product information of the unidentified product is added to the first product list to obtain the second product list; if the verification fails, a manual entry reminder is generated; After receiving the completion information, the amount information is generated based on the second commodity list, and the payment interface is displayed until the settlement completion information or the payment cancellation information is received.

2. The method according to claim 1, characterized in that Identifying the product image includes the following steps: If a barcode area is detected in the product image, the product information is generated based on the barcode, and the corresponding first confidence is set to 100%; if the barcode area is not identified, the product image is identified based on the deep recognition model to obtain the product information and the first confidence; the product type is determined based on the product information; if the product type belongs to a preset type and the first confidence is greater than the first threshold, the product quality continues to be identified; if damaged products are identified, a product quality reminder is generated.

3. The method according to claim 2, characterized in that Identifying the quality of the product includes the following steps: The product image is divided into grids to obtain a plurality of grid areas, an original average pixel value of each of the grid areas is obtained, a plurality of interval ranges are set, each of the interval ranges corresponds to a mapping pixel, and based on the interval range where the original average pixel value is located, it is mapped to the corresponding mapping pixel; The grid area is divided into multiple types of sub-areas, and statistical features of the sub-areas are obtained, wherein the statistical features include the mapped pixels existing in the sub-areas and their occupation ratios; a standard feature library is established, wherein the standard feature library includes standard features of each sub-area of ​​each product type in a damaged state; the statistical features of the sub-areas in the product image are compared with the standard features, and the quality of the product is determined based on the comparison results.

4. The method according to claim 1, characterized in that: Verifying the commodity information based on the pick-and-place information of the selling shelf includes the following steps: Acquire a shelf image of the selling shelf, and when a product is removed from the shelf in the shelf image, determine the product information of the removed product based on the shelf data, track the user who removed the product to obtain a user movement image, capture a facial image and a body image in the user movement image for pre-storage, and mark the product information of the selling shelf in the facial image and the body image as an image tag; when a verification requirement arises, perform a search based on the product information of the unrecognized product and the image tag; if the facial image and the body image containing the product information of the unrecognized product are retrieved, compare them with the current user image; if the comparison is successful, determine that the currently recognized product information is correct.

5. The method according to claim 4, characterized in that Performing joint reasoning based on all the product images and the weight information includes the following steps: Establishing a weight database, the weight database including a standard weight and a floating range of each commodity, setting an error distribution of a weight sensor, obtaining a first weight of an identified commodity based on the weight database, calculating a second weight of an unidentified commodity based on a total weight of commodities in the predetermined area and the first weight, and correcting the second weight based on the floating range and the error distribution to obtain a weight distribution range thereof; Acquire candidate information, where the candidate information is the commodity information whose first confidence is greater than a second threshold in a single commodity image recognition result, acquire the standard weight and the floating range of the candidate information based on the weight database, generate a candidate combination that satisfies the weight distribution range based on the standard weight and the floating range, and calculate a combination score for each candidate combination based on the weight distribution range of the candidate combination; The candidate score of each candidate combination is calculated based on the first confidence and the combination score, and the candidate combination corresponding to the maximum candidate score is screened as the target combination, wherein the candidate information included therein is used as the inference result of the unidentified commodity, and the corresponding candidate score is used as the second confidence.

6. The method according to claim 5, characterized in that Calculating the candidate score of the candidate combination comprises the following steps: The purchase probability of each type of candidate information is obtained based on historical purchase records, and the first confidence, the combination score and the purchase probability are weighted and summed based on preset weights to obtain a joint score of the candidate combination, and the joint score is normalized to obtain a normalized score, a penalty value is generated based on the number of candidate combinations currently existing, and all the normalized scores are corrected based on the penalty value to obtain the candidate score.

7. The method according to claim 6, characterized in that The preset condition includes that both the first number and the second number are smaller than a third threshold.

8. The method according to claim 4, characterized in that If it is detected in the shelf image that the user moves the product back to the shelf, or the user leaves the shopping mall area, the facial image and the body image of the user are deleted.

9. A self-service cashier system based on multimodal interaction, used to implement the method according to any one of claims 1 to 8, characterized in that: include, The recognition module, after receiving the start instruction, captures an initial image of a predetermined area, splits the initial image into a plurality of product images, recognizes each of the product images based on a deep recognition model to obtain product information and a first confidence level included therein, and generates a first product list based on the product information whose first confidence level is greater than a first threshold. a first verification module, which counts a first number and a second number of recognized commodities and unrecognized commodities in the initial image, obtains weight information of the predetermined area if the first number and the second number meet a preset condition, and performs joint reasoning based on all the commodity images and the weight information to obtain the commodity information and a second confidence level of the unrecognized commodity; a second verification module, if the second confidence level is less than the first threshold, locating a corresponding sales shelf based on the product information of the unidentified product, verifying the product information based on the pick-and-place information of the sales shelf, and if the verification is successful, adding the product information of the unidentified product to the first product list to obtain a second product list, and generating a manual entry reminder if the verification fails; The settlement module, after receiving the completion information, generates amount information based on the second commodity list and displays a payment collection interface until receiving the completion information or the payment cancellation information.

10. A computer storage medium, characterized in that: The computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images