Unmanned vending system based on dynamic vision and gravity dual mode

By combining dynamic visual recognition technology with gravity sensors and high-definition cameras in unmanned sales systems, the problem of visual recognition errors in the prior art is solved, and more accurate product recognition and more efficient purchasing experience are achieved.

CN118762437BActive Publication Date: 2024-11-26Hangzhou Qianjing Technology Co., Ltd. +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411254558.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-26
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

The existing unmanned sales control system only recognizes the purchased goods through visual recognition, which can easily lead to identification errors, especially when the goods overlap, which will not be accurately identified, which will affect the purchasing experience.

Method used

Adopt a dual-mode unmanned sales system based on dynamic vision and gravity, combining gravity sensors and high-definition cameras, and multi-level identification and calibration of products through the initial entry module and the sales confirmation module, and multiple verifications are performed using gravity sensors and image recognition technology. , ensure the accuracy of product identification.

Benefits of technology

It effectively avoids inaccurate calculation of gravity sensors, reduces the recognition error rate, improves the accuracy of product recognition and purchase experience, and reduces recognition errors caused by a single recognition method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762437B_ABST
    Figure CN118762437B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of unmanned vending control systems, and specifically to an unmanned vending system based on dynamic vision and gravity dual modes, including a vending machine with multi-layer shelves, a camera is provided above each shelf, a gravity sensor is provided on each shelf, and also includes: an initial entry module and a sold confirmation module. The present invention can not only determine the placement of goods during the placement of goods, thereby determining whether the goods are sold based on the vacancy of the placement position, but also use a multiple verification method to verify the type set of sold goods, and further add a gravity recognition method to the image recognition technology to verify the type of purchased goods. Compared with only verifying the image comparison before and after the sale or the video comparison during the purchase process, the multiple verification can effectively reduce the misjudgment rate of the recognition unit and improve the overall recognition efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned vending control systems, and in particular to an unmanned vending system based on dynamic vision and gravity dual modes. Background Art

[0002] Unmanned vending machines are intelligent retail terminals that integrate modern technology and automation technology. They can provide 24-hour uninterrupted self-service shopping services without human supervision. This type of equipment can accurately identify the goods selected by customers through a built-in intelligent recognition system, and complete transactions with a variety of convenient payment methods such as mobile payment and card payment.

[0003] The unmanned vending control system can dynamically identify the goods selected by the customer during the purchase process through the intelligent recognition system, and intelligently generate purchase orders, thereby completing order settlement and improving the customer's purchasing experience. However, the unmanned vending control system in the prior art only identifies the purchased goods through visual recognition, which is prone to recognition errors. In particular, when the goods overlap, the goods cannot be accurately identified, and it is easy to generate erroneous purchase order information, which affects the purchasing experience and has certain defects. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides an unmanned vending system based on dynamic vision and gravity dual modes, which can effectively solve the problem that the unmanned vending control system in the prior art only identifies the purchased goods by visual recognition.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] The present invention provides an unmanned vending system based on dynamic vision and gravity dual modes, which at least comprises: a vending machine with multiple shelves, a camera is provided above each shelf, a gravity sensor is provided on each shelf, and also comprises: an initial entry module and a sales confirmation module;

[0007] The initial entry module includes a product scanning unit and a position determination unit. The product scanning unit scans and summarizes product information and records the products with information entry as incoming products. The position determination unit places the incoming products on the shelves in sequence, calibrates the gravity sensor by comparing the weight difference of the shelves before and after placement with the mass of the incoming products, and determines the specific placement position of the incoming products based on the difference in product images on the shelves before and after placement.

[0008] The sales confirmation module includes a gravity confirmation unit and an image confirmation unit. The goods placed on the shelves are recorded as goods on sale, and the goods purchased by consumers are recorded as sold goods. The image confirmation unit takes high-definition images of the goods on sale on the shelves before and after the consumer purchases, as well as high-definition images of the goods on sale on the shelves during the purchase process, and analyzes them to obtain a first sold set and a second sold set containing multiple sold goods respectively. The gravity confirmation unit records the total mass change on the shelves before and after the consumer purchases and compares it with the first sold set and the second sold set to confirm the actual sold set.

[0009] Furthermore, the initial input module information input summary process is as follows:

[0010] The product name, barcode, total mass and high-definition images from multiple angles of each product are obtained, a product serial number is assigned to each product, each product serial number corresponds to a product name, and multiple data sets are established with the product serial number as the set serial number. The data sets contain the barcode, total mass and high-definition images from multiple angles of the product corresponding to the product serial number.

[0011] Furthermore, the location determination unit determines the location process as follows:

[0012] Step 1: Place incoming goods on the shelves in sequence. A placement interval threshold is preset. Place one incoming product at a time. The interval between two consecutive placements is greater than the placement interval threshold. Place incoming products in order with the front side facing up.

[0013] Step 2: Before placing the incoming goods, scan the barcode of the incoming goods to obtain the product name of the incoming goods, record the scanning time of the barcode during each placement process, and record the time interval formed by two adjacent scanning times as the placement interval. Each placement interval is bound to the product name corresponding to the previous scanning time;

[0014] Step 3: In each placement interval, the total mass of the goods on each shelf is monitored in real time and recorded as the total weight of the shelf. The shelf whose total weight changes in the placement interval is recorded as the target shelf. A line graph of the total weight of the target shelf over time is drawn. A stable time threshold is set. When the total weight of the shelf maintains a certain value and the duration is greater than or equal to the stable time threshold, the value is recorded as the stable value. The range of all stable values ​​in the placement interval is calculated and recorded as the screening weight. The incoming goods with a total mass equal to the screening weight are obtained and form a set of goods to be selected;

[0015] Step 4: Continuously capture and save high-definition images above the shelf within the placement interval, obtain the maximum and minimum values ​​of the stability value corresponding to the target shelf within the placement interval, record the time when the minimum stability value appears as the first moment, and record the time when the maximum stability value appears as the second moment, compare the high-definition image collected at the first moment with the high-definition image collected at the second moment, extract the difference between the two high-definition images, perform image recognition to confirm, and record the commodity name corresponding to the difference as the pending commodity;

[0016] Step 5: Obtain the total mass of the pending product as the comparison weight, compare the screening weight with the comparison weight, and when the screening weight is equal to the comparison weight, record the position of the different part as the placement position of the incoming product. Each placement position corresponds to an incoming product. When the screening weight is not equal to the comparison weight, generate an abnormal recognition signal.

[0017] Furthermore, the image recognition process is specifically as follows:

[0018] A high-definition image collected at a first moment and a high-definition image collected at a second moment are obtained and recorded as a first comparison image and a second comparison image respectively. The first comparison image and the second comparison image are subjected to the same grayscale processing and overlapped, and pixels with different grayscale values ​​are retained to obtain a difference distribution map. A portion of the second comparison image that overlaps with the difference distribution map is extracted and recorded as a recognition image. The recognition image is compared with high-definition images of all commodities in the set of commodities to be selected at various angles for similarity, and the commodity name of the high-definition image with the highest similarity is selected as the commodity name corresponding to the recognition image.

[0019] Remove the discrete pixels in the difference distribution map and record it as a grayscale difference map. Draw a range circle with the center point of the pixel in the grayscale difference map as the center. The radius r of the range circle is calculated as follows: , where N is the total number of pixels in the grayscale difference map, k is the preset proportional coefficient, and the position corresponding to the range circle is recorded as the placement position of the incoming goods.

[0020] Furthermore, it also includes a placement optimization unit for optimizing the placement of goods. The optimization placement process is specifically as follows:

[0021] Get the total mass of all incoming goods, re-sort the serial numbers of all incoming goods according to the total mass and record them in order. , i is the total number of product serial numbers, and if the total number of shelf layers is j, then the product serial number is The incoming goods are placed on the shelves in turn, and the corresponding product numbers are The incoming goods are placed in Layer shelves, including It represents the remainder when i is divided by j.

[0022] Furthermore, the image confirmation unit comparison analysis confirmation process is specifically as follows:

[0023] The purchase time interval is set with the opening and closing moments of the vending machine door as the interval endpoints. During the purchase time interval, the high-definition images of the goods on sale on each shelf are continuously captured by the camera. The total mass difference of the total mass of the goods on sale on each shelf at the interval endpoint moments is calculated. The shelves whose total mass difference is not 0 are recorded as changing shelves. The high-definition images taken by the changing shelves within the purchase time interval are identified and analyzed. Two high-definition images with different pictures at adjacent shooting times are recorded as dynamic changing images. A plurality of continuously shot dynamic changing images constitute a dynamic changing video. The dynamic changing part in the dynamic changing video is captured and extracted to obtain a continuous dynamic changing picture. Each individual picture in the dynamic changing picture is compared with the high-definition images of all the goods in the warehouse for similarity. The product name of the high-definition image with the highest similarity is selected as the product name recognized in the picture. When multiple dynamic changing pictures all recognize the same product name, the product name is used as the sold product corresponding to the dynamic changing video. The sold products corresponding to all the dynamic changing videos of the same shelf are obtained and constitute the first sold set of the shelf.

[0024] Each shelf is analyzed independently, and high-definition images of the goods on sale on the shelf at the two endpoints of the interval are obtained and recorded as pre-sale images and after-sale images respectively. The pre-sale images and after-sale images are subjected to the same grayscale processing and overlapped, and the pixels with different grayscale values ​​in the two images are retained to obtain a before-and-after difference map. The pixels retained in the before-and-after difference map are recorded as difference points, and all the placement positions of the shelf on this layer are drawn on the before-and-after difference map. A certain threshold is set, and when the area occupied by the difference points in the placement position is greater than or equal to the certain threshold, the placement position is recorded as a sold-out position, and the goods on sale corresponding to the sold-out position are recorded as sold goods, and all the sold goods on the same shelf are obtained to form a second sold set of the shelf.

[0025] Furthermore, the gravity confirmation unit comparison analysis confirmation process is as follows:

[0026] Obtain the total mass difference of the total mass of the goods on sale on each shelf at the time of the interval endpoint and record it as the actual difference; calculate the sum of the total mass of all the goods on sale in the first sold set and the second sold set respectively and record them as the first ideal difference and the second ideal difference respectively; when any one of the first ideal difference and the second ideal difference is equal to the actual difference, take the sold set corresponding to the ideal difference as the actual sold set, and further check the actual sold set; when the first ideal difference, the second ideal difference and the actual difference are not equal, generate a recognition failure signal, and the staff will perform manual check;

[0027] Draw a line graph of the total shelf weight of each shelf over time within the purchase time interval and extract the stable value in the line graph. Record all stable values ​​in chronological order to form the number of stable values. , where n is the total number of stable values, They represent the total weight of the shelf when the door is open and the total weight of the shelf when the door is closed respectively. The difference between two adjacent stable values ​​is calculated and recorded as the product quality difference, and n-1 product quality differences are obtained. When each product quality difference can find a unique sold product corresponding to it in the actual sold set, the actual sold set is recorded as the consumption set after verification, and all the sold products in the consumption set are regarded as the products purchased by the consumer on this shelf. The products purchased by the consumer are equal to the sum of all the sold products in the consumption set. When any actual sold set cannot pass the verification, a verification failure signal is generated and the staff will perform manual verification.

[0028] Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects:

[0029] 1. The present invention determines whether the gravity sensor can normally identify and calculate the weight difference of the shelf before and after the goods are placed by comparing the weight of the goods and the weight change value of the shelf when placing the goods. It can effectively calibrate the sensitivity of the gravity sensor, thereby avoiding inaccurate calculation when the vending machine is used, and by arranging the incoming goods in layers according to the order of weight, the goods with the same or similar weight can be placed on shelves of different layers, thereby reducing the situation where the same total mass of goods appear on the shelves of the same layer, which is conducive to the gravity sensor to be more accurate in determining the name of the sold goods based on the analysis of the weight changes of the goods on each shelf.

[0030] 2. The present invention can not only dynamically identify dynamically changing videos and effectively identify consumers' purchasing behaviors, namely, a set of actions of "reaching out - taking - taking back", and then determine the commodities purchased by consumers according to the commodities appearing in each set of actions, which is conducive to the final confirmation of the consumer's final order, but also can determine the placement of incoming commodities on the shelves and compare the high-definition images of commodities on sale on the shelves before and after the purchase, so as to learn the commodities purchased by consumers according to the vacant spaces on the shelves.

[0031] 3. The present invention adopts a multiple verification method to verify the type set of sold goods, and further adds a gravity recognition method to the image recognition technology through the gravity confirmation unit to verify the types of purchased goods. Compared with only verifying the image comparison before and after the sale or the video comparison during the purchase process, the multiple verification can effectively reduce the misjudgment rate of the recognition unit and reduce the recognition error caused by a single recognition method. When a single recognition method cannot identify the correct set of sold goods, another recognition method can be used for recognition and comparison, thereby obtaining a correct recognition result and improving the overall recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is the overall module block diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention will be further described below in conjunction with the embodiments.

[0036] See also Figure 1 The unmanned vending system based on dynamic vision and gravity dual modes includes a vending machine with multi-layer shelves. A camera for photographing goods on the shelves is provided above each shelf. A gravity sensor is provided on each shelf to measure the total mass of the goods on the shelves. The system at least includes: an initial entry module and a sales confirmation module.

[0037] The initial entry module includes a commodity scanning unit for collecting and summarizing information and a position determination unit for determining the placement position of the commodity.

[0038] The commodity scanning unit scans and enters the commodity information and summarizes the information, and records the commodities after information entry as warehoused commodities. The information entry and summary process is as follows:

[0039] Obtain the product name, barcode, total mass and high-definition images from multiple angles of each product, assign a product serial number to each product, each product serial number corresponds to a product name (the same product name corresponds to the same product serial number), and establish multiple data sets with the product serial number as the set serial number. The data set contains the barcode, total mass and high-definition images of the product corresponding to the product serial number. When the total mass and high-definition image of a product are determined, the product serial number of the product can be confirmed, thereby obtaining the product name of the product. The product name can also be directly obtained by barcode scanning. Therefore, through image recognition technology and gravity recognition technology, the serial number and product name of the product can be accurately identified when the total mass and image of the product are known.

[0040] The location determination unit records the commodities placed on the shelf and determines the placement of each commodity. The location determination process is as follows:

[0041] Step 1: Place incoming goods on the shelf in sequence. A placement interval threshold is preset (in a specific embodiment, the value is 5 seconds). Place one incoming goods each time. The interval between two adjacent placements is greater than the placement interval threshold, so as to ensure that the gravity sensor can accurately identify the total mass of the goods. The incoming goods are placed in sequence with the front side facing up (the front of the goods is often printed with eye-catching brand logos, product names, main patterns or attractive slogans, while the back may be relatively simple, or printed with auxiliary information such as ingredient lists and instructions for use, so as to distinguish the front and back. Sequential placement means placing the goods in rows starting from a corner of the shelf until the shelf is full);

[0042] Step 2: Before placing the incoming goods, scan the barcode of the incoming goods to obtain the product name of the incoming goods, record the scanning time of the barcode during each placement process, and record the time interval formed by two adjacent scanning times as the placement interval. Each placement interval is bound to the product name corresponding to the previous scanning time (that is, the time period from each scan to the completion of the next scan is bound to the product name of the scan, so that the incoming goods scanned this time are analyzed independently);

[0043] Step 3: In each placement interval, the total mass of the goods on each shelf is monitored in real time and recorded as the total weight of the shelf. The shelf whose total weight changes in the placement interval is recorded as the target shelf. A line graph of the total weight of the target shelf over time is drawn, and a stable time threshold is set (in a specific embodiment, the value is 1 second). When the total weight of the shelf maintains a certain value and the duration is greater than or equal to the stable time threshold, the value is recorded as the stable value, and the range of all stable values ​​in the placement interval (the maximum value of all stable values ​​in the placement interval minus the minimum value, that is, the difference in the total weight of the shelf before and after the storage goods are placed, that is, the weight of the storage goods) is calculated and recorded as the screening weight, and the storage goods with a total mass equal to the screening weight are obtained and form a set of selected goods;

[0044] Step 4: Continuously capture and save high-definition images above the shelf within the placement interval, obtain the maximum and minimum values ​​of the stability value corresponding to the target shelf within the placement interval, record the time when the minimum stability value appears as the first moment, and record the time when the maximum stability value appears as the second moment, compare the high-definition image collected at the first moment with the high-definition image collected at the second moment, extract the difference between the two high-definition images, perform image recognition to confirm, and record the commodity name corresponding to the difference as the pending commodity;

[0045] Step 5: Obtain the total mass of the pending product as the comparison weight, compare the screening weight with the comparison weight, and when the screening weight is equal to the comparison weight, record the position of the different part as the placement position of the incoming product. Each placement position corresponds to an incoming product. When the screening weight is not equal to the comparison weight, generate an abnormal recognition signal.

[0046] By comparing the screening weight and the comparison weight, we can determine whether the gravity sensor can normally identify and calculate the weight difference between the front and rear shelves where the goods are placed. Because when the gravity sensor is operating normally, the weight difference between the front and rear shelves where the goods are placed should be equal to the total mass of the goods. By comparing the two sets of weight values ​​front and back, we can effectively calibrate the sensitivity of the gravity sensor, thereby avoiding inaccurate calculations when the vending machine is used.

[0047] Furthermore, the image recognition process is as follows:

[0048] A high-definition image collected at a first moment and a high-definition image collected at a second moment are obtained and recorded as a first comparison image and a second comparison image respectively. The first comparison image and the second comparison image are subjected to the same grayscale processing and overlapped, and pixels with different grayscale values ​​are retained to obtain a difference distribution map. A portion of the second comparison image that overlaps with the difference distribution map is extracted and recorded as a recognition image. The recognition image is compared with high-definition images of all commodities in the set of commodities to be selected at various angles for similarity, and the commodity name of the high-definition image with the highest similarity is selected as the commodity name corresponding to the recognition image.

[0049] Remove the discrete pixels in the difference distribution map and record it as a grayscale difference map. Draw a range circle with the center point of the pixel in the grayscale difference map as the center. The radius r of the range circle is calculated as follows: , where N is the total number of pixels in the grayscale difference map, k is the preset proportional coefficient, and the position corresponding to the range circle is recorded as the placement position of the incoming goods. By determining the placement position of the incoming goods on the shelf, it can help to learn the goods purchased by consumers based on the empty positions on the shelf during the image recognition process. Compared with directly judging the type of goods sold by identifying the image of the goods themselves, it can overcome the problem of recognition errors caused by blurred product images and overlapping products during dynamic recognition.

[0050] Furthermore, it also includes a placement optimization unit for optimizing the placement of goods. The optimization process is as follows:

[0051] Get the total mass of all incoming goods, re-sort the serial numbers of all incoming goods according to the total mass and record them in order. , i is the total number of product serial numbers, and if the total number of shelf layers is j, then the product serial number is The incoming goods are placed on the shelves in turn, and the corresponding product numbers are The incoming goods are placed in Layer shelves, including It represents the remainder when i is divided by j.

[0052] By arranging the incoming goods in layers according to the order of weight, goods of the same or similar weight can be placed on shelves of different layers, thereby reducing the situation where goods of the same total weight appear on the same shelf. This is conducive to the gravity sensor to be more accurate in determining the name of the goods sold by analyzing the changes in the quality of goods on each shelf. It should be noted that the optimized placement is only for different incoming goods and will not affect the placement of the same incoming goods. In other words, the same incoming goods (the same kind of goods) can still be placed on the same shelf even if the total weight is the same, because this situation will not affect the gravity sensor's recognition of the goods.

[0053] The sales confirmation module includes a gravity confirmation unit and an image confirmation unit, which records the goods placed on the shelf as goods on sale and the goods purchased by consumers as goods sold.

[0054] The image confirmation unit takes high-definition images of the goods on the shelf before and after the consumer purchases, as well as high-definition images of the goods on the shelf during the purchase process, and performs comparison and analysis to determine the type of goods purchased by the consumer. The comparison, analysis and confirmation process is as follows:

[0055] The purchase time interval is set with the opening and closing moments of the vending machine door as the interval endpoints. During the purchase time interval, the high-definition images of the goods on sale on each shelf are continuously captured by the camera. The total mass difference of the total mass of the goods on sale on each shelf at the interval endpoint moments is calculated. The shelves whose total mass difference is not 0 are recorded as changing shelves. The high-definition images taken of the changing shelves within the purchase time interval are identified and analyzed. Two high-definition images with different pictures in adjacent shooting time are recorded as dynamic changing images. Multiple continuously shot dynamic changing images constitute a dynamic changing video (there may be multiple dynamic changing videos in the purchase time interval, and each dynamic changing video corresponds to a sold product). The dynamically changing part in the changing video is captured and extracted to obtain a continuous dynamically changing picture, and each individual picture in the dynamically changing picture is compared with the high-definition images of all the warehoused goods for similarity, and the product name of the high-definition image with the highest similarity is selected as the product name recognized in the picture. When multiple dynamically changing pictures all recognize the same product name (that is, when pictures that account for more than X of the dynamically changing pictures all correspond to the recognition of the same product name, and X is 40% in a specific embodiment), the product name is used as the sold product corresponding to the dynamically changing video, and the sold products corresponding to all the dynamically changing videos of the same shelf are obtained and form a first sold set of the shelf;

[0056] By dynamically identifying the dynamically changing video, when a moved commodity on sale is identified in the dynamically changing video, it means that the commodity on sale has been taken out by the consumer, which means that the commodity may be a sold commodity, and it is thus classified into the first sold set for further verification and analysis. By extracting the dynamically changing image to form a dynamically changing video, the consumer's purchasing behavior, that is, the set of actions of "reaching out - taking - taking back", can be effectively identified, and then the commodities purchased by the consumer can be determined based on the commodities appearing in each set of actions, which is conducive to the final confirmation of the consumer's final order.

[0057] Each shelf is analyzed independently, and high-definition images of the goods on sale on the shelf at the two endpoints of the interval are obtained and recorded as pre-sale images and after-sale images respectively. The pre-sale images and after-sale images are subjected to the same grayscale processing and overlapped, and the pixels with different grayscale values ​​in the two images are retained to obtain a before-and-after difference map, and the pixels retained in the before-and-after difference map are recorded as difference points. All the placement positions (i.e., range circles) of the shelf on this layer are drawn on the before-and-after difference map, and a determination threshold is set (in a specific embodiment, the value is 80%). When the area occupied by the difference points in the placement position is greater than or equal to the determination threshold, the placement position is recorded as a sold-out position, and the goods on sale corresponding to the sold-out position are recorded as sold goods. All the sold goods on the same shelf are obtained to form a second sold set of the shelf.

[0058] When the area occupied by the difference points in the placement position is greater than or equal to a determined threshold, it means that the product image corresponding to the position has changed, which means that the product on sale at the placement position has been taken away by the consumer, thereby making the placement position vacant and identified by comparison by the image confirmation unit. By confirming all the vacant positions, all the products on sale taken away by consumers can be confirmed.

[0059] The gravity confirmation unit records the total weight change of the goods on sale on the shelf before and after the consumer purchases, and performs a comparison analysis to confirm the type of goods purchased by the consumer. The comparison analysis confirmation process is as follows:

[0060] Obtain the total mass difference of the total mass of the goods on sale on each shelf at the time of the interval endpoint and record it as the actual difference; calculate the sum of the total mass of all the goods on sale in the first sold set and the second sold set respectively and record them as the first ideal difference and the second ideal difference respectively; when any one of the first ideal difference and the second ideal difference is equal to the actual difference, take the sold set corresponding to the ideal difference as the actual sold set, and further check the actual sold set; when the first ideal difference, the second ideal difference and the actual difference are not equal, generate a recognition failure signal, and the staff will perform manual check;

[0061] Compared with only comparing images before and after the sale or videos during the purchase process, multiple verifications are used to verify the set of types of goods sold. Multiple verifications can effectively reduce the misjudgment rate of the recognition unit and the recognition error caused by a single recognition method. When a single recognition method cannot identify the correct set of goods sold, another recognition method can be used for recognition and comparison to obtain the correct recognition result, thereby improving the overall recognition efficiency.

[0062] Draw a line graph of the total shelf weight of each shelf over time within the purchase time interval and extract the stable value in the line graph. Record all stable values ​​in chronological order to form the number of stable values. , where n is the total number of stable values, They represent the total weight of the shelf when the door is open and the total weight of the shelf when the door is closed respectively. The difference between two adjacent stable values ​​is calculated and recorded as the product quality difference, and n-1 product quality differences are obtained. When each product quality difference can find a unique sold product corresponding to it in the actual sold set, the actual sold set is recorded as the consumption set after verification, and all the sold products in the consumption set are regarded as the products purchased by the consumer on this shelf. Therefore, the products purchased by the consumer are equal to the sum of all the sold products in the consumption set. When any actual sold set cannot pass the verification, a verification failure signal is generated and the staff will perform manual verification.

[0063] When each product with poor quality can find a unique sold product corresponding to it in the actual sold set, it means that each product on sale can find the corresponding shelf weight change caused by purchase, which indirectly proves that all the products on sale are accurately identified during the image recognition process. Verifying the image recognition results based on the shelf weight change can make the output results more accurate, improve the consumer experience when purchasing, and reduce consumer disputes caused by incorrect identification.

[0064] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unmanned vending system based on dynamic vision and gravity dual modes, including a vending machine with multiple shelves, a camera is provided above each shelf, and a gravity sensor is provided on each shelf, characterized in that: Also includes: Initial entry module and sales confirmation module; The initial entry module includes a product scanning unit and a position determination unit. The product scanning unit scans and summarizes product information and records the products with information entry as incoming products. The position determination unit places the incoming products on the shelves in sequence, calibrates the gravity sensor by comparing the weight difference of the shelves before and after placement with the mass of the incoming products, and determines the specific placement position of the incoming products based on the difference in product images on the shelves before and after placement. The location determination unit determines the location process as follows: Step 1: Place incoming goods on the shelves in sequence. A placement interval threshold is preset. Place one incoming product at a time. The interval between two consecutive placements is greater than the placement interval threshold. Place incoming products in order with the front side facing up. Step 2: Before placing the incoming goods, scan the barcode of the incoming goods to obtain the product name of the incoming goods, record the scanning time of the barcode during each placement process, and record the time interval formed by two adjacent scanning times as the placement interval. Each placement interval is bound to the product name corresponding to the previous scanning time; Step 3: In each placement interval, the total mass of the goods on each shelf is monitored in real time and recorded as the total weight of the shelf. The shelf whose total weight changes in the placement interval is recorded as the target shelf. A line graph of the total weight of the target shelf over time is drawn. A stable time threshold is set. When the total weight of the shelf maintains a certain value and the duration is greater than or equal to the stable time threshold, the value is recorded as the stable value. The range of all stable values ​​in the placement interval is calculated and recorded as the screening weight. The incoming goods with a total mass equal to the screening weight are obtained and form a set of goods to be selected; Step 4: Continuously capture and save high-definition images above the shelf within the placement interval, obtain the maximum and minimum values ​​of the stability value corresponding to the target shelf within the placement interval, record the time when the minimum stability value appears as the first moment, and record the time when the maximum stability value appears as the second moment, compare the high-definition image collected at the first moment with the high-definition image collected at the second moment, extract the difference between the two high-definition images, perform image recognition to confirm, and record the commodity name corresponding to the difference as the pending commodity; Step 5: Obtain the total mass of the pending product as the comparison weight, compare the screening weight with the comparison weight, and when the screening weight and the comparison weight are equal, record the position of the difference as the placement position of the incoming product. Each placement position corresponds to an incoming product. When the screening weight and the comparison weight are not equal, generate an abnormal identification signal; The image recognition process is as follows: A high-definition image collected at a first moment and a high-definition image collected at a second moment are obtained and recorded as a first comparison image and a second comparison image respectively. The first comparison image and the second comparison image are subjected to the same grayscale processing and overlapped, and pixels with different grayscale values ​​are retained to obtain a difference distribution map. A portion of the second comparison image that overlaps with the difference distribution map is extracted and recorded as a recognition image. The recognition image is compared with high-definition images of all commodities in the set of commodities to be selected at various angles for similarity, and the commodity name of the high-definition image with the highest similarity is selected as the commodity name corresponding to the recognition image. Remove the discrete pixels in the difference distribution map and record it as a grayscale difference map. Draw a range circle with the center point of the pixel in the grayscale difference map as the center. The radius r of the range circle is calculated as follows: , where N is the total number of pixels in the grayscale difference map, k is the preset proportional coefficient, and the position corresponding to the range circle is recorded as the placement position of the incoming goods; It also includes a placement optimization unit for optimizing the placement of goods. The optimization process is as follows: Get the total mass of all incoming goods, re-sort the serial numbers of all incoming goods according to the total mass and record them in order. , i is the total number of product serial numbers, and if the total number of shelf layers is j, then the product serial number is The incoming goods are placed on the shelves in turn, and the corresponding product numbers are The incoming goods are placed in Layer shelves, including It represents the remainder when i is divided by j; The sales confirmation module includes a gravity confirmation unit and an image confirmation unit. The goods placed on the shelves are recorded as goods on sale, and the goods purchased by consumers are recorded as sold goods. The image confirmation unit takes high-definition images of the goods on sale on the shelves before and after the consumer purchases, as well as high-definition images of the goods on sale on the shelves during the purchase process, and analyzes them to obtain a first sold set and a second sold set containing multiple sold goods respectively. The gravity confirmation unit records the total mass change on the shelves before and after the consumer purchases and compares it with the first sold set and the second sold set to confirm the actual sold set.

2. The unmanned vending system based on dynamic vision and gravity dual mode according to claim 1 is characterized in that: The initial entry module information entry summary process is as follows: The product name, barcode, total mass and high-definition images from multiple angles of each product are obtained, a product serial number is assigned to each product, each product serial number corresponds to a product name, and multiple data sets are established with the product serial number as the set serial number. The data sets contain the barcode, total mass and high-definition images from multiple angles of the product corresponding to the product serial number.

3. The unmanned vending system based on dynamic vision and gravity dual mode according to claim 1 is characterized in that: The image confirmation unit comparison analysis confirmation process is specifically as follows: The purchase time interval is set with the opening and closing moments of the vending machine door as the interval endpoints. During the purchase time interval, the high-definition images of the goods on sale on each shelf are continuously captured by the camera. The total mass difference of the total mass of the goods on sale on each shelf at the interval endpoint moments is calculated. The shelves whose total mass difference is not 0 are recorded as changing shelves. The high-definition images taken by the changing shelves within the purchase time interval are identified and analyzed. Two high-definition images with different pictures at adjacent shooting times are recorded as dynamic changing images. A plurality of continuously shot dynamic changing images constitute a dynamic changing video. The dynamic changing part in the dynamic changing video is captured and extracted to obtain a continuous dynamic changing picture. Each individual picture in the dynamic changing picture is compared with the high-definition images of all the goods in the warehouse for similarity. The product name of the high-definition image with the highest similarity is selected as the product name recognized in the picture. When multiple dynamic changing pictures all recognize the same product name, the product name is used as the sold product corresponding to the dynamic changing video. The sold products corresponding to all the dynamic changing videos of the same shelf are obtained and constitute the first sold set of the shelf. Each shelf is analyzed independently, and high-definition images of the goods on sale on the shelf at the two endpoints of the interval are obtained and recorded as pre-sale images and after-sale images respectively. The pre-sale images and after-sale images are subjected to the same grayscale processing and overlapped, and the pixels with different grayscale values ​​in the two images are retained to obtain a before-and-after difference map. The pixels retained in the before-and-after difference map are recorded as difference points, and all the placement positions of the shelf on this layer are drawn on the before-and-after difference map. A certain threshold is set, and when the area occupied by the difference points in the placement position is greater than or equal to the certain threshold, the placement position is recorded as a sold-out position, and the goods on sale corresponding to the sold-out position are recorded as sold goods, and all the sold goods on the same shelf are obtained to form a second sold set of the shelf.

4. The unmanned vending system based on dynamic vision and gravity dual mode according to claim 3 is characterized in that: The gravity confirmation unit comparison analysis confirmation process is as follows: Obtain the total mass difference of the total mass of the goods on sale on each shelf at the time of the interval endpoint and record it as the actual difference; calculate the sum of the total mass of all the goods on sale in the first sold set and the second sold set respectively and record them as the first ideal difference and the second ideal difference respectively; when any one of the first ideal difference and the second ideal difference is equal to the actual difference, take the sold set corresponding to the ideal difference as the actual sold set, and further check the actual sold set; when the first ideal difference, the second ideal difference and the actual difference are not equal, generate a recognition failure signal, and the staff will perform manual check; Draw a line graph of the total shelf weight of each shelf over time within the purchase time interval and extract the stable value in the line graph. Record all stable values ​​in chronological order to form the number of stable values. , where n is the total number of stable values, They represent the total weight of the shelf when the door is open and the total weight of the shelf when the door is closed respectively. The difference between two adjacent stable values ​​is calculated and recorded as the product quality difference, and n-1 product quality differences are obtained. When each product quality difference can find a unique sold product corresponding to it in the actual sold set, the actual sold set is recorded as the consumption set after verification, and all the sold products in the consumption set are regarded as the products purchased by the consumer on this shelf. The products purchased by the consumer are equal to the sum of all the sold products in the consumption set. When any actual sold set cannot pass the verification, a verification failure signal is generated and the staff will perform manual verification.