Unmanned vending system based on artificial intelligence

Abstract

The present invention relates to the technical field of control systems, and in particular to an unmanned vending system based on artificial intelligence, comprising: a placement video processing unit, which is used to crop and intercept a commodity placement video to obtain a comparison image; a coordinate construction unit, which is used to analyze the shelf serial number to which the commodity placement area belongs based on the comparison image, and determine the column sequence, row sequence and width of the commodity placement area; and to construct placement coordinates based on the shelf serial number to which the commodity placement area belongs and the corresponding column sequence, row sequence and width; an order generation unit, which is used to set a verification area on the top shelf inside an unmanned cabinet, construct a shopping order, and obtain all commodities in the shopping order as target commodities; and a purchase verification unit, which is used to intercept and screen a pickup video to obtain a valid image, obtain a hand area contour in the valid image for analysis, obtain a plurality of pickup coordinates, and compare the placement coordinates corresponding to the target commodity with the pickup coordinates one by one to generate a verification correct signal or a verification error signal.

Description

Technical Field

[0001] The present invention relates to the technical field of control systems, and in particular to an unmanned vending system based on artificial intelligence. Background Art

[0002] Unmanned smart vending machines are a new type of retail equipment that combines artificial intelligence, the Internet of Things, sensor technology, big data analysis and other cutting-edge technologies. Through intelligent identification technology and an automated settlement system, they can automatically sell and manage goods. They display, sell, pay and manage goods in an intelligent way without the need for human supervision, providing consumers with a convenient and efficient shopping experience. Consumers can purchase goods by scanning a QR code, scanning their face or using other payment methods without human intervention.

[0003] Unmanned smart vending machines in the prior art generally use deep learning-based video recognition algorithms to dynamically identify pickup videos to determine the order of goods purchased by users. However, the dynamic recognition algorithm has the defect of insufficient robustness, which can easily lead to order generation errors, and there is a lack of a system that can further intelligently verify the order information in combination with the pickup video, resulting in poor actual application results. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides an unmanned vending system based on artificial intelligence, which can effectively solve the problem that the unmanned vending system in the prior art is difficult to perform intelligent verification on the generated shopping orders.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] The present invention provides an unmanned vending system based on artificial intelligence, comprising:

[0007] The display video processing unit is used to crop and cut the product display video to obtain a comparison image;

[0008] The coordinate construction unit records the vertical space area where the multi-layer shelf is located as the placement area, records the space area between the shelf and the cabinet door as the observation area, and captures the image of the comparison image located in the observation area as the placement observation map;

[0009] Screening observation images based on the proportion of background area in the image observation area;

[0010] Set the same-layer comparison areas at the four corners of the placement area, determine the shelf numbers corresponding to each placement observation map based on the image changes in the four same-layer comparison areas in the placement observation map, and select the goods placement map corresponding to each shelf layer;

[0011] The area where the same type of goods are placed is recorded as the goods placement area, and the column order, row order and width of the goods placement area are determined based on the distribution of different types of goods in the goods placement diagram;

[0012] Construct placement coordinates based on the shelf number of the product placement area and the corresponding column sequence, row sequence and width;

[0013] The order generation unit sets a verification area on the top shelf inside the unmanned cabinet, identifies the commodities in the verification area based on the image recognition algorithm and builds a shopping order, and obtains all commodities in the shopping order as target commodities;

[0014] The purchase verification unit captures and filters the pickup video to obtain a valid image, obtains the hand area contour in the valid image for analysis, obtains multiple pickup coordinates, compares the placement coordinates corresponding to the target product with the pickup coordinates one by one, and generates a correct verification signal or an incorrect verification signal.

[0015] Furthermore, the process of placing the video processing unit to obtain the comparison image is as follows:

[0016] The state in which all commodity shelves are empty is recorded as an empty state, and an image of the cabinet captured by the main camera in the empty state is preset and recorded as an empty image;

[0017] A comparison cycle is preset. A frame image corresponding to the placed video is captured every comparison cycle and compared with the cleared image for similarity. A comparison value is generated based on the comparison result, where:

[0018] When the similarity between the two images is greater than a preset first similarity threshold, the comparison value is assigned to 1; when the similarity between the two images is less than or equal to the preset first similarity threshold, the comparison value is assigned to 0;

[0019] Construct a time series of comparison values ​​changing over time, split the time series into multiple zero-value sequences and sort them in chronological order. The zero-value sequence only contains comparison values ​​with a value of 0. Extract the last zero-value sequence and obtain the corresponding time interval as the analysis time zone.

[0020] The frame images of the placement video captured within the analysis time zone are recorded as comparison images.

[0021] Furthermore, the screening process of the goods placement diagram corresponding to each shelf is as follows:

[0022] Step 1: A bright background board is set at the bottom of the observation area, and the placement observation map is grayed out to calculate the number of pixels corresponding to different gray values. A background gray interval is preset, and the percentage of the number of pixels whose gray values ​​are within the background gray interval is calculated and recorded as the background percentage. When the background percentage is less than or equal to the preset background percentage threshold, the placement observation map is marked as active, otherwise, the placement observation map is marked as static, and the static placement observation map is selected and step 2 is performed;

[0023] Step 2: Construct a same-layer calibration set, which includes four same-layer calibration images, and intercept the images in the four same-layer comparison areas in the cleared image as the initial elements in the same-layer calibration set, and proceed to step 3;

[0024] Step 3: intercept the images in the four same-layer comparison areas in the placement observation map and record them as real-time calibration maps. Perform similarity comparison between the real-time calibration map and the corresponding same-layer calibration map in the same-layer proofreading set. When the similarity between the two is less than or equal to a preset second similarity threshold, record the real-time calibration map as a change map. When the number of change maps in the real-time proofreading set is equal to 4, use the real-time proofreading set as a new same-layer proofreading set and repeat steps 2 and 3.

[0025] Step 4: Set a layer sequence value for the placement observation map, assign a layer sequence value based on the number of times the same-layer proofreading set is changed, the layer sequence value is equal to the number of times the same-layer proofreading set is changed, and divide the placement observation map into different image sets according to different layer sequence values;

[0026] Step 5: Get the first placement observation image in the image set as the goods placement image of the corresponding shelf.

[0027] Furthermore, the column order, row order and width of the commodity placement area are determined as follows:

[0028] The types of goods and their outlines in the goods placement diagram are identified through an image recognition algorithm, and multiple area rectangles are constructed. Goods of the same type are divided into the same area rectangle. The area rectangle just contains the outlines of all goods of the same type. Each area rectangle corresponds to a product placement area.

[0029] With the shelf width as the horizontal axis and the shelf depth as the vertical axis, a plane rectangular coordinate system is constructed to obtain the coordinates of the center point of each area rectangle, and the horizontal coordinate distance between the center points of two adjacent area rectangles is calculated and recorded as the offset distance. When the offset distance is less than or equal to the preset offset threshold, the two adjacent area rectangles are recorded as rectangles in the same column;

[0030] Construct multiple sets of rectangles in the same column, any two adjacent area rectangles in the set are rectangles in the same column, sort the sets of rectangles in the same column according to the average value of the horizontal coordinates of the center points of all the area rectangles in the set, assign a column sequence value to each set of rectangles in the same column, the column sequence value is equal to the sorting sequence number of the set of rectangles in the same column, obtain the average value of the horizontal coordinates of the center points of all the area rectangles in the set of rectangles in the same column and record it as the column sequence reference value, the column sequence reference value corresponds to the column sequence value;

[0031] Sort all the area rectangles in the same column rectangle set, sort the area rectangles according to the ordinate size of the center point of the area rectangle, and assign a row sequence value to each area rectangle, which is equal to the sorting sequence number of the area rectangle in the same column rectangle set;

[0032] Get the length of the vertical axis of each area rectangle as the width of the area rectangle.

[0033] Furthermore, the effective image acquisition process is as follows:

[0034] The pickup video is captured with the comparison period as the interval period to obtain multiple pickup images. The pickup images are divided into static state and active state. The pickup images in the active state are obtained and recorded as target images. Based on the proportion of the hand area in the target image, the valid images in the target image are screened out.

[0035] Furthermore, the effective image screening process is as follows:

[0036] The image in the observation area of ​​the target image is intercepted and recorded as the pickup observation image. The bright background area in the pickup observation image is removed to obtain the hand area. The outline of the hand area is drawn, and a rectangle with the smallest area containing the outline of the hand area is constructed and recorded as the analysis rectangle. The area of ​​the analysis rectangle is , let the area of ​​the hand region contour be , substitute into the formula Calculate in and get the hand ratio value , a hand proportion threshold is preset, and the target image whose hand proportion value is greater than the hand proportion threshold is recorded as a valid image.

[0037] Furthermore, the process of constructing the pickup coordinates is as follows:

[0038] Divide the valid images into multiple valid image sets according to the shooting time , n represents the serial number of the valid image set. The valid image set contains multiple continuous valid images. The mean value of the hand area contour corresponding to each valid image in each valid image set is calculated and recorded as the distance judgment value of the valid image set. ;

[0039] The maximum value of the extracted distance judgment value is recorded as , there are j proportional coefficients preset , i is the serial number of the proportional coefficient from small to large, j is the total number of shelves for placing goods, each proportional coefficient corresponds to a layer sequence value, substitute it into the formula In the calculation, the distance reference value corresponding to each layer is obtained , for each valid image set A layer sequence judgment value is assigned, and the layer sequence judgment value is equal to the distance reference value subscript closest to the distance judgment value.

[0040] Furthermore, a plurality of analysis rectangles corresponding to each valid image set are drawn in a rectangular coordinate system, the mean of the horizontal coordinates of the center points of each analysis rectangle is calculated and recorded as a horizontal distance judgment value, the horizontal distance judgment value is compared with each column sequence reference value, and a column sequence judgment value is assigned to each valid image set, and the column sequence judgment value is equal to the column sequence value corresponding to the column sequence reference value closest to the horizontal distance judgment value;

[0041] Get the column order judgment value of the valid image set and sequence judgment value Composition of pickup coordinates .

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above system when executing the computer program.

[0043] A computer-readable storage medium stores a computer program, which implements the above system when executed by a processor.

[0044] Compared with the known prior art, the technical solution provided by the present invention has the following beneficial effects:

[0045] 1. The present invention is provided with a verification area, through which the user's purchased goods are statically identified, and static verification is added on the basis of the existing dynamic identification to generate orders. Compared with the product orders generated by only relying on dynamic identification in the prior art, it is more accurate. In addition, since the position of the verification area is fixed and the background is fixed, the accuracy of the image recognition algorithm in identifying goods can be greatly improved, thereby reducing the probability of order generation errors.

[0046] 2. On the one hand, the present invention first analyzes the placement areas of different commodities based on the placement video at the commodity placement stage, thereby constructing the placement coordinates of the commodities, so that the approximate position of each commodity can be determined through the placement coordinates, that is, the shelf layer number, the number of columns and the depth range of the commodity, thereby determining the placement coordinates of each commodity in the customer's shopping order; on the other hand, based on the pickup video, video analysis is performed on the customer's shopping and picking up actions to determine the commodity position corresponding to each time the customer picks up the commodity, thereby obtaining the pickup coordinates corresponding to the commodity picked up each time, and comparing the pickup coordinates with the placement coordinates to determine whether there is a one-to-one correspondence between the commodity picked up by the customer each time and the commodity in the shopping order, and then judging whether the order is generated incorrectly, which solves the problem in the prior art that it is difficult to verify the shopping order generated by artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is the overall module block diagram of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention will be further described below in conjunction with the embodiments.

[0051] See also Figure 1 , an unmanned vending system based on artificial intelligence, used for order building, verification and settlement in unmanned vending cabinets (hereinafter referred to as unmanned cabinets), including:

[0052] The main camera installed at the top of the unmanned container is used to shoot the bird's-eye view video of the customer picking up the goods. The auxiliary camera installed on the side of the unmanned container is used to shoot the side view video of the customer picking up the goods, as well as the multi-layer shelves for placing goods.

[0053] The customer interacts with the control panel of the unmanned vending machine through a mobile terminal (usually a mobile phone) and unlocks the door. After the door is opened, multiple cameras in the unmanned vending machine start recording video until the customer closes the door, and the door is automatically locked after it is closed.

[0054] It should be noted that the interaction methods between customers and the control panel of the unmanned vending machine through the mobile terminal include but are not limited to scanning QR codes, facial recognition and NFC device recognition. The customer's identity information can be quickly confirmed through the above interaction methods, so that the order information can be bound to the customer information, and the order details can be sent to the customer's mobile terminal and connected to the payment platform for subsequent consumption deductions. The above interaction methods are all existing technologies that have been widely used in the field of unmanned vending systems and will not be elaborated here.

[0055] Also includes:

[0056] The placement video processing unit places the goods on the shelves of the unmanned vending machine. When placing the goods, the staff will place them in different areas according to categories and place the same type of goods in the same column, thus obtaining multiple product placement areas with different layers and columns. When the staff is placing the goods, the placement video is recorded by the main camera and processed, including:

[0057] Sort the shelves from bottom to top and record them as commodity shelves , f is the serial number of the commodity shelf. For example, when f is equal to 1, it indicates the first shelf from the bottom up. The state where all commodity shelves are empty is recorded as the clear state. The image inside the cabinet captured by the main camera in the clear state is preset and recorded as the clear image;

[0058] It should be noted that each layer of shelves uses transparent glass partitions or metal mesh partitions, so that the main camera can capture the goods on the lower shelves through the empty upper shelves. In other words, the shelves themselves have little obstruction to the image acquisition of the main camera. When the upper shelves are empty, the main camera can directly capture images of the goods on the shelves below the shelves.

[0059] A comparison period is preset (in a specific embodiment, the comparison period is 1 second), and a frame image corresponding to the placed video is intercepted once every comparison period and compared with the clear image for similarity, and a comparison value is generated based on the comparison result. When the similarity of the two images is greater than a preset first similarity threshold, the comparison value is assigned a value of 1, and when the similarity of the two images is less than or equal to the preset first similarity threshold, the comparison value is assigned a value of 0, and a time series of comparison values ​​changing over time is constructed, and the time series is split into multiple zero-value sequences (the zero-value sequence only contains comparison values ​​with a value of 0, and the comparison values ​​are continuous) and sorted in chronological order, the zero-value sequence only contains comparison values ​​with a value of 0, and the last zero-value sequence is extracted and the corresponding time interval is obtained and recorded as the analysis time zone;

[0060] The frame images of the placement video captured within the analysis time zone are recorded as comparison images.

[0061] It should be noted that the collection process of the placement video is actively turned on and off by the staff. However, the placement video not only includes the actual placement process of the goods. By constructing an analysis time zone, it is possible to determine the time interval in the placement video during which the corresponding goods are actually placed, and then further analyze the placement video within this time interval to determine the location coordinates of different goods.

[0062] The coordinate construction unit constructs placement coordinates for each product placement area. Each placement coordinate corresponds to a type of product. Let the placement coordinates be ,in:

[0063] x represents the serial number of the commodity shelf corresponding to the commodity placement area;

[0064] y represents the same-layer column sequence corresponding to the product placement area;

[0065] z represents the row order of the same column corresponding to the product placement area;

[0066] d represents the width value corresponding to the product placement area;

[0067] The assignment process of placement coordinates is as follows:

[0068] Step 1: The vertical space area where the multi-layer shelf is located is recorded as the placement area, the space area between the shelf and the cabinet door is recorded as the observation area, the image located in the placement area in the comparison image is intercepted and recorded as the placement image, and the image located in the observation area in the comparison image is intercepted and recorded as the placement observation image;

[0069] A bright background plate is set at the bottom of the observation area to set off the display outline of the upper obstruction in the main camera. The placement observation map is grayed, and the number of pixels corresponding to different gray values ​​is calculated. A background gray interval is preset (corresponding to the gray value range of the bright background plate in the gray image). The percentage of the number of pixels whose gray values ​​are in the background gray interval is calculated and recorded as the background percentage. When the background percentage is less than or equal to the preset background percentage threshold, the placement observation map is marked as active. Otherwise, the placement observation map is marked as static. The static placement observation map is selected and step 2 is performed.

[0070] It should be noted that the background ratio is the area ratio of the bright background board in the placement observation map. By calculating the background ratio, it is possible to determine whether there is any obstruction above the bright background board. Specifically, when the staff place the goods, the difference between the static placement observation map and the active placement observation map lies in whether there is the outline of the staff's hand in the image. Normally, the bright background board will be blocked only when the staff puts their hands into the container.

[0071] Step 2: Same-layer comparison areas are set at the four corners of the placement area, and a same-layer correction set is constructed. The same-layer correction set contains images in the four same-layer comparison areas. The images in the same-layer correction set are recorded as same-layer calibration images. The images in the cleared image that are located in the four same-layer comparison areas are intercepted as the initial elements in the same-layer correction set, and step 3 is performed;

[0072] Step 3: intercept the images in the four same-layer comparison areas in the static placement observation map and record them as real-time calibration maps, divide the four real-time calibration maps into the same set to form a real-time proofreading set, each static placement observation map corresponds to a real-time proofreading set, extract any element in the real-time proofreading set and compare it with the corresponding element in the same-layer proofreading set (i.e., the real-time calibration map and the same-layer calibration map corresponding to the same same-layer comparison area) for similarity, when the similarity between the real-time calibration map and the same-layer calibration map corresponding to the same same-layer comparison area of ​​two elements is less than or equal to a preset second similarity threshold, record the real-time calibration map as a change map, and when the number of change maps in the real-time proofreading set is equal to 4, use the real-time proofreading set as a new same-layer proofreading set, and repeat steps 2 and 3;

[0073] It should be noted that by analyzing the selected static placement observation diagram, the influence of the staff's hands on the main camera's recognition can be avoided, so as to more accurately determine the column order, row order and width of different commodity placement areas in the shelf.

[0074] Step 4: Set a layer sequence value for each static placement observation map , assigning a layer sequence value based on the number of times the same-layer proofreading set is changed, the layer sequence value is equal to the number of times the same-layer proofreading set is changed, and the static placement observation images are divided into different image sets according to different layer sequence values;

[0075] It should be noted that when placing goods, the staff must strictly follow the order from bottom to top and from left to right so that the system can correctly identify and generate the placement coordinates. The minimum value of the same-layer proofreading set change number is 0, and the maximum value is equal to the number of shelf layers.

[0076] Step 5: Obtain the first static placement observation image in each image set with a layer sequence value greater than 0 as the goods placement image of the corresponding shelf. For example, the first static placement observation image in the image set with a layer sequence value of 1 is used as the goods placement image of the first shelf.

[0077] It should be noted that the first static placement observation image in the image set is the image after the shelf on that layer is filled with goods. Only when the goods fill up a shelf on that layer will the four real-time calibration images change compared to the calibration images on the same layer, causing the calibration set on the same layer to be replaced.

[0078] Step 6: Determine the column order, row order and width of different commodity placement areas in each shelf layer based on the corresponding cargo placement diagram of each shelf layer, where:

[0079] The types of goods and their outlines in the goods placement diagram are identified through an image recognition algorithm, and multiple area rectangles are constructed. Two sides of the area rectangles are parallel to the width of the shelf. Goods of the same type are divided into the same area rectangle. The size of the area rectangle is the smallest rectangle that can contain the outlines of all goods of the same type. Each area rectangle corresponds to a product placement area.

[0080] With the shelf width direction as the horizontal axis and the shelf depth direction as the vertical axis, a plane rectangular coordinate system is constructed, the coordinates of the center point of each area rectangle are obtained, and the horizontal coordinate distance between the center points of two adjacent area rectangles is calculated and recorded as the offset distance. When the offset distance is less than or equal to the preset offset threshold, the two adjacent area rectangles are recorded as rectangles in the same column, thereby obtaining multiple rectangle sets in the same column. Any two adjacent area rectangles in the rectangle set in the same column are rectangles in the same column. The rectangle sets in the same column are sorted according to the average value of the horizontal coordinates of the center points of all area rectangles in the set, and a column sequence value is assigned to each rectangle set in the same column. The column sequence value is equal to the sorting sequence number of the rectangle set in the same column. The average value of the horizontal coordinates of the center points of all area rectangles in the rectangle set in the same column is obtained and recorded as the column sequence reference value. The column sequence reference value corresponds to the column sequence value.

[0081] For example, if the mean of the horizontal coordinates of the center points of all regional rectangles in a set of rectangles in the same column is the smallest, then it is ranked 1 in all sets of rectangles in the same column, and the corresponding column sequence value is assigned 1.

[0082] Further, all the area rectangles in the same column rectangle set are sorted, and the area rectangles are sorted according to the size of the ordinate of the center point of the area rectangle, and a row sequence value is assigned to each area rectangle, and the row sequence value is equal to the sorting sequence number of the area rectangle in the same column rectangle set;

[0083] For example, if a region rectangle has the smallest value of the ordinate in the set of rectangles in the same column, it is ranked 1 in the set of rectangles in the same column, and the corresponding row sequence value is assigned 1.

[0084] Furthermore, the length value of the longitudinal axis of each area rectangle is obtained as the width value of the area rectangle.

[0085] Step 7: Get the layer sequence value, column sequence value, row sequence value and width value corresponding to each area rectangle, and use them as x, y, z, and d in the placement coordinates to get the placement coordinates .

[0086] Optionally, the cargo placement map may also be obtained by the staff actively taking photos during the placement process to ensure the accuracy of the cargo placement map.

[0087] The order generation unit is located on the top shelf inside the unmanned cabinet and is equipped with a verification area. The verification area is different from the commodity placement area on the shelf. It is used for customers to temporarily place the selected commodities in the verification area during the shopping process. There are also two verification cameras with a shooting angle of 90° at the top of the unmanned cabinet, which are used to take high-definition images of the commodities in the verification area from different angles. The types and quantities of commodities in the verification area are identified based on the image recognition algorithm, and a shopping order is constructed. All commodities in the shopping order are obtained and recorded as target commodities (multiple commodities of the same type are recorded as multiple target commodities, that is, commodities of the same type are not combined and marked).

[0088] It should be noted that by setting up a verification area to identify the goods purchased by users and generate product orders, static verification can be added on the basis of the existing dynamic verification to generate orders. Moreover, since the verification area has a fixed position and background, the accuracy of the image recognition algorithm in identifying goods can be greatly improved, thereby reducing the probability of order generation errors.

[0089] Purchase a verification unit, obtain a pickup video, intercept the pickup video in a comparison cycle to obtain multiple pickup images, distinguish the pickup images into static and active states (the distinction process is the same as step 1 in the coordinate construction unit), obtain the pickup image in the active state as the target image, and filter out the valid images in the target image. The screening process is as follows:

[0090] The image in the observation area of ​​the target image is intercepted and recorded as the pickup observation image. The bright background area in the pickup observation image is removed to obtain the hand area, and the outline of the hand area is drawn. A rectangle with the smallest area containing the outline of the hand area is constructed and recorded as the analysis rectangle. One side of the analysis rectangle is parallel to the width of the shelf. The area of ​​the analysis rectangle is set to , let the area of ​​the hand region contour be , substitute into the formula Calculate in and get the hand ratio value , a hand ratio threshold is preset, and the target images with hand ratio values ​​greater than the hand ratio threshold are screened out as valid images, and further analysis is performed based on the hand area contour in the valid image, where:

[0091] Divide the valid images into multiple valid image sets according to the shooting time , n represents the serial number of the valid image set. The valid image set contains multiple continuous valid images (continuous means that the shooting interval of the valid images is exactly equal to the comparison period). The mean value of the hand area contour corresponding to each valid image in each valid image set is calculated and recorded as the distance judgment value of the valid image set. , extract the maximum value of the distance judgment value and record it as , there are j proportional coefficients preset , i is the serial number of the proportional coefficient from small to large, j is the total number of shelves for placing goods, each proportional coefficient corresponds to a layer value (i.e. i=f), substitute it into the formula In the calculation, the distance reference value corresponding to each layer is obtained , for each valid image set A layer sequence judgment value is assigned, and the layer sequence judgment value is equal to the distance reference value subscript closest to the distance judgment value.

[0092] It should be noted that the proportional coefficient can be obtained by placing the same object at different layer heights and then calculating the image ratio when it is placed on the top layer. When the same customer picks up goods at different layers of shelves, the image display size of his arm in the observation area is different, and because the layer height is fixed, the size of the image corresponding to the same user's arm has a specific proportional relationship with the layer height, that is, the proportional coefficient. When the customer places the goods in the verification area, the area occupied by his arm in the observation area is the largest, corresponding to the maximum value of the distance judgment value.

[0093] Furthermore, multiple analysis rectangles corresponding to each valid image set are drawn in a plane rectangular coordinate system (consistent with the plane rectangular coordinate system in step six), the mean of the horizontal coordinates of the center points of each analysis rectangle is calculated and recorded as the horizontal distance judgment value, the horizontal distance judgment value is compared with each column sequence reference value, and a column sequence judgment value is assigned to each valid image set. The column sequence judgment value is equal to the column sequence value corresponding to the column sequence reference value that is closest to the horizontal distance judgment value.

[0094] Get the column order judgment value of the valid image set and sequence judgment value Composition of pickup coordinates , get the pickup coordinates of all valid image sets Form a pickup coordinate set;

[0095] Get the placement coordinates of each target product , extract the first two coordinates in the placement coordinates to get the range coordinates , get the range coordinates of all products Compose a shopping coordinate set.

[0096] The pickup coordinate set is compared with the shopping coordinate set. When the two coordinate sets are equal, an order verification correct signal is generated. When the two coordinate sets are not equal, an order verification error signal is generated and manual verification is introduced.

[0097] It should be noted that in order to ensure correct verification, the shopping rules of the unmanned vending machine limit the pickup of only one item at a time, so as to avoid the pickup coordinate set being different from the shopping coordinate set due to customers taking multiple items.

[0098] A computer device includes a memory and a processor. The memory stores a computer program. The processor implements the above system when executing the computer program.

[0099] A computer-readable storage medium stores a computer program, which implements the above system when executed by a processor.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. The unmanned vending system based on artificial intelligence is characterized by: include: The display video processing unit is used to crop and cut the product display video to obtain a comparison image; The coordinate construction unit records the vertical space area where the multi-layer shelf is located as the placement area, records the space area between the shelf and the cabinet door as the observation area, and captures the image of the comparison image located in the observation area as the placement observation map; Screening observation images based on the proportion of background area in the image observation area; Set the same-layer comparison areas at the four corners of the placement area, determine the shelf numbers corresponding to each placement observation map based on the image changes in the four same-layer comparison areas in the placement observation map, and select the goods placement map corresponding to each shelf layer; The screening process of the goods placement diagram corresponding to each shelf is as follows: Step 1: A bright background board is set at the bottom of the observation area, and the placement observation map is grayed out to calculate the number of pixels corresponding to different gray values. A background gray interval is preset, and the percentage of the number of pixels whose gray values ​​are within the background gray interval is calculated and recorded as the background percentage. When the background percentage is less than or equal to the preset background percentage threshold, the placement observation map is marked as active, otherwise, the placement observation map is marked as static, and the static placement observation map is selected and step 2 is performed; Step 2: Construct a same-layer calibration set, which includes four same-layer calibration images, and intercept the images in the four same-layer comparison areas in the cleared image as the initial elements in the same-layer calibration set, and proceed to step 3; Step 3: intercept the images in the four same-layer comparison areas in the placement observation map and record them as real-time calibration maps. Perform similarity comparison between the real-time calibration map and the corresponding same-layer calibration map in the same-layer proofreading set. When the similarity between the two is less than or equal to a preset second similarity threshold, record the real-time calibration map as a change map. When the number of change maps in the real-time proofreading set is equal to 4, use the real-time proofreading set as a new same-layer proofreading set and repeat steps 2 and 3. Step 4: Set a layer sequence value for the placement observation map, assign a layer sequence value based on the number of times the same-layer proofreading set is changed, the layer sequence value is equal to the number of times the same-layer proofreading set is changed, and divide the placement observation map into different image sets according to different layer sequence values; Step 5: Obtain the first placement observation image in the image set as the goods placement image of the corresponding shelf; The area where the same type of goods are placed is recorded as the goods placement area, and the column order, row order and width of the goods placement area are determined based on the distribution of different types of goods in the goods placement diagram; Construct placement coordinates based on the shelf number of the product placement area and the corresponding column sequence, row sequence and width; The order generation unit sets a verification area on the top shelf inside the unmanned cabinet, identifies the commodities in the verification area based on the image recognition algorithm and builds a shopping order, and obtains all commodities in the shopping order as target commodities; The purchase verification unit captures and filters the pickup video to obtain a valid image, obtains the hand area contour in the valid image for analysis, obtains multiple pickup coordinates, compares the placement coordinates corresponding to the target product with the pickup coordinates one by one, and generates a correct verification signal or an incorrect verification signal.

2. The unmanned vending system based on artificial intelligence according to claim 1, characterized in that: The process of placing the video processing unit to obtain the comparison image is as follows: The state in which all commodity shelves are empty is recorded as an empty state, and an image of the cabinet captured by the main camera in the empty state is preset and recorded as an empty image; A comparison cycle is preset. A frame image corresponding to the placed video is captured every comparison cycle and compared with the cleared image for similarity. A comparison value is generated based on the comparison result, where: When the similarity between the two images is greater than a preset first similarity threshold, the comparison value is assigned to 1; when the similarity between the two images is less than or equal to the preset first similarity threshold, the comparison value is assigned to 0; Construct a time series of comparison values ​​changing over time, split the time series into multiple zero-value sequences and sort them in chronological order. The zero-value sequence only contains comparison values ​​with a value of 0. Extract the last zero-value sequence and obtain the corresponding time interval as the analysis time zone. The frame images of the placement video captured within the analysis time zone are recorded as comparison images.

3. The unmanned vending system based on artificial intelligence according to claim 2, characterized in that: The process of determining the column order, row order and width of the product placement area is as follows: The types of goods and their outlines in the goods placement diagram are identified through an image recognition algorithm, and multiple area rectangles are constructed. Goods of the same type are divided into the same area rectangle. The area rectangle just contains the outlines of all goods of the same type. Each area rectangle corresponds to a product placement area. With the shelf width as the horizontal axis and the shelf depth as the vertical axis, a plane rectangular coordinate system is constructed to obtain the coordinates of the center point of each area rectangle, and the horizontal coordinate distance between the center points of two adjacent area rectangles is calculated and recorded as the offset distance. When the offset distance is less than or equal to the preset offset threshold, the two adjacent area rectangles are recorded as rectangles in the same column; Construct multiple sets of rectangles in the same column, any two adjacent area rectangles in the set are rectangles in the same column, sort the sets of rectangles in the same column according to the average value of the horizontal coordinates of the center points of all the area rectangles in the set, assign a column sequence value to each set of rectangles in the same column, the column sequence value is equal to the sorting sequence number of the set of rectangles in the same column, obtain the average value of the horizontal coordinates of the center points of all the area rectangles in the set of rectangles in the same column and record it as the column sequence reference value, the column sequence reference value corresponds to the column sequence value; Sort all the area rectangles in the same column rectangle set, sort the area rectangles according to the ordinate size of the center point of the area rectangle, and assign a row sequence value to each area rectangle, which is equal to the sorting sequence number of the area rectangle in the same column rectangle set; Get the length of the vertical axis of each area rectangle as the width of the area rectangle.

4. The unmanned vending system based on artificial intelligence according to claim 3 is characterized in that: The effective image acquisition process is as follows: The pickup video is captured with the comparison period as the interval period to obtain multiple pickup images. The pickup images are divided into static state and active state. The pickup images in the active state are obtained and recorded as target images. Based on the proportion of the hand area in the target image, the valid images in the target image are screened out.

5. The unmanned vending system based on artificial intelligence according to claim 4 is characterized in that: The effective image screening process is as follows: The image in the observation area of ​​the target image is intercepted and recorded as the pickup observation image. The bright background area in the pickup observation image is removed to obtain the hand area. The outline of the hand area is drawn, and a rectangle with the smallest area containing the outline of the hand area is constructed and recorded as the analysis rectangle. The area of ​​the analysis rectangle is , let the area of ​​the hand region contour be , substitute into the formula Calculate in and get the hand ratio value , a hand proportion threshold is preset, and the target image whose hand proportion value is greater than the hand proportion threshold is recorded as a valid image.

6. The unmanned vending system based on artificial intelligence according to claim 4, characterized in that: The process of constructing the pickup coordinates is as follows: Divide the valid images into multiple valid image sets according to the shooting time , n represents the serial number of the valid image set. The valid image set contains multiple continuous valid images. The mean value of the hand area contour corresponding to each valid image in each valid image set is calculated and recorded as the distance judgment value of the valid image set. ; The maximum value of the extracted distance judgment value is recorded as , there are j proportional coefficients preset , i is the serial number of the proportional coefficient from small to large, j is the total number of shelves for placing goods, each proportional coefficient corresponds to a layer sequence value, substitute it into the formula In the calculation, the distance reference value corresponding to each layer is obtained , for each valid image set A layer sequence judgment value is assigned, and the layer sequence judgment value is equal to the distance reference value subscript closest to the distance judgment value.

7. The unmanned vending system based on artificial intelligence according to claim 6, characterized in that: Draw multiple analysis rectangles corresponding to each valid image set in a rectangular coordinate system, calculate the mean of the horizontal coordinates of the center points of each analysis rectangle and record it as a horizontal distance judgment value, compare the horizontal distance judgment value with each column sequence reference value, and assign a column sequence judgment value to each valid image set, and the column sequence judgment value is equal to the column sequence value corresponding to the column sequence reference value closest to the horizontal distance judgment value; Get the column order judgment value of the valid image set and sequence judgment value Composition of pickup coordinates .

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the system according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the system according to any one of claims 1 to 7 is implemented.

Images