Product identification method for smart vending machine purchase checkout

By deploying a dual-camera system on the smart vending machine, combining depth information and product layer height information, and employing border detection, feature filtering, and product trajectory tracking technologies, the problems of high false detection rate and high classification error rate in existing technologies have been solved, achieving accurate product identification and settlement.

CN118072208BActive Publication Date: 2026-11-17BEIJING INHAND NETWORKS TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211412962.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-11
Publication Date
2026-11-17
Estimated Expiration
2042-11-11

AI Technical Summary

Technical Problem

Existing smart vending machine visual recognition technology suffers from high false detection rates, high classification error rates, and incomplete recognition by a single camera, resulting in inaccurate purchase behavior recognition results.

Method used

Employing a dual-camera system, with the main and secondary cameras capturing video separately, and combining depth information and product layer height information, an accurate purchase behavior list is generated through border detection, feature filtering, product trajectory tracking, and classification models.

Benefits of technology

It reduced the false detection rate, improved the accuracy and completeness of product identification, and ensured accurate settlement of purchase transactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118072208B_ABST
    Figure CN118072208B_ABST
Patent Text Reader

Abstract

This invention discloses a product identification method for purchase settlement in a smart vending machine. The method includes: deploying a main camera and a secondary camera on the smart vending machine; the main camera capturing and generating a main video; the secondary camera capturing and generating a secondary video; after the purchaser completes the purchase, the main video and the secondary video are transmitted to a remote server; the remote server performs dynamic visual recognition on the main video and the secondary video respectively; and generates a purchase behavior list based on the dynamic recognition results; calculates the depth information of the products; calculates the product layer information based on the product depth information, and integrates the product layer information into the purchase behavior list; and verifies the purchase behavior list containing the product layer information according to preset product layer height information; to generate the final purchase behavior list for product settlement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of vending machines and visual recognition technology, and in particular to a product identification method for purchasing and settling transactions in smart vending machines. Background Technology

[0002] With the continuous development of vending machine technology, door-opening vending machines have gradually become the most common type of vending machine in the unmanned retail sector. Compared to the older button-based product selection method, the method of consumers actively opening the door to retrieve goods significantly improves the shopping experience. It also reduces the hardware complexity requirements of vending machines, allowing for the arrangement of different product varieties within the same vending machine.

[0003] Currently, automatic vending machines typically use visual recognition to analyze consumer purchasing behavior, determining the quantity and price of purchased items. Based on this information, the transaction is then processed and payment deducted from the purchase price on a remote management platform. Accurately identifying the type and quantity of goods actually taken by the customer is a crucial technical step for ensuring accurate payment during the purchase process.

[0004] Currently, the main solution for dynamic visual intelligent vending machines primarily relies on capturing video of the user's purchase to identify the purchased item, typically using a single video stream. The identification process involves the following steps: First, using object detection algorithms, determine if a product is being purchased in each frame. If so, determine the product's image position and category within the current frame. Second, based on the movement of the purchased item across multiple video frames, construct its trajectory on the timeline to determine the user's action—whether it's a picking-up or putting-back action. Third, based on the user's actions in the video, determine the final purchase information and complete the relevant payment process. Existing solutions have the following drawbacks: First, relying solely on object detection algorithms to determine the location and category of the purchased item easily leads to false detections and classification errors. This is mainly because vending machines are open environments, consumers' clothing varies greatly, some clothing patterns resemble product details, and users often hold mobile phones while purchasing. This makes it easy for the detector to misidentify objects such as mobile phones, clothing patterns, watches, and shoes as products, resulting in identification errors. Furthermore, directly using detectors to distinguish product categories becomes increasingly difficult as the variety of products increases, often leading to classification errors. Secondly, directly using multi-frame detectors to construct product movement trajectories at purchase locations along a timeline is prone to trajectory confusion. If a false detection occurs within the vending machine's camera area at a particular moment, it will cause incorrect purchase trajectories, leading to confusion and ultimately incorrect purchase results. Thirdly, using only a single camera to determine purchase behavior is susceptible to errors in final purchase information, especially under conditions of obstruction. Uncertain user purchase behavior, such as a user picking up multiple items with one hand, can result in only one item being identified in a single-angle video shot, leading to incorrect purchase information.

[0005] Therefore, there is a need in the existing technology for a dynamic visual recognition method for purchasing behavior that can be applied to the self-service purchasing behavior of smart vending machines for accurate product identification. Summary of the Invention

[0006] The technical objective of this invention is to reduce the impact of detector false detections on the final recognition result in dynamic visual recognition methods, improve the classification accuracy of recognition algorithms in dynamic visual recognition methods, and simultaneously reduce the impact of incomplete recognition in single-camera solutions on the final recognition result.

[0007] Based on the above-mentioned technical objectives, the present invention provides a product identification method for purchase and settlement in smart vending machines, the method comprising:

[0008] A main camera and a secondary camera are deployed on the smart vending machine; the main camera captures and generates a main video; the secondary camera captures and generates a secondary video; after the purchaser completes the purchase, the main video and the secondary video are transmitted to a remote server;

[0009] The remote server performs dynamic visual recognition on the main video and the secondary video respectively; and generates a purchase behavior list based on the dynamic recognition results.

[0010] Calculate the depth information of the product; calculate the number of product layers based on the depth information, and integrate the number of product layers into the purchase behavior list; verify the purchase behavior list containing the number of product layers based on the preset product layer height information; and generate the final purchase behavior list for product settlement.

[0011] The process of calculating the depth information of the product includes:

[0012] First, extract the product trajectory generated during the dynamic visual recognition process in the secondary video;

[0013] Obtain the frame image data of the trajectory starting frame of each product trajectory; the frame image data is the border coordinate data of the product in the trajectory starting frame;

[0014] Generate a list of starting frames based on the frame image data of the starting frame of each product trajectory.

[0015] The depth information of each product is calculated based on the list of starting frames, specifically using the following formula:

[0016] z = (w * f) / p

[0017] Where z represents the depth information of the product, the focal length f is known from the camera parameters, p is the pixel width of the current product, and w is the product appearance size corresponding to the pre-determined product category.

[0018] In one embodiment, the process of verifying the purchase behavior list containing product layer information based on preset product layer height information refers to verifying the product SKU in the trajectory classification result of each product in the purchase list and its corresponding calculated product layer information, and verifying whether the correspondence is consistent with the preset product layer height information. If they are consistent, the product trajectory classification result is retained in the final purchase behavior list R; if they are inconsistent, the product trajectory classification result is removed from the final purchase behavior list R and marked as pending verification.

[0019] In one embodiment, the product trajectory classification and identification includes:

[0020] Determine the video data for which dynamic visual recognition will be performed;

[0021] Motion detection is performed on the video data to obtain the start frame t1 corresponding to the door opening time and the end frame t2 corresponding to the door closing time in the video data.

[0022] Step 1 is executed, starting from the initial frame, the border detector is used to detect borders in the current frame image; the border detector is used to obtain the border coordinate information of the goods in the current frame image;

[0023] Execute step 2, and extract a partial sub-image of the product based on the border coordinates in the border detection results detected in the previous steps;

[0024] Step 3 involves performing feature filtering on the product sub-images to remove invalid product border coordinate information corresponding to the product sub-images.

[0025] Step 4 involves inputting the current frame image and the filtered product border coordinates into the product trajectory tracker for product trajectory tracking.

[0026] Perform step 5 to filter features for each product trajectory to obtain feature values ​​for each product trajectory;

[0027] Perform step 6 to input the feature values ​​into the product classification model to obtain product SKU information and classification confidence.

[0028] Execute step 7 to determine the purchase behavior type of each product trajectory based on the preset image auxiliary lines;

[0029] Step 8: Generate a list of purchase behaviors based on the trajectory classification results of all product trajectories.

[0030] In one embodiment, the bounding box detector is used to obtain the bounding box coordinate information of the product in the current frame image; the bounding box detector is constructed using R-CNN, Fast R-CNN, Faster R-CNN, YOLO or SSD bounding box detection algorithms.

[0031] In one embodiment, removing invalid product partial sub-images in step 3 refers to removing product partial sub-images that are not products sold by the smart vending machine.

[0032] In one embodiment, the product classification model in step 6 uses an OSNet, TransReid, Van, ConvNext, or VIT model. In another embodiment, the product classification model in step 6 uses a gallery image library for Reid classification, where the gallery image library consists of pre-collected product images representing categories supported by the current algorithm model.

[0033] In one embodiment, the image auxiliary line is a dividing line set in the video image to determine whether a user takes or puts back an item. The image auxiliary line divides the video image into two areas: one area is the vending machine's product area, and the other area is the buyer's area. If the current trajectory starts in the vending machine's product area and ends in the buyer's area, it is a purchase behavior; if the current trajectory starts in the buyer's area and ends in the vending machine's product area, it is a put-back behavior.

[0034] Compared with the prior art, one or more embodiments of the present invention may have the following inventive points and advantages:

[0035] To reduce false detections, the technical solution of this invention adds a feature filter to avoid the influence of items such as mobile phones, clothing patterns, and watches on the detection results. To make classification more accurate, an image classification module is used, which only uses the detector to obtain the current item sub-image, processes it through the feature filter, and then uses a specific image classification model to classify the goods. To avoid chaotic motion trajectories, target tracking technology is used to form the product trajectory, preventing falsely detected items from interfering with the purchase trajectory. Dual cameras are used to recognize user purchase behavior, reducing the possibility of errors in purchase information.

[0036] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 This is a schematic diagram of the purchase and settlement process of the intelligent vending machine of this invention;

[0039] Figure 2 This is a schematic diagram of the commodity trajectory classification and recognition process according to the first embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the start frame recognition process according to the first embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the end frame recognition process according to the first embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the product layer number calculation process according to the first embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] Before proceeding with the detailed description below, it may be necessary to define certain words and phrases used throughout this invention. The terms “coupled,” “connected,” and their derivatives refer to any direct or indirect communication or connection between two or more elements, regardless of whether those elements are physically in contact with each other. The terms “transmit,” “receive,” and “communicate,” and their derivatives cover both direct and indirect communication. The terms “comprising” and “including,” and their derivatives refer to including, but not limited to, other than. The term “or” is inclusive, meaning and / or. The phrase “associated with,” and its derivatives refer to including, being contained within, interconnecting, containing, being included in, being connected or connected to, being coupled or coupled to, communicating with, cooperating, intertwining, juxtaposing, proximate, binding or binding to, having, having attributes, having a relationship or being related to, etc. The term “controller” refers to any device, system, or part thereof that controls at least one operation. Such a controller may be implemented in hardware, or a combination of hardware and software and / or firmware. The functionality associated with any particular controller may be centralized or distributed, local or remote. The phrase "at least one" when used with a list of items means that different combinations of one or more of the listed items may be used, and that only one item from the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, A and B and C.

[0045] In this invention, the description of the first and second terminals of a resistor, capacitor, or inductor is merely to distinguish the two connection terminals of the device, facilitating the description of the device's connection relationship with other devices. It does not specifically specify which terminal of the resistor, capacitor, or inductor is actually in use. Those skilled in the art should understand that in actual circuit construction, either terminal of a resistor, capacitor, or inductor can be defined as the first terminal, and when the first terminal is defined, the other terminal of the device is automatically designated as the second terminal.

[0046] In this invention, the use of terms such as "first," "second," "third," etc., to describe various components or elements is solely for distinguishing each component and expressing the distinct relationships between them. The descriptive methods themselves do not imply any association between components. For example, the appearance of only "first" and "third" does not imply the existence of a "second" component; rather, the descriptions of "first" and "third" simply indicate the presence of two different, independent components.

[0047] The definitions of other specific words and phrases provided throughout this invention are consistent with those of ordinary skill in the art. It will be understood that, in many, if not most, cases, such definitions apply to the prior and future use of words and phrases thus defined.

[0048] In this invention, the application combination of modules and the hierarchical division of sub-modules are for illustrative purposes only. Without departing from the scope of this disclosure, the application combination of modules and the hierarchical division of sub-modules can be in different ways.

[0049] Example 1

[0050] like Figure 1 The diagram shown illustrates the purchase and settlement process of the intelligent vending machine of the present invention. The purchase and settlement process includes:

[0051] The system can detect when a customer opens the smart vending machine door using various devices installed on the door, such as mechanical contacts, high-precision sliding rheostats, or three-axis gyroscopes.

[0052] Once the cabinet door is detected to be open, the main camera and secondary camera on the smart vending machine are activated, capturing images of the cabinet door area from different angles. For example, the main camera is installed diagonally above the cabinet door, capturing the buyer's actions of taking and placing goods from a downward angle; while the secondary camera is installed directly above the cabinet door, capturing the buyer's actions of taking and placing goods from a downward angle.

[0053] After the cabinet door is detected to be closed, the main video captured by the main camera and the secondary video captured by the secondary camera are uploaded to the remote server.

[0054] The remote server performs dynamic visual recognition on the main video and the secondary video respectively; and generates a purchase behavior list based on the dynamic recognition results.

[0055] Calculate the depth information of the product; calculate the number of product layers based on the product depth information, and integrate the number of product layers into the purchase behavior list;

[0056] The remote server verifies the purchase behavior list containing product layer information based on the preset product layer height information;

[0057] Based on the inspection results, confirm the type and quantity of goods taken by the buyer, and complete the payment settlement.

[0058] In this embodiment, dynamic visual recognition is performed on the main video and the secondary video respectively to obtain a purchase behavior list R1 for the main video and a purchase behavior list R2 for the secondary video; then the purchase behavior list R1 and the purchase behavior list R2 are merged to obtain a merged purchase behavior list R.

[0059] like Figure 2 As shown, in this embodiment, the dynamic visual recognition process for the main video and the secondary video includes:

[0060] First, determine the video data for which dynamic visual recognition will be performed.

[0061] Motion detection is performed on the video data to obtain the start frame t1 corresponding to the door opening time and the end frame t2 corresponding to the door closing time. The motion detection process is as follows: Figure 3 and Figure 4 As shown, the motion detection process for the starting frame includes:

[0062] Get the grayscale image F1 of frame t and the grayscale image F2 of frame t+1;

[0063] ANDing the two frames F1 and F2 yields the difference image D. It's worth noting that when the two frames are identical, the difference image is a matrix of all zeros.

[0064] To suppress the effects of light or other noise, a threshold p is used for threshold filtering. This involves setting all pixel values ​​in the difference image D below the threshold p to 0, and setting the values ​​of pixels above the threshold p to 1. This yields a rough motion region image F3 of the moving target.

[0065] Connectivity analysis is performed on the motion region image. This connectivity analysis refers to obtaining the difference images between adjacent frames, that is, using mathematical morphological operations to seal the edges of the motion region in F3, making it continuous and complete. The mathematical morphological operations can include operations such as erosion or dilation.

[0066] Calculate the area of ​​the connected motion region. If the area is greater than the motion threshold P2, then the current frame is considered to be the frame at the moment when the gate state changes.

[0067] Motion detection for the last frame is similar to the process described above, except that motion detection is performed in reverse from the last frame.

[0068] After determining the start frame t1 and end frame t2 of the video data;

[0069] Step 1 is executed, starting from the start frame, the border detector is used to detect borders in the current frame image; the border detector is used to obtain the border coordinate information of the goods in the current frame image.

[0070] For example, the content of acquiring the t-th frame image of the video data is and t1<t≤t2, w×h is the resolution of the video. Then a bounding box detector is used to detect the content of the t-th frame image to perform bounding box detection, and the obtained bounding box detection result of the current frame is expressed as

[0071] wherein, d i =[x1,y1,x2,y2,c,p], 0≤i<n. (x1,y1) represents the pixel coordinates of the top-left corner of the detected commodity bounding box, (x2,y2) represents the pixel coordinates of the bottom-right corner of the detected commodity bounding box, p is the recognition category of the detection algorithm, c is the confidence score of the detection algorithm, and the confidence score reflects the possibility that the current detected bounding box [x1,y1,x2,y2] contains a target object and the possibility that it belongs to the category p. n represents the number of bounding boxes detected by the current detector. The bounding box detector may use bounding box detection algorithms such as R-CNN, Fast R-CNN, Faster R-CNN, YOLO series, and ssd.

[0072] In this embodiment, the bounding box detector only distinguishes between positive and negative samples, and does not distinguish the SKU information corresponding to each commodity bounding box, that is, the recognition category p is always object. In other words, the bounding box detector only determines whether there is a commodity picked up by the user in the content of the current frame. If there is, the category is object, otherwise no output is generated.

[0073] Subsequently, step 2 is executed, and a local sub-image of the commodity is intercepted according to the bounding box coordinates in the bounding box detection result detected in the previous step.

[0074] Subsequently, step 3 is executed, feature filtering is performed on the local commodity sub-images to remove the commodity bounding box coordinate information corresponding to invalid local commodity sub-images; that is, a feature filter is used to filter out the content falsely detected by the detector. This includes: feeding n local sub-images into the feature filter to determine whether they are commodities. A sub-image is output if it is a commodity, otherwise it is discarded. The commodity matrix A obtained after feature filtering is A=[a o …a j , where 0≤j≤n. The feature filter may use deep learning classification models such as EfficientNetV2, ConvNeXt, RegNet, and VGG.

[0075] Subsequently, step 4 is executed, and the current frame image and the filtered commodity bounding box coordinate information are input into a commodity trajectory tracker for commodity trajectory tracking; specifically, j pieces of local image data in the commodity matrix A=[a o …a j are sequentially fed into the tracker for target tracking, which mainly includes the following two steps:

[0076] I. Obtain each local image a i Corresponding tracking information β i = [β_x1,β_y1,β_x2,β_y2,λ]. Where (β_x1,β_y1) represents the top-left pixel coordinates of the detected product border in the current frame image, and (β_x2,β_y2) represents the bottom-right pixel coordinates of the detected product border in the current frame image. λ represents the trajectory ID number formed by the current rectangle.

[0077] 2. Update the j tracking results at the current moment to the results with different trajectory IDs, and obtain the trajectory results E = [E1…E1…E2] for the j items. j The trajectory results for each ID are listed in chronological order, encompassing the location and local image data of the product corresponding to the current ID at different times. The product trajectory tracker can be implemented using target tracking algorithms such as deepsort and Bytetrack. For example, the trajectory result for the product with trajectory ID λ is E. λ =[e t-i …e t ]. Where e t-i For the tracking results at time ti, e t This represents the tracking result at time t. i =[β_x1,β_y1,β_x2,β_y2,c i (β_x1,β_y1) represents the top-left pixel coordinates of the detected product border in the current frame image, and (β_x2,β_y2) represents the bottom-right pixel coordinates of the detected product border in the current frame image.

[0078] It's important to emphasize that the number of products is equal to the number of trajectories, and each product corresponds to a unique ID. If, at the next time step, the number of products increases from j to j+1, the number of trajectories will also increase accordingly. For example, if the tracker inputs product A and product B at time t, it will generate two trajectories with IDs 1 and 2 respectively. Trajectory 1 corresponds to the movement trajectory of product A, and trajectory 2 corresponds to the movement trajectory of product B. At time t+1, if the tracker inputs product A, product B, and product C, it will additionally output trajectory 3. That is, trajectory 1 still corresponds to product A, trajectory 2 to product B, and trajectory 3 to product C.

[0079] Steps 1 to 3 are performed on each frame of the image sequentially until the end frame t2; thus obtaining the motion trajectories E = [E1…E2] of k items. k ].

[0080] Subsequently, step 5 is executed to perform feature filtering for each product trajectory to obtain feature values ​​specific to that product trajectory. Specifically, feature filtering is performed using the position of each trajectory ID at different times, selecting the top θ features with the largest values, and obtaining the feature selection value H = [h0…h0] for each trajectory ID. θ-1 The so-called feature filtering refers to using the coordinates [β] corresponding to the current trajectory ID at different times. t _x1,β t _y1,β t _x2,β t [_y2], by calculating the area, select the x local image data with the largest area. The area calculation formula is s t =[(β t _x2-β t _x1)×(β t _y2-β t _y1)).

[0081] For example, let θ = 3, the tracking result corresponding to trajectory id λ is:

[0082]

[0083] Assume that the area obtained through calculation at different times is Then the feature selection value is

[0084] Subsequently, step 6 is executed, inputting the feature values ​​into the product classification model to obtain product SKU information and classification confidence. Specifically, this involves processing the θ feature local image data H = [h0…h...]. θ-1 The system performs product classification, obtaining the product SKU information corresponding to the current trajectory ID and the trajectory classification confidence score. The classification model can use OSNet, TransReID, VAN, ConvNext, or VIT models based on ReID technology.

[0085] It's important to note that each local feature data point corresponds to a classification result SKU and a corresponding classification confidence score. The classification confidence score refers to the probability that the current local feature data belongs to the current classification result. Therefore, the θ feature local image data points H = [h0…h...] θ-1 The corresponding classification result is Among them, c i For specific product SKU information, s i For c i The corresponding classification confidence level.

[0086] If all θ class SKUs are consistent, then the current SKU is taken as the classification result for the trajectory; if none of the θ class SKUs are consistent, then the SKU with the highest confidence is selected as the classification result for the trajectory; otherwise, the SKU with the most repetitions among the θ SKUs is selected as the classification result for the trajectory. The trajectory classification confidence ζ = ∑sku / θ, where ∑sku is the sum of all confidence scores of the trajectory SKU in the classification result C.

[0087] Subsequently, step 7 is executed, determining the purchase behavior type of the product trajectory based on preset image auxiliary lines, i.e., whether it is a take-up or return behavior. The image auxiliary line is a dividing line set in the video image to determine whether the user takes or returns the product. The image auxiliary line divides the video image into two areas: one area is the vending machine product area, and the other area is the buyer's area. If the current trajectory starts in the vending machine product area and ends in the buyer's area, it is a purchase behavior. If the current trajectory starts in the buyer's area and ends in the vending machine product area, it is a return behavior; all others are invalid behaviors.

[0088] Repeat steps 5 to 7 above to obtain the trajectory classification results of all product trajectories. The trajectory classification results include product SKU information, trajectory classification confidence, and purchase behavior type.

[0089] Then, step 8 is executed to generate a list of purchase behaviors based on the trajectory classification results of all product trajectories. Specifically, all trajectory classification results are divided into two groups according to the purchase behavior type in the trajectory classification results. One group is the purchase behavior group, which is the trajectory classification result of all purchase behaviors that are taking behaviors, and the other group is the return behavior group, which is the trajectory classification result of all purchase behaviors that are return behaviors.

[0090] Then, the two sets of data are filtered. This filtering involves comparing each trajectory classification result in the purchase behavior group with the trajectory classification results in the replacement behavior group. If the product SKU information of a trajectory classification result in the purchase behavior group matches the product SKU information of a trajectory classification result in the replacement behavior group, then that trajectory classification result is deleted from the purchase behavior group, and simultaneously, the trajectory classification result with the matching product SKU information is deleted from the replacement behavior group. The trajectory classification results in the purchase behavior group after this data filtering process constitute the purchase behavior list.

[0091] In this embodiment, the purchase behavior list R1 and purchase behavior list R2 are merged to obtain the final purchase behavior list R. Specifically: if the purchase quantities identified in purchase behavior list R1 and purchase behavior list R2 are inconsistent, that is, the number of trajectory classification results in the purchase behavior group that has completed data filtering is inconsistent, then the result with the larger quantity is selected for output. If the quantities are the same but the categories are inconsistent, the result with the higher trajectory classification confidence is selected for output. Otherwise, purchase behavior list R1 is selected as the output as the final purchase behavior list R.

[0092] like Figure 5 As shown, the process of calculating the number of product layers and verifying the dynamic visual recognition results and the depth information of the products based on the preset product layer height information in this embodiment includes:

[0093] First, extract the product trajectory generated during the dynamic visual recognition process in the secondary video;

[0094] Obtain the frame image data of the trajectory starting frame of each product trajectory; the frame image data is the border coordinate data of the product in the trajectory starting frame;

[0095] Generate a list of starting frames φ based on the frame image data of the starting frame of each product trajectory, i.e.: φ = [φ0…φ i ], 1≤k≤i.

[0096] The depth information of each product is calculated based on the starting frame list φ, specifically using the following formula.

[0097] z = (w * f) / p:

[0098] Where z represents the product depth information, the focal length f is determined from the camera parameters, and p is the pixel width of the current product. w represents the product's external dimensions corresponding to a pre-determined product category. Then, using the height range information of each shelf layer, the specific shelf number corresponding to the current product's depth information z is calculated to obtain the product's shelf number information.

[0099] The calculated product layer information is integrated into the final purchase behavior list R.

[0100] The purchase list R is checked using preset product layer height information. The check refers to checking the product SKU and its corresponding calculated product layer number information in the trajectory classification result of each product in the purchase list. The check is to check whether the correspondence is consistent with the preset product layer height information. If they are consistent, the product trajectory classification result is retained in the final purchase behavior list R; if they are inconsistent, the product trajectory classification result is removed from the final purchase behavior list R and marked as pending verification.

[0101] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A product identification method for purchase and settlement in smart vending machines, characterized in that, The method includes: A main camera and a secondary camera are deployed on the smart vending machine; the main camera captures and generates a main video; the secondary camera captures and generates a secondary video; after the purchaser completes the purchase, the main video and the secondary video are transmitted to a remote server; The remote server performs dynamic visual recognition on the main video and the secondary video respectively; and generates a purchase behavior list based on the dynamic recognition results. Calculate the depth information of the product; calculate the number of product layers based on the depth information, and integrate the number of product layers into the purchase behavior list; verify the purchase behavior list containing the number of product layers based on the preset product layer height information; and generate the final purchase behavior list for product settlement. The process of calculating the depth information of the product includes: First, extract the product trajectory generated during the dynamic visual recognition process in the secondary video; Obtain the frame image data of the trajectory starting frame of each product trajectory; the frame image data is the border coordinate data of the product in the trajectory starting frame; Generate a list of starting frames based on the frame image data of the starting frame of each product trajectory. The depth information of each product is calculated based on the list of starting frames, specifically using the following formula: in, For product depth information, focal length According to the camera parameters, The pixel width of the current product. The physical dimensions of the product corresponding to a predetermined product category.

2. The product identification method for purchase settlement in intelligent vending machines according to claim 1, characterized in that, The process of verifying the purchase behavior list containing product layer information based on the preset product layer height information refers to verifying the product SKU in the trajectory classification result of each product in the purchase list and the calculated product layer information corresponding to the product SKU in the trajectory classification result. The verification is to check whether the calculated product layer information corresponding to the product SKU in the trajectory classification result is consistent with the preset product layer height information. If they are consistent, the product trajectory classification result is retained in the final purchase behavior list R. If there is a discrepancy, the product trajectory classification result will be removed from the final purchase behavior list R and marked as pending verification.

3. The product identification method for purchase settlement in intelligent vending machines according to claim 1, characterized in that, The product trajectory classification and recognition includes: Determine the video data for which dynamic visual recognition will be performed; Motion detection is performed on the video data to obtain the start frame t1 corresponding to the door opening time and the end frame t2 corresponding to the door closing time in the video data. Step 1 is executed, starting from the initial frame, the border detector is used to detect borders in the current frame image; the border detector is used to obtain the border coordinate information of the goods in the current frame image; Execute step 2, and extract a partial sub-image of the product based on the border coordinates in the border detection results detected in the previous steps; Step 3 involves performing feature filtering on the product sub-images to remove invalid product border coordinate information corresponding to the product sub-images. Step 4 involves inputting the current frame image and the filtered product border coordinates into the product trajectory tracker for product trajectory tracking. Perform step 5 to filter features for each product trajectory to obtain feature values ​​for each product trajectory; Perform step 6 to input the feature values ​​into the product classification model to obtain product SKU information and classification confidence. Execute step 7 to determine the purchase behavior type of each product trajectory based on the preset image auxiliary lines; Step 8: Generate a list of purchase behaviors based on the trajectory classification results of all product trajectories.

4. The product identification method for purchase settlement in intelligent vending machines according to claim 3, characterized in that, The bounding box detector is used to obtain the bounding box coordinate information of the product in the current frame image; the bounding box detector is constructed using R-CNN, Fast R-CNN, Faster R-CNN, YOLO or SSD bounding box detection algorithms.

5. The product identification method for purchase settlement in intelligent vending machines according to claim 3, characterized in that, The removal of invalid product sub-images in step 3 refers to removing product sub-images that are not products sold by the smart vending machine.

6. The product identification method for purchase settlement in intelligent vending machines according to claim 3, characterized in that, The product classification model in step 6 uses the osnet, transreid, van, convnext, or vit model.

7. The product identification method for purchase settlement in intelligent vending machines according to claim 3, characterized in that, In step 6, the product classification model uses a gallery image library for re-id classification. The gallery image library consists of product images of categories supported by the current algorithm model that have been collected in advance.

8. The product identification method for purchase settlement in intelligent vending machines according to claim 3, characterized in that, The image auxiliary line is a dividing line set in the video image to determine whether a user takes or puts back an item. The image auxiliary line divides the video image into two areas: one area is the vending machine's product area, and the other area is the buyer's area. If the current trajectory starts in the vending machine's product area and ends in the buyer's area, it is a purchase behavior; if the current trajectory starts in the buyer's area and ends in the vending machine's product area, it is a put-back behavior.

9. A smart vending machine, characterized in that, include: The system includes at least one cabinet door, a main camera and a secondary camera; and at least one control unit; the control unit uses the product identification method for purchase settlement of smart vending machines as described in any one of claims 1-8 to settle the purchase.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform one of the product identification methods for purchasing and settling transactions at a smart vending machine as described in 1-8.

Citation Information

Patent Citations

  • Automatic vending recognition method based on dynamic vision

    CN113743530A

  • Dynamic identification system and method for goods in vending cabinet, medium, equipment and terminal

    CN115294325A