HBL intelligent vending cabinet order management platform

Abstract

The invention discloses an HBL intelligent vending cabinet order management platform, and relates to the field of automatic vending. According to the scheme, a commodity recognition result is evaluated from multiple dimensions by combining comprehensive confidence and comprehensive error calculation, and when the video recognition confidence is high, the video recognition result is directly adopted, and when complete recognition cannot be performed, the video recognition result is directly taken out; the commodity combination is determined according to the weight difference, the comprehensive confidence coefficient and the comprehensive error, the type and the number of single or multiple commodities can be accurately judged, the problem that the traditional image recognition technology cannot recognize the commodities under the conditions of commodity stacking and shielding is solved, the commodity recognition reliability is greatly improved, and the commodity recognition efficiency is improved. Compared with the prior art, the method reduces the risk of settlement omission caused by wrong recognition, reduces the dependence on the manual check of the customer service side, greatly shortens the comparison and selection period of the commodities, and improves the judgment accuracy and the disposal efficiency compared with the traditional complex commodity taking condition.

Description

Technical Field

[0001] The present invention relates to the field of automatic vending, and in particular to an HBL intelligent vending cabinet order management platform. Background Art

[0002] HBL smart vending machines are widely deployed in some public places. They occupy a small space and adopt an unattended operation mode to reduce labor costs and operating costs. The current HBL smart vending machines have improved operational efficiency through intelligent management systems and accurate commodity identification technology, reduced errors and losses caused by human factors, and helped to increase overall revenue. However, the current HBL smart vending machines are vulnerable to some malicious behaviors, such as taking out multiple commodities in a stacked state, resulting in some commodities being mostly or completely obscured. This poses a great challenge to the commodity determination mechanism that relies on image recognition technology. At this time, it is often necessary to rely on the back-end customer service end to process and verify the specific commodity combination through the customer's pickup video in real time. The customer service end judges the type of commodity taken out by the user in the video based on experience. This method not only has the defects of long time consumption and low efficiency, but also has a good accuracy for partially exposed commodities, but it is often difficult to control the accuracy of judgment for completely obscured commodities, and it is very easy to miss settlements, resulting in direct economic losses. Summary of the invention

[0003] The purpose of the present invention is to propose an HBL intelligent vending cabinet order management platform in order to solve the problem of automatic vending.

[0004] In order to achieve the above-mentioned object, the present invention adopts the following technical scheme: an HBL smart vending cabinet order management platform, comprising a client end for remotely managing the smart vending cabinet system, and also comprising a video retrieval unit and a weight verification module communicating with the vending cabinet system, wherein the video retrieval unit is used to retrieve video data and extract key video frames, and the weight verification module is used to record the change in the weight difference ΔW of the goods before and after picking up the goods from the vending cabinet;

[0005] The output end of the video acquisition module is electrically connected to a video judgment module, which is used to extract a set of commodities that are clearly identified in the key video frame and obtain the confidence scores of each commodity set. (i) ;

[0006] The output ends of the video judgment module and the weight verification module are electrically connected to a commodity verification module, and the commodity verification module determines the quantity of commodities according to the commodity weight difference ΔW. After the commodity quantity is determined, the comprehensive confidence C i After the comprehensive error E is calculated, the specific type and quantity of goods to be taken out are determined and fed back to the customer service.

[0007] As a further description of the above technical solution: the judgment logic of the product proofreading module is:

[0008] When the video recognition score (i) Greater than or equal to the preset confidence threshold T v , then the image recognition algorithm based on convolutional neural network is directly adopted to obtain the video recognition result;

[0009] When video recognition cannot fully identify all products, the comprehensive confidence C i Value, select the product combination;

[0010] If there is a possibility of multiple items, and the combined solution satisfies ΔW, then according to the comprehensive error E and comprehensive confidence C of all candidate solutions i Evaluate and select the one with the smallest comprehensive error E and C i The combination with the highest sum is sent as output to the client.

[0011] As a further description of the above technical solution: the video retrieval unit retrieves video data to extract key video frames in the following method:

[0012] When the weight verification module detects a change in the weight of the product, the video data is marked with an initial frame, and when the cabinet door is closed, the video data is marked with an end frame;

[0013] Candidate frames are screened based on the initial frame and the end frame, and step a is:

[0014] a1. For each pair of adjacent frames F t and F t-1 , and calculate its pixel difference by the following formula:

[0015] D(t)=∑ x，y |F t (x, y)-F t-1 (x, y)|;

[0016] Where D(t) represents the difference measure between two adjacent frames at time t, and F t (x, y) and F t-1 (x, y) represents the pixel values ​​of the current frame and the previous frame at the coordinate (x, y) at time t;

[0017] a2. If D(t) exceeds the preset value T1, F t Mark as candidate frames, and then quickly filter out frames with significant changes between the initial frame and the end frame;

[0018] Based on the candidate frame, the motion vector between the previous and next frames is calculated by the optical flow method, and step b is:

[0019] b1. The motion vector between the frames before and after the candidate frame is calculated as:

[0020]

[0021] Where M(t) represents the average motion vector magnitude at time t, represents the motion vector at coordinates (x, y), The motion vector The amplitude of , N is the total number of pixels;

[0022] b2. If M(t) exceeds the preset value T2, it is determined that the candidate frame contains the movement information when picking up the goods;

[0023] b3. Select the frame with the most prominent motion vector between the previous and next candidate frames per second as the key frame.

[0024] As a further description of the above technical solution: the weight verification module includes the following steps:

[0025] When the goods are put into the cabinet, the standard weight W of each product i is 1 Pre-enter and ensure that the weight error of product i is within the error tolerance of the weight sensor within ±δ w Inside;

[0026] When the cabinet door is opened, record the initial total weight of the goods W 0 , then the calculation process of the weight difference ΔW is:

[0027] ΔW=W 0 -W 1 ;

[0028] The total weight W 0 Resets each time the door is opened.

[0029] As a further description of the above technical solution: the video judgment module extracts the clearly identified product set in the key frame through the convolutional neural network image recognition algorithm, and obtains the confidence score of each product set. (i) , and the confidence score (i) The value is between 0 and 1.

[0030] As a further description of the above technical solution: the method for the commodity proofreading module to determine the quantity of commodities by the commodity weight difference ΔW is:

[0031] After the video judgment module extracts the commodity set through the video frame, the weight verification module verifies through the weight difference ΔW, and the commodity proofreading module makes a judgment. If the number of commodities extracted is a certain number, the following formula is satisfied:

[0032] ΔW=∑ i (ω(i) *n i )+ε

[0033] Among them, n i is the quantity of commodity i taken out, ω (i) is the standard weight of the i-th commodity, ε is the sensor error, and |ε|≤δ w ;

[0034] Based on the above formula, if a single item is taken out, then the single item i satisfies:

[0035] |ΔW-ω (i) |≤δ w

[0036] Based on this, it can be determined that what is taken out is a single item;

[0037] If multiple items are taken out, the number of items taken out for the i-th item is n. i , and satisfy the following formula:

[0038] ΔW≈∑ i (ω (i) *n i );

[0039] Integer n i Indicates the number of the i-th commodity taken out, which is used to calculate the total weight ΔW of the actual goods taken out and compare it with the theoretical weight ω (i) Compare to determine the combination of goods to be taken out.

[0040] As a further description of the above technical solution: when the product proofreading module determines the quantity of products, if the weight difference ΔW of multiple products is greater than the weight of a single product, it is necessary to find a set of integer solutions n (i) , this scheme uses a greedy algorithm to combine all candidate solutions, and the solution is n (i) , in order to quantify the degree of matching, the error is defined according to the following formula:

[0041] E=|ΔW-∑ i (ω (i) *n i )|;

[0042] Among them, the error E should be ≤δ w .

[0043] As a further description of the above technical solution: the comprehensive confidence C of the product proofreading module i The calculation method is:

[0044] For each product i, the data is integrated and the comprehensive confidence C is calculated. i , find the optimal product combination:

[0045]

[0046] Where ΔW and ω (i) When it is close, the value is larger, otherwise the value decays rapidly. α and β are weight coefficients, which are adjusted according to the video clarity and the stability of the weight sensor, and socre (i) is the confidence of product i in video recognition.

[0047] As a further description of the above technical solution: the output end of the customer service end is electrically connected to an order data retrieval module, an order data export module, an order details viewing module, an order data correction module, an order closing processing module, a user authority management module and an order collection management module.

[0048] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0049] This solution evaluates the product recognition results from multiple dimensions by combining comprehensive confidence and comprehensive error calculation. When the video recognition confidence is high, it is directly adopted. When it cannot be fully recognized, the product combination is determined based on the weight difference, comprehensive confidence and comprehensive error. Whether it is a single item or multiple items, the type and quantity can be accurately judged. This overcomes the problem that traditional image recognition technology cannot recognize stacked or obscured items, greatly improves the reliability of product recognition, and reduces the risk of settlement omissions due to recognition errors.

[0050] This method also reduces the reliance on manual verification on the customer service side. Compared with the traditional complex pickup situation, it often takes time to combine products and make judgments based on experience, and there is often a blank period for user payment settlement. By providing highly reliable specific product types and quantity information and feeding it back to the customer service side for verification, it greatly shortens the product comparison and selection cycle and improves judgment accuracy and handling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the platform architecture of the present invention;

[0052] Figure 2 This is a logic block diagram of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] like Figure 1 - Figure 2 As shown, the present invention provides: an HBL smart vending cabinet order management platform, including a client end for remotely managing the smart vending cabinet system, characterized in that: it also includes a video retrieval unit and a weight verification module that communicate with the vending cabinet system, the video retrieval unit is used to retrieve video data and extract key video frames, and the weight verification module is used to record the change of the weight difference ΔW of the goods before and after the vending cabinet is picked up;

[0055] This method uses the video retrieval unit and the weight verification unit to extract key video frames and monitor weight changes respectively, providing the platform with a real-time basic data with strong timeliness. It extracts key video frames through video data and can implement screening in advance. Combined with the monitoring, identification and calculation of weight data, it greatly reduces the amount of calculation in the subsequent product identification process, effectively improves the efficiency of order handling, and achieves rapid response to the customer service end.

[0056] The output end of the video retrieval module is electrically connected to a video judgment module, which is used to extract a set of clearly identified commodities in the key video frame, and obtain the confidence scores of each commodity set through the convolutional neural network image recognition algorithm of the video retrieval module. (i) ;

[0057] The video judgment module identifies video data in key video frames. As a mature and deployed technology, this step has guaranteed efficiency and accuracy. Combined with the recognition method of key video frames, its recognition speed is further improved, and the confidence of various products can be accurately and quickly judged.

[0058] The output ends of the video judgment module and the weight verification module are electrically connected to a commodity verification module, and the commodity verification module determines the quantity of commodities according to the commodity weight difference ΔW. After the commodity quantity is determined, the comprehensive confidence C i After the comprehensive error E is calculated, the specific type and quantity of goods to be taken out are determined and fed back to the customer service.

[0059] Regardless of the number of commodity types, the core idea of ​​this scheme is to infer the actual number of commodities taken out through the linear combination of weight change and standard weight of commodities. Therefore, this scheme will not fail due to the increase of commodity types. When commodities are put into the existing HBL smart vending cabinet, due to the limited space, the number of commodities provided is often limited, and the number of commodities put into the cabinet is improved. Therefore, this algorithm can cope with the normal use of HBL smart vending cabinet. However, for larger supermarkets, there are dozens or hundreds of commodity types. In this case, this scheme will use the greedy algorithm combination to solve n (i)The strategy needs to be replaced with a more efficient method. For example, the existing dynamic programming, backtracking pruning, heuristic algorithms, or the combination of prior information can be used to optimize the combined solution process, thereby achieving more efficient and accurate judgments in actual systems in different scenarios.

[0060] The order data retrieval module is connected to the customer service end and supports basic condition searches, such as organization, business order number, user information and time range. It can also perform advanced condition searches, such as container code, payment status and refund status searches.

[0061] The order data export module is connected to the client and supports exporting orders or product information under the above search conditions.

[0062] The order details viewing module is connected to the customer service end to view the basic information of the order, including shopping videos, product details, product adjustment details and customer complaint details. The product adjustment details mainly refer to adjustments to the product caused by product non-availability, unfriendly product, product quantity adjustment, refund, etc.

[0063] The order data correction module is connected to the customer service end. When an order has problems such as product not being on the shelf, unfriendly behavior, and customer complaints, it is necessary to adjust the product data, such as product category, quantity, price, etc. The order can be corrected through the order data correction module. At the same time, if a transaction has been generated at the wrong product price, an additional refund can be made.

[0064] The order closing processing module is used to automatically save when the customer service closes the order.

[0065] The user rights management module can blacklist designated accounts and reject consumption from unfriendly accounts under the operation of the client.

[0066] The order collection management module is connected to the customer service end. If there is an order in the payment status, a text message containing a payment link is sent to the user via SMS to remind the user to pay, or the payment information is pushed online through the app link, and the number of collection reminders is recorded each time.

[0067] When the goods are put into the cabinet, the standard weight W of each product i 1 Pre-enter and ensure that the weight error of product i is within the error tolerance of the weight sensor within ±δ w Inside;

[0068] When the cabinet door is opened, record the initial total weight of the goods W 0 , and use convolutional neural network image recognition algorithms to perform video recognition and record the customer pickup process;

[0069] After the pickup is completed, the weight verification module records the final weight W 1 , and calculate the weight difference ΔW:

[0070] ΔW=W 0 -W 1 ;

[0071] The video data is processed into video frames by a video retrieval unit, and the video retrieval unit retrieves the video data to extract key video frames in a method as follows:

[0072] When the weight verification module detects a change in the weight of the product, the video data is marked with an initial frame, and when the cabinet door is closed, the video data is marked with an end frame;

[0073] Candidate frames are screened based on the initial frame and the end frame, and step a is:

[0074] a1. For each pair of adjacent frames F t and F t-1 , and calculate its pixel difference by the following formula:

[0075] D(t)=∑ x，y |F t (x, y)-F t-1 (x, y)|;

[0076] Where D(t) represents the difference measure between two adjacent frames at time t, and F t (x, y) and F t-1 (x, y) represents the pixel values ​​of the current frame and the previous frame at the coordinate (x, y) at time t;

[0077] a2. If D(t) exceeds the preset value T1, F t Mark as candidate frames, and then quickly filter out frames with significant changes between the initial frame and the end frame;

[0078] Based on the candidate frame, the motion vector between the previous and next frames is calculated by the optical flow method, and step b is:

[0079] b1. The motion vector between the frames before and after the candidate frame is calculated as:

[0080]

[0081] Where M(t) represents the average motion vector magnitude at time t, represents the motion vector at coordinates (x, y), The motion vector The amplitude of , N is the total number of pixels;

[0082] b2. If M(t) exceeds the preset value T2, it is determined that the candidate frame contains the movement information when picking up the goods;

[0083] b3. Select the frame with the most prominent motion vector between the previous and next candidate frames per second as the key frame.

[0084] The key video frames are sent to the video judgment module, and the convolutional neural network image recognition algorithm is used to extract the set of products that are clearly identified during the action process, and the confidence score of each product set is obtained. (i) , and the confidence score (i) The value is between 0 and 1;

[0085] After the video judgment module extracts the commodity set through the video frame, the weight verification module verifies through the weight difference ΔW, and the commodity proofreading module makes a judgment. If the number of commodities extracted is a certain number, the following formula is satisfied:

[0086] ΔW=∑ i (ω (i) *n i )+ε

[0087] Among them, n i is the quantity of commodity i taken out, ω (i) is the standard weight of the i-th commodity, ε is the sensor error, and |ε|≤δ w ;

[0088] Based on the above formula, if a single item is taken out, then the single item i satisfies:

[0089] |ΔW-ω (i) |≤δ w ;

[0090] Based on this, it can be determined that what is taken out is a single item;

[0091] If multiple items are taken out, the number of items taken out for the i-th item is n. i , and satisfy the following formula:

[0092] ΔW≈Σ i (ω (i) *n i ); If the weight difference ΔW is greater than the weight of a single product, then we need to find a set of integer solutions n (i) , this scheme uses a greedy algorithm combination to solve n (i) , in order to quantify the degree of matching, the error is defined according to the following formula:

[0093] E=|ΔW-∑ i (ω (i) *n i )|;

[0094] Among them, the error E should be ≤δ w;

[0095] For each product i, the data is integrated and the comprehensive confidence C is calculated. i , find the optimal product combination:

[0096]

[0097] Where ΔW and ω (i) When the distance is close, the value is larger, otherwise the value decays rapidly. α and β are weight coefficients, which are adjusted according to the video clarity and the stability of the weight sensor. (i) is the confidence level of product i in video recognition (if the product is not recognized, the value is 0);

[0098] Through comprehensive confidence evaluation, the types and quantities of goods to be taken out are determined. By introducing the calculation of comprehensive confidence, combined with the confidence of image recognition and the matching degree of weight calculation, the recognition results are comprehensively evaluated. This method considers the accuracy of product recognition from multiple dimensions, further improves the reliability of determining the types and quantities of goods to be taken out, and reduces the error impact that may be caused by a single factor.

[0099] When the video recognition score (i) Greater than or equal to the preset confidence threshold T v , then the image recognition algorithm based on convolutional neural network is directly adopted to obtain the video recognition result;

[0100] When video recognition cannot fully identify all products, it compares the C i value, select the most likely product combination;

[0101] If there is a possibility of multiple products (for example, the combined solution satisfies ΔW), the comprehensive error E and comprehensive confidence of all candidate solutions are evaluated, and the solution with the smallest error and C is selected. i The combination with the highest sum is sent as output to the client.

[0102] Compared with the traditional method of directly handling all the products manually through the customer service end when the products cannot be identified, this method selects the best product combination by combining the confidence calculation of the product combination weight and the comparison of the comprehensive error. The customer service end does not need to spend time comparing the exposed corners of the obscured products to make judgments. By using weight records and calculations, combined with the standard weight of the products, whether it is a single product or multiple products, the number of products to be taken out can be determined more accurately through the set rules and calculation methods. For multiple products, the number can be accurately judged by finding an integer solution combination that meets the weight difference condition, avoiding omissions or errors that may occur when manually counting the number.

[0103] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An HBL smart vending cabinet order management platform, comprising a client terminal for remotely managing a smart vending cabinet system, characterized in that: It also includes a video retrieval unit and a weight verification module that communicate with the vending cabinet system, wherein the video retrieval unit is used to retrieve video data and extract key video frames, and the weight verification module is used to record the change in the weight difference ΔW of the goods before and after the goods are picked up from the vending cabinet; The output end of the video acquisition module is electrically connected to a video judgment module, which is used to extract a set of commodities that are clearly identified in the key video frame and obtain the confidence scores of each commodity set. (i) ; The output ends of the video judgment module and the weight verification module are electrically connected to a commodity verification module, and the commodity verification module determines the quantity of commodities according to the commodity weight difference ΔW. After the commodity quantity is determined, the comprehensive confidence C i After the comprehensive error E is calculated, the specific type and quantity of goods to be taken out are determined and fed back to the customer service.

2. The HBL smart vending machine order management platform according to claim 1, characterized in that: The judgment logic of the product proofreading module is: When the video recognition score (i) Greater than or equal to the preset confidence threshold T v , then the image recognition algorithm based on convolutional neural network is directly adopted to obtain the video recognition result; When video recognition cannot fully identify all products, the comprehensive confidence C i Value, select the product combination; If there is a possibility of multiple items, and the combined solution satisfies ΔW, then according to the comprehensive error E and comprehensive confidence C of all candidate solutions i Evaluate and select the one with the smallest comprehensive error E and C i The combination with the highest sum is sent as output to the client.

3. The HBL smart vending machine order management platform according to claim 1, characterized in that: The video retrieval unit retrieves video data to extract key video frames in the following method: When the weight verification module detects a change in the weight of the product, the video data is marked with an initial frame, and when the cabinet door is closed, the video data is marked with an end frame; Candidate frames are screened based on the initial frame and the end frame, and step a is: a1. For each pair of adjacent frames F t and F t-1 , and calculate its pixel difference by the following formula: D(t)=∑ x，y [F t (x,y)-F t-1 (x,y)|; Where D(t) represents the difference measure between two adjacent frames at time t, and F t (x, y) and F t-1 (x, y) represents the pixel values ​​of the current frame and the previous frame at the coordinate (x, y) at time t; a2. If D(t) exceeds the preset value T1, F t Mark as candidate frames, and then quickly filter out frames with significant changes between the initial frame and the end frame; Based on the candidate frame, the motion vector between the previous and next frames is calculated by the optical flow method, and step b is: b1. The motion vector between the frames before and after the candidate frame is calculated as: Where M(t) represents the average motion vector magnitude at time t, represents the motion vector at coordinates (x, y), The motion vector The amplitude of , N is the total number of pixels; b2. If M(t) exceeds the preset value T2, it is determined that the candidate frame contains the movement information when picking up the goods; b3. Select the frame with the most prominent motion vector between the previous and next candidate frames per second as the key frame.

4. The HBL smart vending machine order management platform according to claim 2, characterized in that: The weight verification module comprises the following steps: When the goods are put into the cabinet, the standard weight W1 of each product i is pre-entered, and the weight error of product i is ensured to be within the error tolerance of the weight sensor within ±δ w Inside; When the cabinet door is opened, the initial total weight of the goods W0 is recorded, and the calculation process of the weight difference ΔW is: ΔW=W0-W1; The total weight W0 is reset each time the cabinet door is opened.

5. The HBL smart vending machine order management platform according to claim 1, characterized in that: The video judgment module extracts the clearly identified product set in the key frame through the convolutional neural network image recognition algorithm, and obtains the confidence score of each product set. (i) , and the confidence score (i) The value is between 0 and 1.

6. The HBL smart vending machine order management platform according to claim 4, characterized in that: The method for the commodity checking module to determine the quantity of commodities by the commodity weight difference ΔW is as follows: After the video judgment module extracts the commodity set through the video frame, the weight verification module verifies through the weight difference ΔW, and the commodity proofreading module makes a judgment. If the number of commodities extracted is a certain number, the following formula is satisfied: ΔW=∑ i (oh (i) *n i )+e; Among them, n i is the quantity of commodity i taken out, ω (i) is the standard weight of the i-th commodity, ε is the sensor error, and |ε|≤δ w ; Based on the above formula, if a single item is taken out, then the single item i satisfies: |ΔW-ω (i) |≤δ w ; Based on this, it can be determined that what is taken out is a single item; If multiple items are taken out, the number of items taken out for the i-th item is n. i , and satisfy the following formula: ΔW≈∑ i (oh (i) *n i ); Integer n i Indicates the number of the i-th commodity taken out, which is used to calculate the total weight ΔW of the actual goods taken out and compare it with the theoretical weight ω (i) Compare to determine the combination of goods to be taken out.

7. The HBL smart vending machine order management platform according to claim 6, characterized in that: When the product verification module determines the quantity of products, if the weight difference ΔW of multiple products is greater than the weight of a single product, it is necessary to find a set of integer solutions n (i) , this scheme uses a greedy algorithm to combine all candidate solutions, and the solution is n (i) , in order to quantify the degree of matching, the error is defined according to the following formula: E=|ΔW-∑ i (oh (i) *n i )|; Among them, the error E should be ≤δ w .

8. The HBL smart vending machine order management platform according to claim 7, characterized in that: The comprehensive confidence C of the product proofreading module i The calculation method is: For each product i, the data is integrated and the comprehensive confidence C is calculated. i , find the optimal product combination: Where ΔW and ω (i) When the distance is close, the value is larger, otherwise the value decays rapidly. α and β are weight coefficients, which are adjusted according to the video clarity and the stability of the weight sensor. (i) is the confidence of product i in video recognition.

9. The HBL smart vending machine order management platform according to claim 1, characterized in that: The output end of the customer service end is electrically connected to an order data retrieval module, an order data export module, an order details viewing module, an order data correction module, an order closing processing module, a user authority management module and an order collection management module.

Images