Shopping order identification method and system based on vision and gravity sensing

Abstract

The invention provides a shopping order identification method and system based on vision and gravity sensing. The method comprises the following steps: detecting commodity information in a video stream of a shopping area of a vending machine and position information of hand key points; according to the position information of the key points of the hand at different moments, calculating the angular velocity of hand motion, judging whether a hand rotation action on the commodity exists or not, calculating finger distance change, the relative motion velocity of the two hands and the relative motion direction of the two hands, and judging whether a tearing action on a commodity packaging bag exists or not; if the hand rotation action or the tearing action exists, and the weight data on the goods selling shelf is reduced, it is confirmed that the shopping abnormal behavior occurs. According to the method, the visual data and the weight data of the vending machine shopping area are combined, abnormal behaviors in the shopping process are detected and recognized, the problems that a pure visual algorithm cannot solve the commodity number problem and a pure weight algorithm cannot recognize the commodity category are solved, and the shopping order recognition precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image algorithm recognition, and more specifically, to a shopping order recognition method and system based on vision and gravity sensing. Background Art

[0002] With the development of technology and changes in consumer demand, vending machines, as a convenient retail method, have been widely used in various public places, such as shopping malls, airports, subway stations, etc. Vending machines realize the storage, display and sales of goods through automated means, greatly improving sales efficiency and reducing operating costs.

[0003] At present, a large number of sales orders are generated every day in the market. The recognition of these orders has not yet been fully automated. Some simple cases can be recognized by algorithms, while others require manual processing. Low efficiency and insufficient accuracy have always been the pain points of the industry. It is urgent to design a new recognition solution to solve this problem. Summary of the invention

[0004] Pure vision and pure gravity cannot well solve the problem of vending application scenarios. The present invention provides a shopping order recognition method and system based on vision and gravity sensing.

[0005] According to a first aspect of the present invention, there is provided a shopping order recognition method based on vision and gravity sensing, comprising: Get the video stream of the vending machine shopping area; Detecting commodity information in the video stream based on the target detection model, and acquiring position information of key points of the hand within a set time period based on the hand detection model; Calculating the hand movement angular velocity according to the position information of the hand key points at different times; and calculating the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the hand key points of the two hands; Judging whether there is a hand rotation action on the commodity according to the hand movement angular velocity, and judging whether there is a tearing action on the commodity packaging bag according to the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands; If there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf decreases, it means that the shopping behavior is abnormal and an abnormal alarm is issued.

[0006] According to a second aspect of the present invention, there is provided a shopping order recognition system based on vision and gravity sensing, comprising: An acquisition module, used to acquire a video stream of a shopping area of ​​a vending machine; A detection module, used to detect commodity information in the video stream based on a target detection model, and to obtain position information of key points of a hand within a set time period based on a hand detection model; A calculation module, used to calculate the hand movement angular velocity according to the position information of the hand key points at different times; and to calculate the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the hand key points of the two hands; A judgment module, used to judge whether there is a hand rotation action on the commodity according to the hand movement angular velocity, and to judge whether there is a tearing action on the commodity packaging bag according to the change of the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands; The confirmation module is used to confirm that abnormal shopping behavior has occurred and issue an abnormal alarm if there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf is reduced.

[0007] The present invention provides a shopping order recognition method and system based on vision and gravity sensing, which detects the commodity information and the position information of the key points of the hand in the video stream of the shopping area of ​​the vending machine; according to the position information of the key points of the hand at different times, the angular velocity of the hand movement is calculated to determine whether there is a hand rotation action on the commodity, and the change in the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands are calculated to determine whether there is a tearing action on the commodity packaging bag; if there is a hand rotation action or a tearing action, and the weight data on the shelf is reduced, it is confirmed that abnormal shopping behavior has occurred. The present invention combines the visual data and weight data of the shopping area of ​​the vending machine to detect and identify the shopping orders in the shopping process, solves the problem that the pure visual algorithm cannot solve the problem of identifying the number of commodities, and solves the problem that the pure weight algorithm cannot identify the category of commodities, improves the accuracy of automatic shopping order recognition, and reduces recognition cost and labor cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flow chart of a shopping order recognition method based on vision and gravity sensing provided by the present invention; Figure 2 A schematic diagram of the process of identifying purchase behavior based on the combination of visual perception and gravity sensing; Figure 3 A schematic diagram of the structure of a shopping order recognition system based on vision and gravity sensing provided by the present invention. DETAILED DESCRIPTION

[0009] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0010] Figure 1 A flow chart of a shopping order recognition method based on vision and gravity sensing provided by the present invention, such as Figure 1 As shown, the method includes: Step 1, obtain the video stream of the vending machine shopping area.

[0011] It is understandable that two cameras are usually installed on the top and upper right side of the vending machine to obtain video images in the shopping area. After obtaining the video captured by the top camera, the cabinet door area is detected and a virtual cabinet door is drawn to facilitate the identification and judgment of the goods entering and exiting the back.

[0012] Step 2: Detect the commodity information in the video stream based on the target detection model, and obtain the position information of the key points of the hand within a set time period based on the hand detection model.

[0013] It is understandable that after step 1 obtains the video stream image of the vending machine shopping area, the product information is identified from the video stream image based on the target detection model, mainly the category information of the product is identified; and the position information of the hand key points within a period of time is detected from the video stream based on the hand detection model. The hand key points mainly include the position information of the wrist center point and each finger. According to the position information of the hand key points, it is determined whether the hand has a rotating action on the product, or whether there is a tearing action on the product packaging bag.

[0014] Step 3, calculating the angular velocity of the hand movement according to the position information of the hand key points at different times; and calculating the change in finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the hand key points of the two hands.

[0015] It is understandable that the position information of the key points of the hand is obtained through deep learning models such as MediaPipe Hands. In the process of detecting the bottle cap twisting action, it is necessary to pay attention to the position information of the center point of the wrist. and the position information of the second joint of the index finger .

[0016] According to the position information of the center point of the wrist and the position information of the second joint point of the index finger, calculate the direction of the hand:

[0017] According to the hand direction, the hand direction angle is calculated by the inverse tangent function arctan:

[0018] Where t represents the timestamp or frame index of the current frame.

[0019] In order to detect rotation, it is necessary to calculate the rate of change of the hand's orientation angle over time, i.e., the angular velocity , let the time interval between two adjacent frames be , the angular velocity ω is defined as the change in angle per unit time: .

[0020] Since the video stream may contain noise, directly using the above formula to calculate the hand movement angular velocity may lead to misjudgment. Therefore, the present invention introduces a smoothing process and a dynamic threshold setting mechanism.

[0021] The moving average filter is used to smooth the angular velocity to reduce the impact of instantaneous noise. Assuming the window size is n, the smoothed hand angular velocity :

[0022] in, is the angular velocity of the hand at the i-th moment.

[0023] After calculating the angular velocity of the hand movement, the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands are calculated according to the position information of the hand key points of the two hands. The calculation of the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands is mainly to prepare for the subsequent hand tearing action recognition.

[0024] Specifically, first, the position information of the thumb and index finger tips of each hand is recorded from the video stream image. , where j represents different fingers.

[0025] Based on the wrist center and the second joint of the index finger , the hand orientation is determined by calculating the vector between these two points:

[0026] The hand facing angle can be calculated using the arctan function:

[0027] Calculate the initial distance between the index fingertips of the left and right hands

[0028] in , They are the positions of the index fingertips of the left and right hands respectively.

[0029] Changes in distance between fingers: monitoring If the distance remains stable within a certain period of time (such as 0.5 seconds), it is considered that the positioning has been completed.

[0030] The rapid relative movement stage of the two hands mainly includes the following points: (1) Pull the product packaging bag open quickly: Pull the bag open quickly in opposite directions with both hands, causing the bag to tear at the contact point.

[0031] (2) Relative speed of the two hands: For the key point k of the hand, the speed at time t It can be calculated by the following formula: ,in Represents the position vector of joint point k at time t.

[0032] Pay special attention to the relative velocity Δv between the two index fingers. When Δv increases significantly, it indicates that the two hands are performing rapid relative movement.

[0033] (3) Tear detection: Use a visual detection model (such as yolov5) to detect changes in the packaging appearance. Once cracks or other abnormal shapes are found, it is confirmed that a tear has occurred. The process of tearing the product packaging bag mainly includes the above three processes. Therefore, it is necessary to calculate the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands based on the position information of the key points of the two hands.

[0034] Record the initial positions of the two hands, and the distance between the thumbs and index fingers of the left and right hands in the previous frame (i.e. the initial frame). In the process of tearing the product bag, the left and right hands will stretch in opposite directions, so we can determine whether they are moving in opposite directions by calculating the relative movement direction of the two hands. This detection can be achieved using the vector angle: For the calculation of finger distance, remember the positions of the left thumb and index finger tips as follows: (thumb tip vector), The positions of the thumb and index finger tips of the right hand are: (thumb tip vector), (Index finger tip vector). Calculate the distance between the thumb and index finger of each hand and the distance between the left thumb and the right thumb:

[0035]

[0036]

[0037] in, is the distance between the thumb and index finger of the left hand, The distance between the thumb and index finger of the right hand, is the distance between the left thumb and the right thumb, Represents the Euclidean norm of a vector (that is, the straight-line distance between two points).

[0038] Calculation of the relative speed of the two hands: The relative speed Δv refers to the speed change between the fingertips of the left and right hands, reflecting the relative movement of the two hands. The relative speed between the two hands can be calculated by the following formula:

[0039] in, The time interval between frames (usually in seconds).

[0040] If Δv increases significantly, and the distance between the left and right thumbs If it gradually increases, it may have entered the tearing action stage.

[0041] Calculation of the relative movement direction of the two hands during the tearing process: During the tearing process of the product bag, the left and right hands will stretch in opposite directions, so the relative movement direction of the two hands can be calculated to determine whether they are moving in opposite directions. This detection can be achieved using the vector angle.

[0042] Specifically, the motion vector between the left and right hands in the previous frame and the current frame is calculated, where the motion vector of the left index finger is , the motion vector of the right index finger , is the position vector of the left index finger in the current frame, is the position vector of the left index finger in the previous frame, is the position vector of the right index finger in the current frame, is the position vector of the right index finger in the previous frame.

[0043] Then, calculate the angle between the motion vector of the left index finger and the motion vector of the right index finger :

[0044] If the angle If it is close to 180°, it means that the two fingers are stretching in opposite directions, indicating that a tearing action is occurring.

[0045] Step 4, judging whether there is a hand rotation action on the commodity based on the hand movement angular velocity, and judging whether there is a tearing action on the commodity packaging bag based on the change in the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands.

[0046] It is understandable that step 3 calculates the angular velocity of the hand movement, and the hand angular velocity can be used to determine whether the hand is rotating the product. First, the angular velocity dynamic threshold can be set. The angular velocity dynamic threshold can be set to 60 degrees per second and fine-tuned according to actual conditions. Set the initial angular velocity dynamic threshold .

[0047] When the smoothed angular velocity of the hand movement exceeds the set dynamic threshold, it is considered that a significant rotational movement has occurred.

[0048] Detection conditions for hand rotation: If there are m consecutive frames (such as 5 frames) that meet the following conditions, rotation detection is triggered:

[0049] And if the image algorithm detects that the product is a drink at the current moment, it can be determined that there may be a stealing behavior. Through the above detailed calculation method, the rotation of the hand can be detected more accurately, and the robustness of the system is improved through smoothing and dynamic threshold setting.

[0050] For the tearing detection of product packaging bags, the tearing action is confirmed by combining the distance change between the fingers, the relative speed and the reverse movement of the two hands through the following conditions: The distance between the thumbs and index fingers of the left and right hands is very close and remains stable for a period of time.

[0051] Next, the relative speed Δv between the left and right fingertips increases significantly.

[0052] The vector angle is close to 180°, that is, the movement directions of the index fingers of both hands are basically opposite.

[0053] Visual detection (such as the YOLO model) confirms that the item in the hand is a bagged snack.

[0054] Verification of commodity weight data in the cabinet.

[0055] If the above conditions are met at the same time, it can be confirmed that the tearing action is in progress.

[0056] In summary, the final determination of hand tearing action includes: Assume that the following parameters are calculated between the current frame and the previous frame: Initial distance: , ,

[0057] Relative speed: Δv Vector Angle:

[0058] The tearing action can be judged by the following judgment conditions:

[0059] in, is the relative speed between the left thumb and the right thumb, is the angle between the motion vector of the left index finger and the motion vector of the right index finger, is the distance between the thumb and index finger of the left hand, is the distance between the thumb and index finger of the right hand, is the distance between the left thumb and the right thumb, is the velocity dynamic threshold. In the pixel coordinate system, the relative velocity threshold It should be adjusted according to the image resolution, frame rate and motion characteristics. For low-resolution images, it is recommended to set the threshold to 5-10 pixels / frame, and for high-resolution images, it can be set to 10-15 pixels / frame. a, b, and c are all distance thresholds. Usually, a, b, and c are all 5.

[0060] By combining the change in finger distance, relative speed calculation, and angle detection of the two hands' reverse motion, the action of tearing bagged goods by hand can be accurately identified. This method not only improves recognition accuracy, but also effectively combines visual and motion data to form a multi-level detection framework.

[0061] Step 5: If there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf decreases, abnormal shopping behavior occurs and an abnormal alarm is issued.

[0062] It is understandable that when the hand's rotation or tearing action on the goods is identified through visual processing of the video stream, it can basically be determined that there is abnormal behavior of stealing food or drinking. However, in order to further improve the accuracy of the rotation action of twisting the bottle cap or the tearing action of the product packaging bag, a weight sensor is added to monitor the weight change of the goods in the cabinet in real time. After judging the existence of the rotation action of the goods or the tearing action of the product packaging bag through the hand movement characteristics, combined with the weight change of the goods on the shelf, when the weight of the goods on the shelf decreases, it means that stealing food and drinking has indeed occurred.

[0063] In summary, the above detailed action decomposition and feature extraction methods can more accurately identify the action of tearing bagged goods by hand and provide reliable input for computer vision algorithms. This method is not only suitable for the detection of tearing actions, but can also be extended to other behavioral analysis involving fine hand movements.

[0064] The present invention uses visual perception and gravity sensing to jointly determine improper behavior in the vending process. In addition, the present invention can also identify the vending situation by combining visual perception and gravity sensing. Among them, the existing vending orders are usually implemented by pure vision or pure gravity, and the pure vision or pure gravity sensing method has some disadvantages.

[0065] In the field of unmanned vending, problems such as grabbing with one hand (grabbing two items AB at the same time with one hand, or grabbing AA), putting the items back after taking them, taking multiple items together, and taking small items are very common. At present, many units are studying pure dynamic vision solutions, but pure visual algorithms are prone to problems such as unclear quantity, missed inspection of small items, and multiple identification when putting them back in the above scenarios. Based on this, the present invention can combine visual perception and gravity sensing to achieve accurate identification of commodity purchases.

[0066] Usually, there are two cameras on the top and upper right side of the vending machine to obtain shopping video images. After obtaining the video shot by the top camera, the cabinet door area is detected and a virtual cabinet door is drawn to facilitate the identification and judgment of the goods entering and exiting.

[0067] According to the captured video stream, the category of goods and the purchaser's purchasing behavior are detected from the video stream based on the target detection model, which mainly includes the actions of taking goods out and putting goods into the vending machine. The purchaser's shopping behavior is judged based on these actions.

[0068] When detecting the category of a commodity, the commodity image in the video stream image may be firstly cut out, and then the commodity image may be matched with commodity images of various categories preset in the database to obtain the category of the commodity grabbed or put back from the shelf.

[0069] Among them, according to the list of goods sold in the vending cabinet, the commodity category library corresponding to the entire cabinet is sorted out, and all the goods in this library are encoded, and the commodity codes are stored in the specified database. For the commodity image extracted from the video stream, search in the constructed commodity code library, and find which commodity in the library has the highest similarity with the extracted image. The label of the searched image is used as the label of the extracted commodity, and the matching similarity is used as the confidence that the extracted commodity is the corresponding label.

[0070] By using the above method, a corresponding product will be identified by processing a tracked sequence. Then use the weight change data obtained in this time period to match the corresponding purchased product. If the visually recognized product information and weight information are correctly matched and equal within the allowable error range, the recognition result of the time series can be accurately output. If the weight information is an integer multiple of the weight of a single product, it can be determined that the shopping behavior corresponding to the time series has purchased multiple pieces of the product. If the weight information and product information do not match (not an integer multiple), there are two situations. If the weight change perceived by the gravity sensor is smaller than the weight value of the matched single product, it is possible that the visual recognition is wrong. If the weight change perceived by the gravity sensor is larger than the weight value of the matched single product, it is possible that there are multiple products in this shopping, but the visual algorithm cannot see another product. At this time, manual intervention is required.

[0071] In some scenarios, a shopper takes out a drink A, but later doesn’t want it anymore and puts it back in the cabinet. When bending over to put it back, the process of bending over blocks the process of putting it back. If a pure visual solution is used, only the purchase of A will be recognized, but if gravity is combined, even if the image of putting A back cannot be captured and “seen”, it can be judged that the shopper did not buy A because there is no relative change in the weight data.

[0072] At this time, the purchase behavior can be identified based on the combination of visual perception and gravity sensing, see Figure 2 , in the case of no weight change, there are the following situations: When the product is not perceived through visual perception (the hand does not enter the cabinet or the hand enters the cabinet to take out the product and then puts it down), and there is no weight change, it is determined that there is no shopping. When it is detected that product A is taken out and then put back, and the weight does not change, it is determined that there is no shopping. When it is detected that product A is taken out and product B is put in, and the weight does not change, it is determined to be abnormal and manual intervention is entered.

[0073] For the case of weight changes, there are the following situations: If the visual perception of the commodity category is 1 and it matches the weight change value sensed by the gravity sensor, the settlement output is directly performed. If the visual perception of the commodity category is 1 and the weight change value is an integer multiple or close to an integer multiple of the weight of the commodity, the settlement output is directly performed. If the commodity category is 1 and the weight change is not an integer multiple of the weight of the commodity, manual intervention is required. For example, there may be obstructions or some commodities are not detected during the purchase process. If the visual perception of the commodity category is 2 and the single weight combination of the two commodities (A+B) is equal to the weight change value, the settlement output is directly performed and the two commodities A+B are purchased. When the visual perception of the commodity category is 2 and the weight combination of the two commodities is not equal to the weight change value, an abnormality occurs and manual intervention is required. In the case where the weight of other commodity combinations does not match the weight change value, manual intervention is required. And for the abnormal orders that are perceived as irregular picking, foreign objects, and commodities not on the shelves, all are manually marked and sent to the operator for processing.

[0074] See also Figure 3 , a shopping order recognition system based on vision and gravity sensing of the present invention is provided, the system comprising: An acquisition module 301 is used to acquire a video stream of a shopping area of ​​a vending machine; A detection module 302, configured to detect commodity information in the video stream based on an object detection model, and to obtain position information of key points of a hand within a set time period based on a hand detection model; The calculation module 303 is used to calculate the hand movement angular velocity according to the position information of the hand key points at different times; and calculate the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the hand key points of the two hands; The judging module 304 is used to judge whether there is a hand rotation action on the commodity according to the hand movement angular velocity, and judge whether there is a tearing action on the commodity packaging bag according to the change of the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands; The confirmation module 305 is used to confirm that abnormal shopping behavior has occurred and issue an abnormal alarm if there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf is reduced.

[0075] It can be understood that the shopping order recognition system based on vision and gravity sensing provided by the present invention corresponds to the shopping order recognition method based on vision and gravity sensing provided by the aforementioned embodiments. The relevant technical features of the shopping order recognition system based on vision and gravity sensing can refer to the relevant technical features of the shopping order recognition method based on vision and gravity sensing, which will not be repeated here.

[0076] The embodiment of the present invention provides a shopping order recognition method and system based on vision and gravity sensing, which detects the commodity information and the position information of the key points of the hand in the video stream of the shopping area of ​​the vending machine; according to the position information of the key points of the hand at different times, the angular velocity of the hand movement is calculated to determine whether there is a hand rotation action on the commodity, and the change in the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands are calculated to determine whether there is a tearing action on the commodity packaging bag; if there is a hand rotation action or a tearing action, and the weight on the shelf is reduced, it is confirmed that the shopping abnormal behavior has occurred. The present invention provides a shopping order recognition solution based on the combination of gravity and vision, which solves the problem that the pure visual algorithm cannot solve the problem of the number of commodities, and solves the problem that the pure weight algorithm cannot identify the category of commodities, further improves the accuracy of algorithm recognition, and reduces the recognition cost and labor cost. In particular, based on hand posture detection, hand rotation, relative movement of fingers, etc., the ability of the algorithm to identify abnormal behaviors such as stealing and drinking is improved, further improving the detection accuracy, improving efficiency, and reducing costs.

[0077] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0083] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A shopping order recognition method based on vision and gravity sensing, characterized in that: include: Get the video stream of the vending machine shopping area; Detecting commodity information in the video stream based on the target detection model, and acquiring position information of key points of the hand within a set time period based on the hand detection model; Calculating the hand movement angular velocity according to the position information of the key points of the hand at different times; and calculating the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the key points of the two hands; Judging whether there is a hand rotation action on the commodity according to the hand movement angular velocity, and judging whether there is a tearing action on the commodity packaging bag according to the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands; If there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf decreases, it means that the shopping behavior is abnormal and an abnormal alarm is issued.

2. The shopping order recognition method according to claim 1, characterized in that: The step of calculating the hand movement angular velocity according to the position information of the key points of the hand at different times includes: Get the position information of the wrist center point and the position information of the second joint of the index finger ; The hand orientation is calculated based on the position information of the wrist center point and the position information of the second joint point of the index finger: According to the hand orientation, the hand orientation angle is calculated by the arctan function: Where t represents the timestamp or frame index of the current frame; According to the hand orientation angles of two adjacent frames, the hand movement angular velocity is calculated: in, is the time interval between two adjacent frames.

3. The shopping order recognition method according to claim 1 or 2, characterized in that: The method further comprises calculating the hand movement angular velocity according to the position information of the hand key points at different times, and then: The hand movement angular velocity is smoothed based on a moving smoothing filter to obtain the smoothed hand movement angular velocity : Where n is the window size of the moving smoothing filter, is the angular velocity of the hand at the i-th moment.

4. The shopping order recognition method according to claim 1, characterized in that: The step of calculating the finger distance change, the relative movement speed of the two hands, and the relative movement direction of the two hands according to the position information of the hand key points of the two hands includes: Get the fingertip position vector of the left thumb respectively , the position vector of the left index finger tip , the position vector of the right thumb tip and the right thumb tip position vector ; Calculate the distance between the thumb and index finger of the left hand , the distance between the thumb and index finger of the right hand and the distance between the left thumb and the right thumb ; Calculate the relative movement speed of the two hands: in, is the relative speed between the left thumb and the right thumb, is the time interval between two frames.

5. The shopping order recognition method according to claim 4, characterized in that: The step of calculating the finger distance change, the relative movement speed of the two hands, and the relative movement direction of the two hands according to the position information of the hand key points of the two hands includes: According to the motion vector of the left index finger and the motion vector of the right index finger in the previous frame and the current frame, the angle between the motion vector of the left index finger and the motion vector of the right index finger is calculated. , the angle Represents the relative movement direction of the two hands.

6. The shopping order recognition method according to claim 5, characterized in that: The motion vector of the left index finger is expressed as , the motion vector of the right index finger is expressed as ,in, is the position vector of the left index finger in the current frame, is the position vector of the left index finger in the previous frame, is the position vector of the right index finger in the current frame, is the position vector of the right index finger in the previous frame; Calculate the angle between the motion vector of the left index finger and the motion vector of the right index finger : in, Representation vector The Euclidean norm of , Representation vector The Euclidean norm of .

7. The shopping order recognition method according to claim 1, characterized in that: The determining, based on the hand movement angular velocity, whether there is a hand rotation action on the commodity includes: When there are m consecutive frames of hand motion angular velocities that all satisfy When the product perceived by vision is a beverage, it is confirmed that the hand is rotating the product, wherein: is the angular velocity dynamic threshold.

8. The shopping order recognition method according to claim 1, characterized in that: Judging whether there is a tearing action on the product packaging bag according to the change in the finger distance, the relative movement speed of the two hands, and the relative movement direction of the two hands, includes: When the following judgment conditions are met, it is confirmed that the product packaging bag is torn: in, is the relative speed between the left thumb and the right thumb, is the angle between the motion vector of the left index finger and the motion vector of the right index finger, is the distance between the thumb and index finger of the left hand, is the distance between the thumb and index finger of the right hand, is the distance between the left thumb and the right thumb, is the speed dynamic threshold, a, b, c are the distance thresholds.

9. The shopping order recognition method according to claim 1, characterized in that: If there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf decreases, an abnormal shopping behavior occurs, and an abnormal alarm is issued, including: The gravity sensor is used to sense the weight changes of the goods on the vending machine in real time. When there is a hand rotation action on the goods or a tearing action on the product packaging bag, and the weight data of the goods on the vending machine becomes smaller, it is determined that the goods are stolen and an abnormal alarm is issued.

10. A shopping order recognition system based on vision and gravity sensing, characterized in that: include: An acquisition module, used to acquire a video stream of a shopping area of ​​a vending machine; A detection module, used to detect commodity information in the video stream based on a target detection model, and to obtain position information of key points of a hand within a set time period based on a hand detection model; A calculation module, used to calculate the angular velocity of the hand movement according to the position information of the key points of the hand at different times; and calculating the finger distance change, the relative movement speed of the two hands and the relative movement direction of the two hands according to the position information of the key points of the two hands; A judgment module, used to judge whether there is a hand rotation action on the commodity according to the hand movement angular velocity, and to judge whether there is a tearing action on the commodity packaging bag according to the change of the finger distance, the relative movement speed of the two hands and the relative movement direction of the two hands; The confirmation module is used to confirm that abnormal shopping behavior has occurred and issue an abnormal alarm if there is a hand rotation action on the product or a tearing action on the product packaging bag, and the weight data on the shelf is reduced.

Images