A Dynamic Commodity Recognition Method and Related Device for a Vending Cabinet Based on Target Tracking

Abstract

The present invention provides a dynamic commodity recognition method for a vending cabinet based on target tracking, belonging to the field of dynamic recognition. The present invention improves the multi-target tracking algorithm and uses uniformly-variable Kalman filtering to predict trajectories, which can better adapt to the acceleration, deceleration, and sudden pause of user behavior. The matching strategy proposed by the present invention simultaneously focuses on the motion characteristics and appearance characteristics of commodities, can better recover the trajectories that are temporarily lost due to occlusion, and ensure the coherence of the commodity motion trajectories. The judgment scheme proposed in step 4 of the present invention can make a judgment on the user's shopping behavior based on the motion direction of the commodity and has a certain ability to identify abnormal situations. On the other hand, the present invention can be photographed based on a single camera, requiring fewer deployed cameras and lower deployment costs. Since the present invention can adopt the method of an external camera, the space utilization rate inside the cabinet can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of dynamic recognition, and in particular to a dynamic commodity recognition method for a vending cabinet based on target tracking and related devices. Background Art

[0002] Unmanned retail is an emerging field in recent years. Unmanned vending focuses on solving the vending problem more automatically. Compared with traditional vending methods, unmanned vending has higher time and space flexibility. Vending cabinets can be deployed in office buildings, schools, roadside, etc., and can also achieve 24-hour vending, greatly improving shopping convenience and economic efficiency.

[0003] In traditional vending cabinets, users manually select the goods they need to purchase in advance, and then complete the transaction process by coin insertion or code scanning payment. Nowadays, with the emerging payment methods, the coin insertion process has been basically replaced. However, the most core pain point of this type of solution is that the goods to be purchased must be selected in advance, and then the vending cabinet returns the goods. The entire shopping process is closed, and the vending cabinet has a complex mechanical structure inside. The emerging vending cabinets propose an open vending service of "scanning the code to directly open the door, pick up the goods, and automatically settle the bill when closing the door" for consumers, which is a more convenient consumption process.

[0004] Currently, common solutions for vending cabinets include RFID radio frequency tags, static vision recognition, and dynamic vision recognition, etc. RFID is a radio frequency identification technology, which achieves the purpose of identifying target goods through short-range communication between RFID tags and readers. Specifically, the specific information of the goods is first written into an RFID tag, and then the tag is pasted on the commodity packaging. When making a determination, the vending cabinet can judge which goods the user has purchased by detecting which tag has disappeared. The recognition method of RFID tags has a high accuracy rate, but its disadvantages are very obvious. That is, the cost of the tags is relatively high, and if the paste is not firm or the tag falls off due to human damage, it will lead to inaccurate recognition. Currently, this solution has basically stopped being used. The static vision recognition solution is to identify the goods in the cabinet before and after opening and closing the door, and then compare the differences to obtain the final shopping result. Specifically: take a photo of the items in the cabinet before opening the door, and then take another photo after closing the door. The goods missing in the photo after closing the door are the goods purchased by the user. In terms of the structure of the smart cabinet, the static recognition solution requires a camera to be installed on each layer inside the cabinet to be able to completely obtain the status of all goods inside the cabinet. On the other hand, due to the limited viewing angle of the camera, in order to be able to completely photograph all goods, it is necessary to require sufficient height between each layer of the vending cabinet, and the goods cannot be stacked on top of each other to avoid affecting the quantity detection.

[0005] The internal mechanical structure of the traditional solution for containers is complex, the cost of the RFID tag solution is high, and the tags are prone to falling off; the static vision recognition solution requires a large number of cameras to be installed, resulting in low space utilization inside the container. The dynamic vision recognition solution can better solve the shortcomings of the above solutions. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a dynamic commodity recognition method and related device for unmanned vending cabinets based on target tracking.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions:

[0008] A dynamic commodity recognition method for unmanned vending cabinets based on target tracking, including the following steps:

[0009] Step 1, obtain the consumption behavior video of the user in the unmanned vending cabinet, extract frames and label the video data set;

[0010] Step 2, use the video data set to train the YOLOX target detection network model;

[0011] Step 3, use the YOLOX target detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection frames that appear in each frame of the picture;

[0012] Use the improved multi-target tracking algorithm ByteTrack to process the video to be recognized frame by frame, and obtain the motion trajectories of each commodity in each frame;

[0013] Match and associate each frame of commodity detection frame with the motion trajectory of the commodity to obtain the complete motion trajectory of the commodity, and perform secondary matching and association on the detection frames that are not successfully matched and the trajectories in the lost state;

[0014] The improved multi-target tracking algorithm ByteTrack uses a Kalman filter containing an acceleration state component;

[0015] Step 4, based on the complete trajectory information of the commodity, judge whether it is sold based on the motion direction of the commodity itself, and finally obtain the user's shopping result.

[0016] Further, in step (3), the improved multi-target tracking algorithm ByteTrack is used to process the video to be recognized frame by frame to obtain the motion trajectory information of each commodity in the video. The specific operation is as follows:

[0017] Use a uniformly accelerated Kalman filter to predict the state of each commodity trajectory to obtain the predicted position of the trajectory; the state vector of the uniformly accelerated Kalman filter in the improved ByteTrack algorithm is:

[0018] (x, y, r, h, x v , y v , r v , h v , x a , y a , r a , h a ) T

[0019] Among them, x and y represent the coordinates of the target, r represents the width-to-height ratio of the target, h represents the height of the target, x v , y v , r v , h v then correspond to the velocity components of the four states of x, y, r, and h respectively, x a , y a , r a , h a are the acceleration components;

[0020] The state transition matrix of the uniformly-variable Kalman filter is:

[0021]

[0022] In the formula, Δt is the sampling period, which is the frame interval in a frame-by-frame processed video;

[0023] The state transition equation and the observation equation of the uniformly-variable Kalman filter are:

[0024]

[0025] In the formula, S is the first four items (x, y, r, h) of the state vector; V is the (x v , y v , r v , h v ) of the state vector, representing the velocity components of the four states of x, y, r, and h; a is the (x a , y a , r a , h a ) of the state vector, representing the acceleration components of the four states of x, y, r, and h; ω t represents the process noise at time t, which is used to describe the error between the state transition matrix and the actual motion process, corresponding to the Gaussian noise with an expected value of 0 and a covariance of Q in x t+1 ; θ t+1 is the observation noise at time t + 1, which follows a Gaussian distribution with an expected value of 0 and a covariance of R; R, as the covariance of the observation noise, and Q and R are both noise covariance matrices.

[0026] Further, in step 3, the commodity detection frames in each frame of image are matched and associated with the movement trajectories of the commodities, and the implementation steps are as follows:

[0027] 301. Divide the commodity detection frames into high-confidence detection frames and low-confidence detection frames according to the confidence level;

[0028] 302. For the high-confidence detection frames and the movement trajectories of each commodity, construct a cost matrix according to the IoU distance, use the Hungarian algorithm for matching, associate the matched detection frames with the trajectories, and at the same time obtain the detection frames and the movement trajectories of the commodities that are not successfully matched;

[0029] 303. For the movement trajectories of the commodities that are not successfully matched and the low-confidence detection frames, construct a cost matrix according to the IoU distance, use the Hungarian algorithm for matching, and again obtain the detection frames and the movement trajectories of the commodities that are not successfully matched, corresponding to the newly emerged detection frames and the trajectories that are not in the lost state.

[0030] Further, perform secondary matching and association on the detection frames that are not successfully matched and the trajectories in the lost state. The specific implementation steps are as follows:

[0031] 304. For the trajectories in the lost state, calculate the maximum matching distance on each frame. At the same time, use a lightweight feature extraction network model to extract the appearance features of the target and obtain the target appearance feature vector;

[0032] 305. Match the newly emerged detection frames with the trajectories in the lost state. First, calculate the distance between the new detection frame and the last appearance position of the lost trajectory;

[0033] Then use the feature extraction network model to extract the appearance feature vector of the new detection frame, and calculate the Euclidean distance between the appearance feature vector of the new detection frame and the target appearance feature vector;

[0034] When the spatial distance D m is less than the maximum matching distance D A , and the Euclidean distance D a is less than the preset threshold, then match and associate the target p with the trajectory A, and wake up the lost trajectory A.

[0035] Further:

[0036] In step 304, calculate a maximum matching distance on each frame. The calculation formula is:

[0037] D A =(T current -T last )*speed

[0038] In the formula, D A represents the farthest matching distance of the trajectory A, and Tlast Indicates the frame number when the target appeared last time, T current Indicates the frame number currently being detected. speed is a hyperparameter that measures the average moving speed of the target; or / and

[0039] In step 305, calculate the distance D between the remaining lost trajectories and the newly emerged detection boxes m , and the formula is:

[0040]

[0041] In the formula, D m Indicates the distance between the newly detected box p and the trajectory A, x p , y p Indicates the coordinates of the center point of the detection box p, x last , y last Indicates the coordinates of the center point when the trajectory A appeared last time; or / and

[0042] In step 305, calculate the Euclidean distance D a , and the formula is:

[0043]

[0044] In the formula, Indicates the appearance feature vector of the new target p, is the appearance feature vector of the stored lost target A, n is the dimension of the feature vector, and D a is the Euclidean distance between the appearance feature vectors of the target p and A.

[0045] Furthermore, it also includes the following operations:

[0046] Delete the trajectories that have been in the lost state for a time reaching the preset threshold;

[0047] Create new trajectories for the high-score detection boxes that have not been successfully matched twice.

[0048] Furthermore, the implementation steps of step 4 are:

[0049] S401. Read each frame of the video and the commodity trajectory information obtained in step 3;

[0050] S402. Determine the boundary line between the inside and outside of each frame of the video, and divide the video frame into 5 regions: the inside cabinet region, the outside cabinet region, the pre-entry cabinet region, the pre-exit cabinet region. The pre-entry cabinet region is a partial region outside the cabinet close to the boundary line, and the pre-exit cabinet region is a partial region inside the cabinet close to the boundary line;

[0051] S403. Create three collections: a pre-entry collection, a pre-leave collection, and a temporary shopping cart collection. The pre-entry collection stores the merchandise currently in the pre-entry area, the pre-leave collection stores the merchandise currently in the pre-leave area, and the temporary shopping cart stores the merchandise currently being taken out of the sale container.

[0052] S404: If a product crosses the dividing line from the pre-checkout area to the pre-checkout area in the image, it is defined as a checkout line collision. In this case, the user is deemed to have taken the product and the product is added to the temporary shopping cart.

[0053] If an item crosses the dividing line from the pre-check-in area to the pre-check-out area on the screen, it is considered a check-in collision. At this time, it is considered that the user has returned the item and the item will be deducted from the temporary shopping cart.

[0054] If a product hits the checkout line in the image, but the product was not previously in the temporary shopping cart, it is considered that the user has placed an unknown product in the checkout, and the shopping order will be marked as abnormal;

[0055] S405, looping and processing each frame of the video until the video ends;

[0056] After the processing is completed, the types and quantities of goods stored in the temporary shopping cart are the final results of the user's shopping.

[0057] A dynamic commodity recognition device for unmanned vending machines based on target tracking, comprising a video data set acquisition module, a target detection network model training module, a commodity detection frame and motion trajectory matching module, and a shopping result recognition module;

[0058] The video dataset acquisition module is used to acquire the consumption behavior video of the user at the unmanned vending machine, extract the frames and annotate the video dataset;

[0059] The target detection network model training module is used to train the YOLOX target detection network model using the video dataset;

[0060] The commodity detection frame and motion trajectory matching module is used to use the YOLOX target detection network model to perform frame-by-frame recognition on the vending machine video to be identified, and obtain the commodity detection frame appearing in each frame;

[0061] The improved multi-target tracking algorithm ByteTrack is used to process the video to be identified frame by frame to obtain the motion trajectory of the product in the video; it is also used

[0062] Match and associate the product detection frame with the product's motion trajectory, and rematch and associate the unmatched detection frame with the lost trajectory.

[0063] The shopping result recognition module is used to judge whether a product is sold based on the complete trajectory information of the product and the movement direction of the product itself, and finally obtain the user's shopping result.

[0064] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned dynamic product recognition method for an unmanned vending cabinet based on target tracking are implemented.

[0065] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned dynamic product recognition method for an unmanned vending cabinet based on target tracking are implemented.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention provides a dynamic product recognition method for an unmanned vending cabinet based on target tracking. The selected data set comes from real user shopping scenarios, so the trained model can better adapt to the actual shopping scenarios. The present invention improves the multi-target tracking algorithm and uses uniformly-variable Kalman filtering to predict trajectories, which can better adapt to the acceleration, deceleration, and sudden stops of user behaviors. The matching strategy proposed by the present invention simultaneously focuses on the motion characteristics and appearance characteristics of products, can better restore the trajectories temporarily lost due to occlusion, and ensure the coherence of the product motion trajectories. The judgment scheme proposed in step 4 of the present invention can judge the user's shopping behavior based on the movement direction of the product and has a certain ability to identify abnormal situations. On the other hand, the present invention can be photographed based on a single camera, only photographing the user's shopping behavior, requiring fewer deployed cameras and lower deployment costs. Since the present invention can adopt the method of an external camera, there is no requirement for the height of each layer in the vending cabinet, and the products in the cabinet can be randomly overlapped, which can greatly improve the space utilization rate in the cabinet.

[0068] A dynamic product recognition method for an unmanned vending cabinet based on target tracking provided by the present invention includes specific modules for completing the above work method.

[0069] The present invention provides a computer device and a storage medium for a dynamic product recognition method for an unmanned vending cabinet based on target tracking, used to implement the specific steps of the above work method. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 For the comparison of vending cabinets based on two schemes of static vision recognition and dynamic recognition;

[0071] Figure 2The process of using a multi-object tracking algorithm for commodity tracking;

[0072] Figure 3 Schematic diagram of the key area of the video frame;

[0073] Figure 4 Schematic diagram for determining when a commodity leaves the cabinet;

[0074] Figure 5 Schematic diagram for determining when a commodity enters the cabinet;

[0075] Figure 6 Schematic diagram for determining abnormal situations;

[0076] Figure 7 Key frame display of the shopping video of the embodiment. Detailed implementation manners

[0077] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0078] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0079] The present invention will be further described in detail below with reference to the accompanying drawings:

[0080] The present invention designs a dynamic recognition method for commodities in a self-service vending cabinet based on a multi-object tracking algorithm, improves the Kalman filtering module in the ByteTrack algorithm, and designs a matching strategy for lost trajectories and new detection frames to ensure a more coherent tracking effect. The structure of the self-service vending cabinet used in the present invention is as Figure 1As shown on the right, the camera is installed above the container, and the camera's field of view only focuses on the user's behavior without capturing the status of the goods inside the container. In contrast, the container can hold more goods and requires fewer cameras to be deployed. Taking a six-story container as an example, the static solution requires 6 cameras to be installed, while the dynamic solution only requires 1 to 2 cameras.

[0081] As Figure 2 shown, the specific implementation of the present invention has the following steps:

[0082] Step 1: The user opens the cabinet door, the camera is turned on, and starts recording video clips of the user's shopping.

[0083] Step 2: The user finishes shopping, closes the cabinet door, the camera is turned off, the shopping video clips are stored and transmitted to the server.

[0084] Step 3: The server processes the video clips and uses object detection to detect the goods that appear in each frame of the video.

[0085] Step 4: According to the video frame and the goods detection box, process the video frame by frame and generate goods trajectory information.

[0086] Step 5: According to the goods movement trajectory information, analyze the types and quantities of the goods purchased by the user in the video clip.

[0087] As Figure 3 shown, when specifically determining the sales result of the goods, it is necessary to determine according to the movement trajectory of the goods. The method of the present invention is to divide the video frame into two regions with the sales cabinet door as the boundary in the video clip of the user's shopping: the in-cabinet region and the out-of-cabinet region. A part of the in-cabinet area close to the dividing line is set as the pre-leaving cabinet area, and a part of the out-of-cabinet area close to the dividing line is set as the pre-entering cabinet area. When analyzing, three sets are maintained: the pre-leaving cabinet goods set, the pre-entering cabinet goods set, and the temporary shopping cart set. The pre-leaving cabinet goods set stores the goods trajectories that are currently in the pre-leaving cabinet area; the pre-entering cabinet goods set stores the goods trajectories that are currently in the pre-entering cabinet area; the temporary shopping cart stores the goods taken out of the sales cabinet. For the trajectory of a piece of goods in the entire video, there will be three situations:

[0088] 1. Leaving the cabinet: It means that the goods cross the dividing line from the pre-leaving cabinet area into the pre-entering cabinet area in the picture, which is defined as its occurrence of leaving the cabinet and crossing the line. At this time, it is considered that the user has taken a piece of goods and added it to the temporary shopping cart.

[0089] 2. Entering the cabinet: It means that the goods cross the dividing line from the pre-entering cabinet area into the pre-leaving cabinet area in the picture, which is defined as its occurrence of entering the cabinet and crossing the line. At this time, it is considered that the user has put back a piece of goods. If this piece of goods exists in the temporary shopping cart, it will be deducted from the temporary shopping cart.

[0090] 3. Abnormal: This occurs when an item hits the counter line as described in 2, but the item was not previously in the temporary shopping cart. In this case, it is considered that the user has placed an unknown item in the counter, and the purchase order needs to be marked as abnormal.

[0091] In order to better explain the solution, Figures 4 - 6 Let's describe the above three situations in detail:

[0092] See also Figure 4 , Figure 4 This is a diagram of a product leaving the counter. Item A1 is in the sales counter at frame T. At frame T+1, it is removed and placed in the pre-exit area, at which point its item number, A1, is added to the pre-exit collection. At frame T+2, A1 crashes into the pre-exit area and enters the pre-entry area. At this point, A1 is added to the temporary shopping cart and moved from the pre-exit collection to the pre-entry collection. At frame T+3, A1 leaves the pre-entry area and exits the sales counter, at which point it is removed from the pre-entry collection. After frame T+3, item A1 is added to the user's temporary shopping cart.

[0093] See also Figure 5 , Figure 5 Figure 1 shows a schematic diagram of a product entering a counter. Item A (numbered 1) is outside the sales counter at frame T, with item A1 in the temporary shopping cart. At frame T+1, it is removed and placed in the pre-entry area, at which point its number, A1, is added to the pre-entry set. At frame T+2, A1 crashes into the pre-entry area while leaving the pre-entry area, entering the pre-leaving area. At this point, the quantity of item A is deducted from the temporary shopping cart by one, and A1's trajectory is moved from the pre-entry area to the pre-leaving set. At frame T+3, A1 leaves the pre-leaving area and is placed inside the counter, deleting it from the pre-leaving set. After frame T+3, item A is returned to the sales counter, and the temporary shopping cart is empty.

[0094] See also Figure 6 , Figure 6 This is a diagram of an abnormal situation. In frame T, item A is outside the counter, and there is one item A in the temporary shopping cart. In frame T+1, item B (numbered 2) enters the pre-entry area from outside the counter, at which point trajectory B2 is added to the pre-entry set. In frame T+2, B2 enters the pre-entry area after hitting the entry line. At this point, B2 is moved from the pre-entry set to the pre-leaving set. Since there is no item B in the temporary shopping cart, an abnormal behavior is determined and the order is marked as "abnormal." In frame T+3, item B leaves the yellow area and is placed in the counter, removing it from the pre-leaving set. After frame T+3, there is still one item A in the temporary shopping cart, and the order is marked as abnormal.

[0095] The present invention also provides a dynamic commodity recognition device for a vending cabinet based on target tracking, including a video data set acquisition module, a target detection network model training module, a commodity detection box and motion trajectory matching module, and a shopping result recognition module;

[0096] The video data set acquisition module is used to acquire the consumption behavior video of the user in the vending cabinet, extract frames and label the video data set;

[0097] The target detection network model training module is used to use the video data set to train the YOLOX target detection network model;

[0098] The commodity detection box and motion trajectory matching module is used to use the YOLOX target detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection boxes appearing in each frame of the picture; it is also used for

[0099] Using the improved multi-target tracking algorithm ByteTrack to perform frame-by-frame processing on the video to be recognized, and obtain the motion trajectories of the commodities in the video; it is also used for

[0100] Match and associate the commodity detection boxes with the motion trajectories of the commodities, and perform secondary matching and association on the detection boxes that have not been successfully matched and the trajectories in the lost state;

[0101] The shopping result recognition module is used to judge whether the commodity is sold based on the complete trajectory information of the commodity and the motion direction of the commodity itself, and finally obtain the user's shopping result.

[0102] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 4 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, it realizes the method for dynamic commodity recognition of a vending cabinet based on target tracking.

[0103] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are realized: Step 1, acquire the consumption behavior video of the user in the vending cabinet, extract frames and label the video data set;

[0104] Step 2: Use the video dataset to train the YOLOX object detection network model;

[0105] Step 3: Use the YOLOX object detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection frames in each frame;

[0106] Use the improved multi-object tracking algorithm ByteTrack to process the video to be recognized frame by frame, and obtain the motion trajectories of the commodities in the video;

[0107] Match and associate the commodity detection frames with the motion trajectories of the commodities, and perform secondary matching and association on the detection frames that have not been successfully matched with the trajectories in the lost state;

[0108] The improved multi-object tracking algorithm ByteTrack uses a Kalman filter that includes an acceleration state component;

[0109] Step 4: Based on the complete trajectory information of the commodity, judge whether it has been sold based on the motion direction of the commodity itself, and finally obtain the user's shopping result.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Step 1: Obtain the consumption behavior video of the user in the vending cabinet, extract frames and label the video dataset;

[0111] Step 2: Use the video dataset to train the YOLOX object detection network model;

[0112] Step 3: Use the YOLOX object detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection frames in each frame;

[0113] Use the improved multi-object tracking algorithm ByteTrack to process the video to be recognized frame by frame, and obtain the motion trajectories of the commodities in the video;

[0114] Match and associate the commodity detection frames with the motion trajectories of the commodities, and perform secondary matching and association on the detection frames that have not been successfully matched with the trajectories in the lost state;

[0115] The improved multi-object tracking algorithm ByteTrack uses a Kalman filter that includes an acceleration state component;

[0116] Step 4: Based on the complete trajectory information of the commodity, judge whether it has been sold based on the motion direction of the commodity itself, and finally obtain the user's shopping result.

[0117] Embodiment

[0118] See also Figure 7 , Figure 7 The key frames of the shopping video of the embodiment are shown. The key frames of the shopping video describe the process of a user taking out an item and putting it back, then taking out two items in succession, and finally closing the cabinet door. In the 44th frame, the cabinet door is opened and the user starts shopping. In the 79th frame, the user takes a "Yuanqi Forest Alien" (track ID is 4) and enters the pre-leaving cabinet area. In the 85th frame, the item hits the line when leaving the cabinet, and a "Yuanqi Forest Alien" is added to the temporary shopping cart. In the 88th frame, the user puts the item back, and the item enters the pre-leaving cabinet area and hits the line when entering the cabinet. A "Yuanqi Forest Alien" is deducted from the temporary shopping cart. In the 188th frame, the user takes another "Yuanqi Forest Alien" (track ID is 17) and enters the pre-leaving cabinet area. In the 189th frame, the item hits the line when leaving the cabinet, and a "Yuanqi Forest Alien" is added to the temporary shopping cart. In frame 221, the user takes another "Yuanqi Forest Alien" (track ID 19). In frame 223, the product leaves the counter and hits the line, and one "Yuanqi Forest Alien" is added to the temporary shopping cart. In frame 247, the user closes the counter door, and the temporary shopping cart records that the user has taken two "Yuanqi Forest Alien" items.

[0119] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A dynamic commodity recognition method for a vending cabinet based on target tracking, characterized in that, It includes the following steps: Step 1: Obtain the consumption behavior video of the vending cabinet by the user, extract frames and label the video dataset; Step 2: Use the said video dataset to train the YOLOX object detection network model; Step 3: Use the YOLOX object detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection frames appearing in each frame of the picture; Use the improved multi-object tracking algorithm ByteTrack to perform frame-by-frame processing on the video to be recognized, and obtain the motion trajectories of commodities in each frame; Match and associate each frame of commodity detection frame with the motion trajectory of the commodity to obtain the complete motion trajectory of the commodity, and perform secondary matching and association on the detection frames that are not successfully matched and the trajectories in the lost state; The improved multi-object tracking algorithm ByteTrack uses a Kalman filter containing an acceleration state component; Step 4: According to the complete trajectory information of the commodity, judge whether it is sold based on the motion direction of the commodity itself, and finally obtain the user's shopping result.

2. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 1, wherein In step (3), the improved multi-object tracking algorithm ByteTrack is used to perform frame-by-frame processing on the video to be recognized to obtain the motion trajectory information of each commodity in the video. The specific operation is as follows: Use a uniformly-varying Kalman filter to predict the state of each commodity trajectory to obtain the predicted position of the trajectory; the state vector of the uniformly-varying Kalman filter in the improved ByteTrack algorithm is: (x,y,r,h,x v ,y v ,r v ,h v ,x a ,y a ,r a ,h a ) T Among them, x and y represent the coordinates of the target, r represents the width-to-height ratio of the target, h represents the height of the target, x v , y v , r v , h v Then they respectively correspond to the velocity components in the four states of x, y, r, and h, x a , y a , r a , h a are the acceleration components; The state transition matrix of the uniformly-varying Kalman filter is: In the formula, Δt is the sampling period, which is the frame interval in the frame-by-frame processed video; The state transition equation and observation equation of the uniformly-varying Kalman filter are: Wherein, S is the first four items (x, y, r, h) of the state vector; V is the (x v , y v , r v , h v ), representing the velocity components of the four states of x, y, r, and h; a is the (x a , y a , r a , h a ), representing the acceleration components of the four states of x, y, r, and h; ω t represents the process noise at time t, used to describe the error existing between the state transition matrix and the actual motion process, corresponding to the Gaussian noise with an expected value of 0 and a covariance of Q in x t+1 ; θ t+1 is the observation noise at time t + 1, following a Gaussian distribution with an expected value of 0 and a covariance of R; R, as the covariance of the observation noise, and Q and R are both noise covariance matrices.

3. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 1, characterized in that In step 3, match and associate the commodity detection frame in each frame of the picture with the motion trajectory of the commodity. The implementation steps are as follows:

301. Divide the said commodity detection frames into high-confidence detection frames and low-confidence detection frames according to the confidence level; 302. For the high-confidence detection frames and the motion trajectories of each commodity, construct a cost matrix according to the IoU distance, use the Hungarian algorithm for matching, associate the matched detection frames with the trajectories, and at the same time obtain the detection frames that are not successfully matched and the motion trajectories of the commodities; 303. For the motion trajectories of the commodities that are not successfully matched and the low-confidence detection frames, construct a cost matrix according to the IoU distance, use the Hungarian algorithm for matching, and obtain the detection frames that are not successfully matched and the motion trajectories of the commodities again, corresponding to the newly appeared detection frames and the trajectories that are not in the lost state.

4. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 3, wherein, The secondary matching and association of the detection frames that are not successfully matched and the trajectories in the lost state are as follows: the specific implementation steps are:

304. For the trajectories in the lost state, calculate the maximum matching distance for each frame. At the same time, use a lightweight feature extraction network model to extract the appearance features of the target and obtain the target appearance feature vector; 305. Match the newly appeared detection frames with the trajectories in the lost state. First, calculate the distance between the newly detected frame and the last appearance position of the lost trajectory; Then use the feature extraction network model to extract the appearance feature vector of the newly detected frame, and calculate the Euclidean distance between the appearance feature vector of the newly detected frame and the said target appearance feature vector; When the spatial distance D m is less than the maximum matching distance D A , and the Euclidean distance D a is less than the preset threshold, the target p is matched and associated with the trajectory A to wake up the lost trajectory A.

5. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 4, characterized in that: In step 304, a maximum matching distance is calculated for each frame, and the calculation formula is: D A = (T current - T last ) * speed where D A represents the farthest matching distance of trajectory A, T last represents the number of frames when the target appeared last time, T current represents the number of frames of the current detection, speed is a hyperparameter that measures the average moving speed of the target; or / and In step 305, calculate the distance D between the remaining lost trajectories and the newly emerged detection boxes m , and the formula is: Where D m represents the distance between the new detection box p and the trajectory A, and x p , y p represent the center point coordinates of the detection box p, and x last , y last represent the center point coordinates of the last appearance of the trajectory A; or / and In step 305, calculate the Euclidean distance D a , and the formula is: In the formula, represents the appearance feature vector of the new target p, is the appearance feature vector of the stored missing target A, n is the dimension of the feature vector, and D a is the Euclidean distance between the appearance feature vectors of target p and A.

6. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 4, wherein It further includes the following operations: Trajectories that have been in the lost state for a time reaching a preset threshold are deleted; For high-score detection frames that still fail to be successfully matched twice, new trajectories are created for them.

7. The dynamic commodity recognition method for a vending cabinet based on target tracking according to claim 1, characterized in that The implementation steps of step 4 are: S401. Read each frame of the video and the commodity trajectory information obtained in step 3; S402. Determine the boundary line between the inside and outside of each frame of the video, and divide the video frame into 5 regions: the in-cabinet region, the out-of-cabinet region, the pre-entry cabinet region, the pre-exit cabinet region. The pre-entry cabinet region is a partial region outside the cabinet close to the boundary line, and the pre-exit cabinet region is a partial region inside the cabinet close to the boundary line; S403. Create three sets: the pre-entry cabinet set, the pre-exit cabinet set, and the temporary shopping cart set. The pre-entry cabinet set stores the commodities currently in the pre-entry cabinet region, the pre-exit cabinet set stores the commodities currently in the pre-exit cabinet region, and the temporary shopping cart stores the commodities currently taken out of the vending cabinet; S404. If a commodity crosses the boundary line from the pre-exit cabinet region into the pre-entry cabinet region in the frame, it is defined as an out-of-cabinet line crossing. At this time, it is considered that the user has taken the commodity, and the commodity is added to the temporary shopping cart; If a commodity crosses the boundary line from the pre-entry cabinet region into the pre-exit cabinet region in the frame, it is defined as an in-cabinet line crossing. At this time, it is considered that the user has put back the commodity, and the commodity is deducted from the temporary shopping cart; If a commodity has the above-mentioned in-cabinet line crossing in the frame, but the commodity did not exist in the temporary shopping cart before, at this time, it is considered that the user has put an unknown commodity into the cabinet, and the shopping order is marked as abnormal; S405. Process each frame of the video in a loop until the video ends; After the processing is completed, the types and quantities of the commodities stored in the temporary shopping cart are the final results of the user's shopping.

8. A dynamic commodity recognition device for a vending cabinet based on target tracking, characterized in that, It includes a video data set acquisition module, a target detection network model training module, a commodity detection box and motion trajectory matching module, and a shopping result recognition module; The video data set acquisition module is used to acquire the consumption behavior video of the user in the vending cabinet, extract frames and annotate the video data set; The target detection network model training module is used to use the video data set to train the YOLOX target detection network model; The commodity detection box and motion trajectory matching module is used to use the YOLOX target detection network model to perform frame-by-frame recognition on the vending cabinet video to be recognized, and obtain the commodity detection boxes appearing in each frame of the picture; it is also used for Using the improved multi-target tracking algorithm ByteTrack to process the video to be recognized frame by frame, and obtaining the motion trajectories of the commodities in the video; it is also used for Matching and associating the commodity detection boxes with the motion trajectories of the commodities, and performing secondary matching and association on the detection boxes that fail to be successfully matched and the trajectories in the lost state; The shopping result recognition module is used to judge whether a commodity is sold based on the complete trajectory information of the commodity and the motion direction of the commodity itself, and finally obtain the user's shopping result.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the dynamic commodity recognition method for a vending cabinet based on target tracking according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the dynamic commodity recognition method for a vending cabinet based on target tracking according to any one of claims 1-7 are implemented.

Images