Deep Learning-Based Shelf Dynamic Commodity Recognition and Customer Shopping Matching Method

Abstract

A method for identifying dynamic commodities on shelves and matching customer shopping based on deep learning, disclosed by the present invention, comprises the following steps: collecting video streams of customer purchase behaviors and gravity sensing data of corresponding areas of the shelves, and transmitting them to the background server in real time; parsing and processing the video stream data through a trained commodity detection network and a customer trajectory network in the background server; judging the states of commodity taking-out and putting-in in the detection time series; matching the customer ID with the taken commodity according to the distance between the center point coordinates of the commodity detection frame and the customer trajectory frame in the key frames; and assisting in judging the customer's taking-out and putting-in of commodities according to the real-time gravity sensing data of the corresponding areas of the shelves, and generating a purchase list. The present invention can identify the purchase behaviors of multiple customers simultaneously, has a more flexible usage scenario, and uses dual judgments of vision and gravity sensing, with strong stability and anti-interference ability in the identification process.

Description

Technical Field

[0001] The present invention belongs to the technical field of new retail and image recognition, and relates to a method for dynamically recognizing goods on a shelf and matching customer shopping based on deep learning. Background Art

[0002] At present, there are two solutions for identifying the sale of goods on unmanned vending cabinets or shelves, which are divided into non-visual solutions and visual solutions.

[0003] The non-visual solution mainly uses RFID radio frequency identification technology or gravity sensing technology. Among them, the RFID radio frequency identification technology requires manual active writing of RFID tags, which requires a large amount of time cost and tag cost. Secondly, the tags are easily shielded and torn, and the recognition effect on metal and liquid goods will be affected, and the use stability is poor; if the gravity sensing technology is adopted, single weight detection can only identify goods with different weights, and goods with similar weights cannot be identified, with poor flexibility and very limited use scenarios.

[0004] The visual solution is divided into a dynamic solution and a static solution. Compared with the non-visual solution, the visual solution has a higher technical threshold, and at the same time, due to the marginal cost effect, the actual cost is also lower. The static solution detects the purchased goods by comparing the static images of the goods on the shelf before and after purchase. The cameras used to identify the goods need to be installed inside the shelf, so there are certain requirements for the layer distance of the shelf and the placement of the goods, which will result in a lower space utilization rate of the shelf. If the goods fall down or are stacked together, the static recognition method will fail. In addition, this method is not applicable to the situation of multiple people shopping. Since the above methods all have certain limitations, a method for dynamically recognizing goods on an intelligent shelf and matching customer shopping based on deep learning is proposed to overcome the deficiencies of the above methods. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for dynamically recognizing goods on a shelf and matching customer shopping based on deep learning, which solves the problems of high tag cost, easy damage and failure, and poor stability in the existing RFID radio frequency identification technology.

[0006] The technical solution adopted by the present invention is a method for dynamically recognizing goods on a shelf and matching customer shopping based on deep learning, which specifically includes the following steps:

[0007] Step 1, when a customer has a purchase behavior in the detection area, camera a collects the video stream of the customer's purchase behavior in front of the shelf, and the gravity sensor collects the real-time gravity sensing data of the corresponding area of the shelf. The video stream and the gravity sensing data are simultaneously transmitted back to the background server;

[0008] Step 2: For the video stream collected in Step 1, input each frame of the image into the commodity detection network and the customer trajectory detection network to obtain the commodity detection frame and the customer trajectory frame;

[0009] Step 3: In the detection time series, judge the state of the commodity being taken out and put in by the movement of the commodity detection frame in the image;

[0010] Step 4: Match the customer ID with the taken commodity according to the distance between the center point coordinates of the commodity detection frame and the customer trajectory frame in the key frame;

[0011] Step 5: According to the real-time gravity sensing data of the corresponding area of the shelf, assist in judging the customer's taking out and putting in of the commodity, and generate a purchase list.

[0012] The features of the present invention also lie in:

[0013] The specific process of constructing the commodity detection network in Step 2 is as follows:

[0014] Step 2.1.1: Build an image acquisition platform, collect commodity images from different angles and perform manual annotation;

[0015] Step 2.1.2: Perform horizontal or vertical flipping, mirroring and rotation operations on the collected commodity images to expand the commodity data set;

[0016] Step 2.1.3: Generate a commodity data set in the real scene by segmenting a single commodity image from the shelf background;

[0017] Step 2.1.4: Divide the collected commodity data set, the expanded commodity data set and the synthesized commodity data set into a training set, a validation set and a test set according to the ratio of 8:1:1;

[0018] Step 2.1.5: Use the YOLOv5 object detection network to train the divided data set to obtain the commodity detection network.

[0019] Step 2.1.3 specifically includes the following steps:

[0020] Step 2.1.3.1: Crop a single commodity image with a bounding box of one time and two times respectively;

[0021] Step 2.1.3.2: Perform Saliency-Detection on the image with the size of the bounding box twice as large to separate the foreground and background;

[0022] Step 2.1.3.3: Refine the result obtained in Step 2.1.3.2 through the CRF conditional random field, and crop the result with a bounding box of one time to obtain a refined separation result;

[0023] Step 2.1.3.4: Add the result obtained in Step 2.1.3.3 to the product image with a single bounding box, and then make the black background transparent to obtain the segmented product image.

[0024] Step 2.1.3.5: For the segmented result images of all types of products, with each Figure 5 product having an occlusion rate not exceeding 50%, synthesize them with the shelf background image to form a real-scene product dataset.

[0025] The customer trajectory detection process in Step 2 above includes:

[0026] Step 2.2.1: Process the video stream collected in Step 1 in the order of video frames.

[0027] Step 2.2.2: Read the position of the customer detection box in the current frame.

[0028] Step 2.2.3: Perform trajectory processing and state estimation on the customer detection box:

[0029] The state estimation is defined in an eight-dimensional state space as follows:

[0030]

[0031] Among them, (u, v) is the center position of the detection box, γ is the aspect ratio, h is the height, is the velocity of u, v, γ, h in the image coordinates;

[0032] Step 2.2.4: Match the pedestrian target detection box and the valid trajectory, using the motion matching degree and the appearance matching degree:

[0033] Step 2.2.5: Associate the occluded targets through Matching Cascade to obtain the final trajectory ID.

[0034] In Step 2.2.4, the motion matching degree: Use the Mahalanobis distance between the j-th detection box and the i-th trajectory detection box to describe the motion matching degree d (1) (i,j):

[0035]

[0036] Among them, (y i ,S i ) represents the projection of the i-th trajectory distribution into the measurement space, d j represents the state of the j-th detection box; y i represents the predicted value of the i-th trajectory detection box at the current moment, S i represents the covariance matrix of the i-th trajectory detection box in the observation space at the current moment obtained by the Kalman filter;

[0037] Appearance matching degree: The appearance matching degree d is described by the minimum cosine distance between the depth feature vectors of the j-th detection box and the i-th trajectory detection box (2) (i,j):

[0038]

[0039] where r i represents the appearance descriptor of the j-th detection box, represents the depth feature vector of the i-th trajectory detection box; the depth feature vector is obtained by training a CNN model on the pedestrian re-identification dataset MARS. R i represents the set of feature vectors corresponding to the object detection box that has been successfully tracked by the i-th trajectory detection box in the past.

[0040] Step 4 specifically includes the following steps:

[0041] Step 4.1, generate the coordinate vectors of all commodity and customer detection boxes based on the results obtained in Step 2:

[0042] C i =[(x, y), (m, n)];

[0043] P j =[(p, q), (e, f)];

[0044] where C i is the coordinate vector of the i-th customer trajectory box, (x, y) is the upper left coordinate of the i-th customer trajectory box, (m, n) is the lower right coordinate of the i-th customer trajectory box, and P j is the coordinate vector of the j-th commodity detection box, (p, q) is the upper left coordinate of the j-th commodity detection box, and (e, f) is the lower right coordinate of the j-th commodity detection box;

[0045] Step 4.2, calculate the matching set M of the trajectory box x < p, y < q, m > e, n > f of the i-th customer in the current frame with all commodities i ,

[0046] where M i ∈P j , M i satisfies x < p, y < q, m > e, n > f;

[0047] Step 4.3, if commodity j belongs to the matching sets of multiple customers at the same time, calculate the distances between the center point coordinates of the detection boxes of commodity j and all matching customers and sort them, and take the result with the closest distance as the final matching result of commodity j and the customer.

[0048] Step 5 specifically includes the following steps:

[0049] Step 5.1, the moment the customer takes out the product from the shelf, the shelf gravity sensor transmits the current reduction of the shelf weight back;

[0050] Step 5.2, according to the final matching result in Step 4, it is concluded that the customer matches the product taken. Combining the gravity sensing data in Step 5.1, it is confirmed that the customer has purchased the product, and a purchase list is generated.

[0051] The beneficial effects of the present invention are as follows:

[0052] 1. By performing deep learning model inference on the video stream and using weight sensors to achieve dynamic product recognition of intelligent shelves, it avoids the problem that a single weight sensing technology can only shelve products with different weights. There is no need to manually attach RFID wireless radio frequency tags to all shelved products, saving labor costs. There are no special requirements for the placement and layout structure of the shelf products, reducing operation and management costs. By collecting and training the product data set, new products can be quickly shelved, and it is easy for merchants to use;

[0053] 2. It can simultaneously identify the purchase behaviors of multiple customers, assign IDs to customers, and respectively detect what products are purchased corresponding to the IDs, and generate a purchase list;

[0054] 3. Dual judgment and recognition by vision and gravity sensing avoid missed detection and false detection caused by a single vision technology. The recognition process has strong stability and strong anti-interference ability. Description of the Drawings

[0055] Figure 1 is a schematic structural diagram of the shelf in the method for dynamic product recognition of the shelf based on deep learning and customer shopping matching of the present invention;

[0056] Figure 2 is a flow chart of the method for dynamic product recognition of the shelf based on deep learning and customer shopping matching of the present invention;

[0057] Figure 3 is a flow chart of constructing a product detection network in the method for dynamic product recognition of the shelf based on deep learning and customer shopping matching of the present invention;

[0058] Figure 4 is a flow chart of constructing a customer trajectory detection network in the method for dynamic product recognition of the shelf based on deep learning and customer shopping matching of the present invention;

[0059] Figure 5 is a detection effect diagram of the method for dynamic product recognition of the shelf based on deep learning and customer shopping matching of the present invention.

[0060] In the figure, 1. Shelf, 2. Camera, 3. Gravity sensor, 4. Commodity, 5. Detection area, 6. Back-end server, 7. Commodity a, 8. Commodity b, 9. Detection frame for commodity a, 10. Detection frame for commodity b, 11. Customer c, 12. Customer d, 13. Trajectory frame for customer c, 14. Trajectory frame for customer d. Detailed implementation manners

[0061] The present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0062] The method for identifying dynamic commodities on a shelf and matching customer shopping based on deep learning of the present invention adopts an intelligent shelf, such as Figure 1 shown, which includes a camera 2 and a gravity sensor 3. The camera 2 and the gravity sensor 3 are connected to the back-end server 6 to transmit real-time data back. The camera 2 records all the behaviors of purchasing commodities 4 in the purchasing area 5, and the gravity sensor 3 respectively records the gravity sensing data of the corresponding shelf area.

[0063] The method for identifying dynamic commodities on a shelf and matching customer shopping based on deep learning of the present invention, as Figure 2 shown, is specifically implemented according to the following steps:

[0064] Step 1, when a customer has a purchasing behavior in the detection area 5, the camera a2 collects the video stream of the customer's purchasing behavior in the detection area 5 in front of the shelf, and the gravity sensor 3 collects the real-time gravity sensing data of the corresponding shelf area. The video stream and the gravity sensing data are simultaneously transmitted back to the back-end server 6;

[0065] Step 2, the back-end server 6 inputs each frame of the video stream collected in Step 1 into the commodity detection network and the customer trajectory detection network to obtain the commodity detection frame and the customer trajectory frame;

[0066] As Figure 3 shown, the construction of the commodity detection network in Step 2 includes:

[0067] Step 2.1.1, build an image acquisition platform, collect commodity images at different angles and perform manual annotation;

[0068] Step 2.1.2, perform horizontal or vertical flipping, mirroring and rotation operations on the collected commodity images to expand the commodity data set;

[0069] Step 2.1.3, generate a real-scene commodity data set by segmenting a single commodity image from the shelf background;

[0070] Step 2.1.3 specifically includes the following steps:

[0071] Step 2.1.3.1, crop a single commodity image with a one-fold and a two-fold annotation frame;

[0072] Step 2.1.3.2: Perform Saliency-Detection on the image with twice the size of the annotation box to separate the foreground and background areas;

[0073] Step 2.1.3.3: Refine the result obtained in Step 2.1.3.2 through CRF (Conditional Random Field), and crop the result with a single-size annotation box to obtain a refined separation result;

[0074] Step 2.1.3.4: Add the result obtained in Step 2.1.3.3 to the commodity image with a single-size annotation box, and then make the black background transparent to obtain the segmented commodity image;

[0075] Step 2.1.3.5: For the result images after segmentation of all types of commodities, with each Figure 5 piece of commodity having an occlusion rate not exceeding 50%, synthesize them with the shelf background image to form a real-scene commodity dataset.

[0076] Step 2.1.4: Divide the collected commodity dataset, the augmented commodity dataset, and the synthesized commodity dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1;

[0077] Step 2.1.5: Use the YOLOv5 object detection network to train the divided dataset to obtain a commodity detection network.

[0078] As Figure 4 shown, the process of constructing the customer trajectory detection in Step 2 includes:

[0079] Step 2.2.1: Process the video stream collected in Step 1 in the order of video frames;

[0080] Step 2.2.2: Read the position of the customer detection box in the current frame;

[0081] Step 2.2.3: Perform trajectory processing and state estimation on the customer detection box:

[0082] The state estimation is defined in an eight-dimensional state space:

[0083]

[0084] Among them, (u, v) is the center position of the detection box, γ is the aspect ratio, h is the height, are their respective velocities in the image coordinates,

[0085] The trajectory processing is updated using a Kalman filter, adopting a constant velocity model and a linear observation model;

[0086] Step 2.2.4: Match the pedestrian target detection box and the valid trajectory, mainly using the motion matching degree and the appearance matching degree:

[0087] Motion matching degree:

[0088] Use the Mahalanobis distance between the j-th detection box and the i-th trajectory detection box to describe the motion matching degree d (1) (i, j):

[0089]

[0090] where (y i , S i ) represents the projection of the i-th trajectory distribution onto the measurement space, and d j represents the state of the j-th detection box; y i represents the predicted value of the i-th trajectory detection box at the current moment, and S i represents the covariance matrix of the i-th trajectory detection box in the observation space obtained by the Kalman filter;

[0091] Appearance matching degree: Use the minimum cosine distance between the depth feature vectors of the j-th detection box and the i-th trajectory detection box to describe the appearance matching degree d (2) (i, j):

[0092]

[0093] where r i represents the appearance descriptor of the j-th detection box, represents the depth feature vector of the i-th trajectory detection box; the depth feature vector is obtained by training a CNN model on the pedestrian re-identification dataset MARS. R i represents the set of feature vectors corresponding to the i-th trajectory detection box after successfully tracking the object detection box in the past.

[0094] Step 2.2.5, associate the occluded targets through Matching Cascade to obtain the final trajectory ID, as Figure 5 shown, the customer c trajectory box 13 is marked with the customer c11 trajectory ID, and the customer d trajectory box 14 is marked with the customer d12 trajectory ID.

[0095] Step 3, in the detection time series, as Figure 5 shown, judge the taking-out and putting-in states of product a7 and product b8 by the movement of product a detection box 9 and product b detection box 10 in the image;

[0096] Step 4, as Figure 5 shown, match the customer trajectory ID with the product taken according to the coordinate vectors of product a detection box 9 and customer c trajectory box 13 in the key frame;

[0097] Step 4 specifically includes the following steps:

[0098] Step 4.1: Generate the coordinate vectors of all product and customer detection frames based on the results obtained in Step 2:

[0099] C i = [(x, y), (m, n)];

[0100] P j = [(p, q), (e, f)];

[0101] where C i is the coordinate vector of the i-th customer trajectory frame, (x, y) is the upper left coordinate of the i-th customer trajectory frame, and (m, n) is the lower right coordinate of the i-th customer trajectory frame. And P j is the coordinate vector of the j-th product detection frame, (p, q) is the upper left coordinate of the j-th product detection frame, and (e, f) is the lower right coordinate of the j-th product detection frame;

[0102] Step 4.2: Calculate the matching set M i ,

[0103] of the x < p, y < q, m > e, n > f of the i-th customer's trajectory frame in the current frame with all products i where M j ∈ P i and x < p, y < q, m > e, n > f in M;

[0104] Step 4.3: If product j belongs to the matching sets of multiple customers simultaneously, calculate the distances between the center point coordinates of the detection frames of product j and all matching customers and sort them, and take the result with the shortest distance as the final matching result between product j and the customer.

[0105] Step 5: Based on the real-time gravity sensing data of the corresponding area of the shelf, assist in judging whether the customer takes out or puts in products and generate a purchase list.

[0106] The specific steps in Step 5 include the following:

[0107] Step 5.1: As Figure 5 shown, at the moment when customer c11 takes out product a7 from the shelf, the shelf gravity sensor transmits that the current shelf weight has decreased by 500 g;

[0108] Step 5.2: According to the final matching result in Step 4, it is obtained that customer c11 is matched with product a7. Combining with the gravity sensing data in Step 5.1, it is confirmed that customer c11 has purchased product a7 and a purchase list is generated.

[0109] The characteristics of the shelf dynamic product recognition and customer shopping matching method based on deep learning are:

[0110] 1. Dynamic commodity recognition of intelligent shelves is achieved by performing deep learning model inference on the video stream and weight sensors, avoiding the problem that the single weight sensing technology can only shelve commodities with different weights. There is no need to manually attach RFID wireless radio frequency tags to all shelved commodities, saving labor costs. There are no special requirements for the placement and layout structure of the commodities on the shelves, reducing operation and management costs. By collecting and training the commodity data set, new commodities can be quickly shelved, and it is easy for merchants to use;

[0111] 2. As Figure 5 shown, the purchase behaviors of multiple customers can be recognized simultaneously, assign IDs to the customers, detect respectively what commodities are purchased corresponding to the IDs, and generate a purchase list;

[0112] 3. In step 5, double judgment and recognition are performed through vision and gravity sensing, avoiding missed detection and false detection caused by a single vision technology. The recognition process has strong stability and strong anti-interference ability.

Claims

1. A method for dynamic commodity recognition on shelves and customer shopping matching based on deep learning, characterized in that, it specifically includes the following steps: Step 1, when a customer has a purchase behavior in the detection area, camera a collects the video stream of the customer's purchase behavior in front of the shelf, and the gravity sensor collects the real-time gravity sensing data of the corresponding area of the shelf. The video stream and the gravity sensing data are simultaneously transmitted back to the background server; Step 2, for the video stream collected in Step 1, each frame of the image is input into the commodity detection network and the customer trajectory detection network to obtain the commodity detection frame and the customer trajectory frame; The specific process of constructing the commodity detection network in Step 2 is as follows: Step 2.1.1, build an image acquisition platform, collect commodity images from different angles and perform manual annotation; Step 2.1.2, perform horizontal or vertical flipping, mirroring and rotation operations on the collected commodity images to expand the commodity data set; Step 2.1.3, generate a real-scene commodity data set by segmenting a single commodity image from the shelf background; The specific steps of Step 2.1.3 are as follows: Step 2.1.3.1, crop the single commodity image with the annotation box at one time and twice respectively; Step 2.1.3.2, perform Saliency-Detection on the image with the annotation box size twice, and separate the foreground and background; Step 2.1.3.3, refine the result obtained in Step 2.1.3.2 through the CRF conditional random field, and crop the result with the annotation box at one time to obtain a refined separation result; Step 2.1.3.4, accumulate the result obtained in Step 2.1.3.3 with the commodity image with the annotation box at one time, and then make the black background transparent to obtain the segmented commodity image; Step 2.1.3.5, for the segmented result images of all types of commodities, synthesize a real-scene commodity data set with five commodities per image and a occlusion rate not exceeding 50% with the shelf background image; Step 2.1.4, divide the collected commodity data set, the expanded commodity data set and the synthesized commodity data set into a training set, a validation set and a test set according to the ratio of 8:1:1; Step 2.1.5, use the YOLOv5 object detection network to train the divided data set to obtain the commodity detection network; Step 3, in the detection time series, judge the state of commodity taking out and putting in by the movement of the commodity detection frame in the image; Step 4, match the customer ID with the taken commodity according to the distance between the center point coordinates of the commodity detection frame and the customer trajectory frame in the key frame; Step 5, according to the real-time gravity sensing data of the corresponding area of the shelf, assist in judging the customer's taking out and putting in of commodities and generate a purchase list.

2. The method for dynamic commodity recognition on shelves and customer shopping matching based on deep learning according to claim 1, characterized in that, the customer trajectory detection process in Step 2 includes: Step 2.2.1, process the video stream collected in Step 1 in the order of video frames; Step 2.2.2, read the position of the customer detection frame in the current frame; Step 2.2.3, perform trajectory processing and state estimation on the customer detection frame: The state estimation is defined in an eight-dimensional state space, as follows: ； Among them, (u, v) is the center position of the detection box, is the aspect ratio, is the height, is the speed in the image coordinates; Step 2.2.4, match the pedestrian target detection box and the valid trajectory, using the motion matching degree and the appearance matching degree: Step 2.2.5, associate the occluded targets through the Matching Cascade to obtain the final trajectory ID.

3. The method for identifying dynamic commodities on the shelf and matching customer shopping based on deep learning as claimed in claim 2, characterized in that In step 2.2.4, the motion matching degree: The Mahalanobis distance between the j-th detection box and the i-th trajectory detection box is used to describe the motion matching degree : ； Among them, represents the projection of the i-th trajectory distribution into the measurement space, represents the state of the j-th detection box; y i represents the predicted value of the i-th trajectory detection box at the current moment, S i represents the covariance matrix of the i-th trajectory detection box in the observation space at the current moment obtained by the Kalman filter; Appearance matching degree: The appearance matching degree is described by the minimum cosine distance between the depth feature vectors of the j-th detection box and the i-th trajectory detection box. : ； Among them, represents the appearance descriptor of the j-th detection box, represents the depth feature vector of the i-th trajectory detection box; the depth feature vector is obtained by training a CNN model on the pedestrian re-identification dataset MARS; R i represents the set of feature vectors corresponding to the i-th trajectory detection box after successfully tracking the object detection box in the past.

4. The method for identifying dynamic commodities on the shelf and matching customer shopping based on deep learning as claimed in claim 1, characterized in that The specific steps of step 4 are as follows: Step 4.1, generate the coordinate vectors of all commodity and customer detection boxes according to the results obtained in step 2: ； ； Among them, C i is the coordinate vector of the i-th customer trajectory box, (x, y) is the upper left coordinate of the i-th customer trajectory box, and (m, n) is the lower right coordinate of the i-th customer trajectory box, where P j is the coordinate vector of the j-th commodity detection box, (p, q) is the upper left coordinate of the j-th commodity detection box, and (e, f) is the lower right coordinate of the j-th commodity detection box; Step 4.2, calculate the trajectory box of the i-th customer in the current frame matching set with all commodities , Among them, , in ; Step 4.3, if commodity j belongs to the matching sets of multiple customers at the same time, calculate the coordinate distances between the center points of the detection boxes of commodity j and all the matching customers and sort them, and take the result with the closest distance as the final matching result between commodity j and the customer.

5. The method for identifying dynamic commodities on the shelf and matching customer shopping based on deep learning as claimed in claim 1, characterized in that The specific steps of step 5 are as follows: Step 5.1, at the moment when the customer takes out a commodity from the shelf, the shelf gravity sensor transmits back that the current weight of the shelf decreases; Step 5.2, based on the final matching result of step 4, it is concluded that the customer matches the commodity taken, and in combination with the gravity sensing data of step 5.1, it is confirmed that the customer has purchased the commodity, and a purchase list is generated.

Images