A product recognition method based on YOLO detection and feature matching

Abstract

The present invention provides a product identification method based on YOLO detection and feature matching. The present invention combines the feature point information of two cameras to correct the original product confidence. Based on the original YOLO network, the present invention detects the images captured by the two cameras separately, adds a new confidence improvement module, designs a scoring strategy, and adjusts the final product confidence according to the result of feature matching. The confidence improvement module includes a feature point matching module and a confidence calculation module. The feature point matching module detects the valid frame of the product and the detection frame coordinate information and product confidence information contained therein according to YOLO, extracts feature points, and performs feature point matching on the product images of the two perspectives, and outputs the matching results; the confidence calculation module combines the information output by the YOLO network and the feature point matching module, and performs weighted fusion on the product confidence according to the set scoring strategy to determine the final product confidence.

Description

Technical Field

[0001] The present invention relates to a commodity recognition method based on YOLO detection and feature matching, and belongs to the field of computer vision. Background Art

[0002] With the development of new retail, unmanned vending machines have become a new retail trend. Two cameras are installed on the edge of the door frame of the smart vending machine. As the user takes items, the cameras simultaneously identify the category and quantity of the items picked up, thereby generating a bill for the user's purchase. This method has no restrictions on the display of goods within the cabinet, maximizing the effective volume within the cabinet and reducing operating costs. However, the data of the items taken from a single perspective may be subject to occlusion, resulting in low recognition accuracy. Therefore, the product data collected by the two cameras must be separately detected and the resulting confidence information is weighted and fused based on the matching of feature points. This facilitates the subsequent calculation of the user's final consumption result.

[0003] The existing product recognition methods are based on the fact that the target objects in the video data are complete and clear without any obstructions. In addition, the products displayed in the vending machine are detected in a static state, rather than dynamically detecting the process of users picking up the products. In this case, the target detection results under a single camera can be used as the final prediction. Zhang H, Li D et al. [1] A dataset of 155,153 products was constructed from different angles, and data images containing ten types of products were manually annotated using labels and bounding boxes. However, this study used an infrared laser sensor to detect the consumer's hand, thereby taking pictures of the products in the vending machine when the door was opened or closed. The first picture was taken when the user opened the door, and the second picture was taken when the user took out the drink. The products in the smart cabinet were statically detected based on the two pictures, as shown in the attached figure. Figure 1 As shown in the figure, the target objects in the two pictures are completely unobstructed. Product recognition for this type of dataset is relatively simple, but it requires the use of sensors. Fan D, Yang D et al. [2] This paper provides a method to improve the matching accuracy by combining deep learning and image matching. The two can be combined to improve the accuracy of product recognition. Liu L, Cui J et al. [3] A binocular camera is used for product recognition, and a new target detection network is designed based on the YOLOv3 model.

[0004] However, our product data has the following characteristics: the products are in the vending machine, the product category is dynamically identified when the customer takes the product, and the product information is blocked by the hand. Figure 2 Figure 3 As shown in the figure, problems such as false detection and missed detection may occur under a single perspective.

[0005] Since smart locker settlement is determined based on the video information of the user at the time of purchase, there is a problem of low product recognition rate when there is occlusion. Therefore, we weighted the prediction values ​​obtained by two cameras with different perspectives and set up an image feature point matching module to match the feature points of the items recognized by the two cameras, thereby solving the problem of missed detection by a single camera. This can improve the accuracy of product recognition and achieve the purpose of improving the existing product recognition method.

[0006] References:

[0007] [1]Zhang H, Li D, Ji Y, et al.Toward new retail:Abenchmark dataset for smart unmanned vending machines[J].IEEETransactions on Industrialinformatics, 2019, 16(12):7722-7731.

[0008] [2]Fan D, Yang D, Zhang Y.Satellite image matching method based on deepconvolutional neural network[J].Journal ofGeodesy and Geoinformation Science, 2019, 2(2):90.

[0009] [3] Liu L, Cui J, Huan Y, et al. A Design of SmartUnmanned Vending Machine for New Retail Based on BinocularCamera and Machine Vision[J]. IEEE Consumer Electronics Magazine, 2021. Summary of the Invention

[0010] Existing product recognition methods are suitable for images of complete objects with minimal obstruction. This differs from images of products in smart lockers, where customers' hands can significantly obstruct product information when picking up items. This can lead to false detections and missed detections when the left and right cameras independently identify the items, significantly impacting the subsequent determination of the customer's purchase bill. To address this issue, the present invention adds a new confidence improvement module to the existing YOLO network, which detects images captured by the two cameras separately. This module then designs a scoring strategy based on the feature matching results, thereby adjusting the final product confidence.

[0011] The technical solution adopted by the present invention is mainly composed of the following three parts: a product target detection module, a product image feature matching module and a confidence prediction module. The product target detection module predicts the category and coordinate position of the product from the entire image through a convolutional neural network, and the predicted target frame also contains information such as the product's confidence and classification possibility. The feature point matching module combines YOLO to detect the valid frame of the product and the detection frame coordinate information and product confidence information it contains, and performs feature point matching on the product images of the two perspectives to output the matching results. The score calculation module combines the information output by the above-mentioned YOLO network and the feature point matching module, and performs weighted fusion on the product confidence according to the set scoring strategy, thereby determining the final product confidence.

[0012] The specific solution of the present invention is shown in the attached Figure 3 shown.

[0013] Step 1: Smart cabinet product target detection

[0014] The purpose of this step is to predict the category and coordinate position of the product from the entire image through a convolutional neural network. The predicted target box also contains information such as the confidence level of the product and the possibility of classification.

[0015] Step 1.1: Simultaneously obtain image frames from the left and right cameras: The two front-end edge devices simultaneously sample during the time period when the user picks up the product and send the samples to the cloud server.

[0016] Step 1.2 YOLO object detection: The images captured by the left and right cameras are passed as input to the YOLO network model. The feature extraction network extracts features from the input product images to obtain a feature map of a certain size. The output feature map can extract information such as the number of categories, confidence, prediction box coordinates, width and height.

[0017] Step 2: Product Image Matching

[0018] The purpose of this module is to use the image information from the left and right cameras to record the number of successfully matched feature points and related information such as the feature point coordinates under the two perspectives, preparing for the subsequent frame-by-frame calculation of product confidence.

[0019] Step 2.1 Image to Grayscale: Converting product images to grayscale images can reduce the amount of calculation and provide a basis for subsequent feature point detection.

[0020] Step 2.2 Feature point detection: Use the FAST algorithm to detect feature points. Feature points refer to points where the grayscale value of the image changes dramatically or points with large curvature at the edge of the image. They can reflect the essential characteristics of the image and identify the target object in the image.

[0021] Step 2.3: Calculate the feature descriptor: Use the key points extracted in step 2.2 as the center of the circle, use the BRIEF descriptor to establish a coordinate system, and describe the image area around the feature points extracted in the previous step.

[0022] Step 2.4 Feature matching: After steps 2.2 and 2.3, corner point matching is performed. For each successfully matched feature point, its coordinates, neighborhood diameter, direction, and number are recorded to facilitate subsequent prediction of product confidence using the YOLO detection frame.

[0023] Step 3: Product confidence prediction

[0024] The purpose of this step is to use the detection results of the feature point matching network to adjust the product confidence detected by YOLO and perform weighted fusion on the information of each frame to obtain the final product confidence information.

[0025] Step 3.1 Calculate product confidence frame by frame: We combine the product confidence detected by the left and right cameras in each frame with the number of feature point matches and perform a weighted sum to obtain the final score.

[0026] Step 3.2: Summarize and calculate the product confidence: average the confidence obtained for each frame output in step 3.1 and perform normalization to obtain the final product confidence.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The product recognition method based on YOLO detection and feature matching adopted in the present invention leverages the advantages of dual cameras on the basis of YOLO target recognition, dynamically identifies products, and combines the feature point information of the two cameras to correct the original product confidence to a certain extent.

[0029] Second, the present invention uses a product recognition method based on YOLO detection and feature matching, scoring each frame to compensate for false detections and missed detections. For example, in two images produced by the left and right cameras at the same time, the left camera image correctly detects product g, but the right camera image fails to successfully detect it. However, through feature point matching, the frame range of product g in the left camera image identified by YOLO contains feature points that match the right camera image. In this case, we appropriately add points to compensate for the defect that YOLO did not detect product g in the right camera. The second case is that in two images produced by the left and right cameras at the same time, YOLO correctly detects product g, and the range of the frame is consistent with our feature point matching. In this case, we increase YOLO's confidence, that is, the credibility of "the buyer took product g" is higher. The third case is a YOLO misdetection. For example, YOLO misdetects product g and identifies a matching box, but we cannot find the corresponding matching point in the box in the other image. In this case, appropriate score reduction and lowering of the confidence level of product g can compensate for the error in the confidence level of the final product g caused by YOLO's misdetection of a certain frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the type of data sample that existing product identification methods usually target.

[0031] Figure 2 This is the image captured by the left camera of the smart cabinet.

[0032] Figure 3 Image captured by the right camera of the smart cabinet

[0033] Figure 4 This is the overall model diagram of the present invention. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings:

[0035] This invention relates to a product recognition method based on YOLO detection and feature matching. This method uses images from the left and right cameras at the same moment as input, feeds them into the YOLO object detection network, and records the confidence level of the detected product category. This information is then combined with the number of feature points that successfully match the features from the two viewpoints to perform a scoring fusion. The specific process is as follows: In the first stage, images from the left and right cameras are captured at the same moment. YOLO detection is then used to predict the product category and coordinate position from the entire image using a convolutional neural network, recording the product confidence level and the position of the detection box. In the second stage, the two video frames are converted into grayscale images, and their feature points are extracted to calculate feature point descriptors. Next, feature point matching is performed on the two image frames, recording the feature point coordinates and the number of matching points in the images. Matching points within the detection box are then selected based on the position information of the YOLO detection box. In the third stage, the information output by the YOLO network and the feature point matching module is combined to calculate the product confidence level frame by frame according to a predefined scoring strategy. The product confidence levels are then weighted and fused, and finally normalized to obtain the product confidence level.

[0036] Specifically, the method comprises the following steps:

[0037] Step 1: Smart cabinet product target detection

[0038] The purpose of this module is to predict the category and coordinate location of the product from the entire image through a convolutional neural network.

[0039] Step 1.1: Simultaneously acquire image frames from both cameras. The two edge devices (left and right cameras) sample images at the same intervals, when the user opens the door, takes an item, and closes the door. These frames are aggregated and sent to the cloud server. The cloud server then numbers the 2 × m images in chronological order and feeds them into the YOLO network.

[0040] Step 2.2 YOLO Object Detection: The image frames captured by the left and right cameras are fed into the YOLO network model as input. Mosaic image enhancement is performed at the input of the YOLO network to make the product images clearer. During model inference, the image is adaptively scaled to reduce black edges, which can improve object detection speed. After preprocessing, the product image is fed into the Backbone network, which slices the product image to extract common product features. The image is then fed into the Neck network, which uses an FPN+PAN architecture to better extract features from the backbone network, enhancing feature diversity and robustness. Finally, GIOU_Loss is used as the loss function at the output, and non-maximum suppression is used to filter the target bounding box. The product category information, confidence score, and target bounding box coordinates are recorded.

[0041] Step 2: Product Image Matching

[0042] The purpose of this step is to use the image information from the left and right cameras to record the number of successfully matched feature points and related information such as their coordinates from both perspectives, in preparation for the subsequent frame-by-frame calculation of product confidence.

[0043] Step 2.1 Image to Grayscale: The color of each pixel is composed of three color channels: R, G, and B. Each channel has a value of 0 to 255. The computational complexity is high when performing feature point detection. Therefore, converting the original color product image to a grayscale image can reduce the computational complexity and provide a basis for subsequent feature point detection. Since the human eye has different sensitivities to different colors, it is better to use the weighted average method to obtain a grayscale image, that is, using the following formula:

[0044] R=G=B= (ω R R+ω G G+ω B B) (1)

[0045] Among them, R, G, and B represent R channel, G channel, and B channel respectively, ω R 、ω G 、ω B Represent the weight values ​​of the three channels respectively.

[0046] Step 2.2 Feature point detection: Use the FAST algorithm to detect feature points. If there are enough pixels in the area around the candidate point that have a sufficiently large difference in grayscale value from the candidate point, the candidate point is considered a feature point. Use formula (1) to select the feature point P.

[0047] N=∑|I(X)-I(P)|>ε d (2)

[0048] Where I(X) is the grayscale of any point on the circumference, I(P) is the grayscale of the center of the circle, and ε d It is the threshold of gray value difference, which is generally three quarters of the surrounding circle points. If N is greater than the given threshold, p is considered to be a feature point.

[0049] Step 2.3 Calculate the feature descriptor: Using the feature point extracted in step 2.2 as the center of the circle, use the BRIEF descriptor to establish a coordinate system, describe the image area around the feature point, and obtain a binary encoding information for each feature point. The specific method is to use the extracted feature point as the center, select an S×S area as P, and select nk point pairs nk(x,y) within P to define T, where p(x) is the size of the pixel value at point x, and p(y) is the size of the pixel value at point y. The specific encoding rules are as follows:

[0050]

[0051] If the pixel value at point y is greater, it is counted as 1, otherwise it is 0. Next, nd Ts are sequentially formed into a binary coded string from low to high. The process is expressed as follows:

[0052]

[0053] Step 2.4 Feature Matching: After steps 2.2 and 2.3, each feature point in both images is assigned a binary code. The Hamming distance is then used to determine feature pairings. For successfully matched feature points, their coordinates, neighborhood diameter, direction, and number are recorded to facilitate subsequent integration with the YOLO detection box to predict product confidence.

[0054] Step 3: Product confidence prediction

[0055] The purpose of this step is to use the feature point matching network detection results to adjust the confidence of the products detected by YOLO and perform weighted fusion on the information of each frame to obtain the final product confidence information.

[0056] Step 3.1 Calculate the product confidence frame by frame: Take whether the buyer has purchased product g as an example, only keep the frames captured by the left camera or the right camera at a certain moment where YOLO detects product g (that is, YOLO determines that the product exists with a confidence level greater than the preset confidence level), including the frames recorded by the left camera at that moment and the frames recorded by the right camera at that moment. Record them as "valid frames". Suppose there are 2×n frames in total, and they are numbered G according to the left and right cameras. Lgi ~G Lgn , G Rgi ~G Rgn . Secondly, let G output by YOLO be Lgi The confidence level of a product g in the image is C Lgi The detection box is B Lgi .G Rgi The confidence and detection box are similarly recorded as C Rgi and B Rgi , where i∈[1,n]. G Lgi and G Rgi The pairs are fed into the feature point matching network, and n groups of feature point matching results are obtained. Assume that there are k pairs of feature points that are successfully matched in group i, where each pair of feature point matching information includes the coordinate information of the two feature points from the left and right camera images, neighborhood diameter information, direction information, and response strength information. gi (Including G Lgi and G Rgi ) of the product g giThe scoring strategy is as follows:

[0057] C gi =(T Lgi ×C Lgi +T Rgi ×C Rgi ) (5)

[0058] Where T Lgi B Lgi The total number of feature points in the range that are successfully matched. Similarly, T Rgi B Rgi The total number of feature points in the range that were successfully matched.

[0059] Step 3.2: Summarize and calculate the product confidence: average the confidence obtained for each frame in step 3.1, and then use the Sigmoid function to normalize it to obtain the final product confidence C g , the formula is as follows:

[0060]

Claims

1. A commodity recognition method based on YOLO detection and feature matching, characterized in that The following steps are involved: Step 1: Smart cabinet product target detection The convolutional neural network is used to predict the category and coordinate location of the product from the entire image, and the predicted target box also includes the confidence level and classification possibility of the product; Step 1.1: Simultaneously acquire image frames from the left and right cameras: The two front-end edge devices simultaneously sample the image frames during the time period when the user picks up the product and send them to the cloud server. Step 1.2 YOLO object detection: The images captured by the left and right cameras are fed into the YOLO network model as input. The feature extraction network extracts features from the input product images to generate a feature map. The output feature map contains information about the number of categories, confidence level, predicted box coordinates, and width and height. Step 2: Product Image Matching Using the image information from the left and right cameras, record the number of successfully matched feature points and their coordinates under the two viewing angles; Step 2.1 Convert image to grayscale: Step 2.2 Feature point detection: Use FAST algorithm to detect feature points; Step 2.3 Calculate the feature descriptor: Use the feature point extracted in step 2.2 as the center of the circle, use the BRIEF descriptor to establish a coordinate system, and describe the image area around the feature point extracted in the previous step; Step 2.4 Feature matching: After steps 2.2 and 2.3, use corner point matching. For the successfully matched feature points, record their coordinates, neighborhood diameter, direction, and number. Step 3: Product confidence prediction Using the detection results of the feature point matching network, we adjust the product confidence level detected by YOLO and perform weighted fusion on the information of each frame to obtain the final product confidence level information. Step 3.1 Calculate the product confidence score frame by frame: Combine the product confidence scores detected by the left and right cameras with the number of feature point matches in each frame and perform a weighted sum to obtain the final score. Step 3.2: Summarize and calculate the product confidence: average the confidence obtained for each frame output in step 3.1 and perform normalization to obtain the final product confidence.

2. The method according to claim 1, characterized in that The following steps are involved: Step 1: Smart cabinet product target detection Use convolutional neural networks to predict the category and coordinate location of the product from the entire image; Step 1.1: Simultaneously acquire image frames from both the left and right cameras: The left and right cameras each sample at the same interval when the user opens the door, takes the product, and closes the door. These frames are aggregated and sent to the cloud server. After receiving the 2×m images, the server on the cloud numbers them in chronological order and inputs them into the YOLO network one by one. Step 2.2 YOLO target detection: The image frames captured by the left and right cameras are passed as input to the YOLO network model. The product image is preprocessed at the input of the YOLO network and then sent to the Backbone network. The product image is sliced ​​to extract product features. The image is then sent to the Neck network, which uses an FPN+PAN structure. Finally, GIOU_Loss is used as the loss function at the output, and non-maximum suppression is used to filter the target box. At this time, the product category information, confidence level, and target box coordinate information are recorded. Step 2: Product Image Matching Using the image information from the left and right cameras, the system records the number and coordinates of successfully matched feature points from both perspectives, preparing for the subsequent frame-by-frame calculation of product confidence. Step 2.1 Image to grayscale: The color of each pixel is composed of three color channels: R, G, and B. Each channel has a value of 0 to 255. The computational complexity is large when performing feature point detection. The weighted average method is used to obtain a grayscale image with better results. The following formula is used: R=G=B=(ω R R+ω G G+ω B B) (1) Among them, R, G, and B represent R channel, G channel, and B channel respectively, ω R 、ω G 、ω B Represent the respective weight values ​​of the three channels; Step 2.2 Feature point detection: Use the FAST algorithm to detect feature points. If there are enough pixels in the area around the candidate point that have a sufficiently large difference in grayscale value from the candidate point, the candidate point is considered a feature point. Use formula (1) to select the feature point P. N=∑|I(X)-I(P)|>ε d (2) Where I(X) is the grayscale of any point on the circumference, I(P) is the grayscale of the center of the circle, and ε d is the threshold of gray value difference, which is three-quarters of the surrounding circle points. If N is greater than the given threshold, p is considered to be a feature point; Step 2.3 Calculate the feature descriptor: Based on the key point extracted in step 2.2 as the center of the circle, use the BRIEF descriptor to establish a coordinate system, describe the image area around the feature point, and obtain a binary encoding information for each feature point; the specific method is to take the extracted feature point as the center, select an S×S area as P, select nk points in P and define nk(x,y) as T, p(x) is the size of the pixel value of point x, and similarly p(y) is the size of the pixel value of point y. The specific encoding rules are as follows: If the pixel value at point y is large, it is counted as 1, otherwise it is 0. Next, nd Ts are sequentially formed into a binary coded string from low to high, which can be expressed as: Step 2.4 Feature Matching: After steps 2.2 and 2.3, each feature point in the two images is assigned a binary code. The Hamming distance is then used to determine feature pairings. For successfully matched feature points, their coordinates, neighborhood diameter, direction, and number are recorded to facilitate subsequent product confidence prediction using the YOLO detection box. Step 3: Product confidence prediction Using the feature point matching network detection results, we adjust the confidence of the products detected by YOLO and perform weighted fusion on the information of each frame to obtain the final product confidence information. Step 3.1 Calculate the product confidence frame by frame: Based on whether the buyer purchased product g, only retain the frames captured by the left or right camera at a certain moment in which YOLO detected product g, that is, the frames in which YOLO judged the presence of product g greater than the preset confidence level, including the frames recorded by the left camera at that moment and the frames recorded by the right camera at that moment; Recorded as "valid frame", there are 2×n frames in total, which are numbered G according to the left and right cameras. Lgi ~G Lgn , G Rgi ~G Rgn ; Secondly, let G output by YOLO Lgi The confidence level of a product g in the image is C Lgi The detection box is B Lgi ; G Rgi The confidence and detection box are similarly recorded as C Rgi and B Rgi , where i∈[1,n]; G Lgi and G Rgi The feature points are sent into the feature point matching network in pairs, and n groups of feature point matching results are obtained. Assume that there are k pairs of feature points that are successfully matched in group i, where the matching information of each pair of feature points includes the coordinate information of the two feature points from the left and right camera images, the neighborhood diameter information, the direction information, and the response strength information. Then, for G gi (Including G Lgi and G Rgi ) of the product g gi The scoring strategy is as follows: C gi =(T Lgi ×C Lgi +t Rgi ×C Rgi ) (5) where t Lgi B Lgi The total number of feature points in the range that are successfully matched. Similarly, T Rgi B Rgi The total number of feature points in the range that have been successfully matched; Step 3.2: Summarize and calculate the product confidence: average the confidence obtained for each frame in step 3.1, and then use the Sigmoid function to normalize it to obtain the final product confidence C g , the formula is as follows:

Images