A Product Recognition Method Based on Dual-Channel Cameras

Abstract

This invention discloses a product recognition method based on dual cameras, which involves a deep learning-based approach. The method fuses images from two angles for detection, improving product recognition accuracy. By training a feature extraction backbone network using the mixed data, the resulting network can simultaneously extract features from both angles, reducing the training cost of the feature extraction backbone network compared to traditional object detection algorithms.

Description

Technical Field

[0001] This invention relates to a product recognition method based on deep learning, specifically a product recognition method for smart vending machines. Background Technology

[0002] Smart vending machines are an important technology in the new retail era. The industry mainly uses four technical solutions: RFID, gravity sensors, static identification, and dynamic identification. This solution belongs to the dynamic identification category. Dynamic identification typically involves recording the process of a customer opening the door and taking goods using cameras installed on the top or sides of the vending machine. The video is then analyzed to determine the purchased items and process the transaction. Features of dynamic vision solutions include no requirements for product display, stackable products, improved space utilization, and support for selling multiple SKUs. However, to be compatible with the diverse requirements of various merchants and products, dynamic vision recognition demands higher image recognition accuracy. Some solutions improve accuracy by adding cameras to the vending machine, but commonly used recognition algorithms are based on a single camera, such as the widely used YOLO series algorithms and R-CNN series algorithms. Their detection methods involve extracting features, locating, and classifying the image captured by a single camera. However, in multi-camera scenarios, due to the different perspectives of the cameras, information about consumers picking up goods from different angles can be recorded. Therefore, this application proposes a model for fusing and detecting video frames from multiple angles in the same scene, thereby improving the accuracy of product category recognition. Summary of the Invention

[0003] The purpose of this invention is to address the problems existing in the prior art and to provide a new product recognition method based on dual-camera vending machines, taking into account the characteristics of dual-camera vending machines. The method improves the product recognition rate by using an algorithm with feature fusion and attention mechanisms to fuse and detect images captured by dual cameras.

[0004] The technical solution adopted in this invention is as follows:

[0005] A product recognition method based on dual-channel cameras includes dataset construction, model construction, model training, and model recognition.

[0006] Step 1: Dataset Construction

[0007] The main task of dataset construction is to capture keyframes from the videos recorded simultaneously by the main and secondary cameras at the same moment, ensuring the time difference between the captured frames, and to denoise and annotate the captured image frames, under the premise that the shopping scene is captured at the same moment.

[0008] Step 2: Model Building

[0009] The main task of model building is to construct feature extraction networks and corresponding single-channel detection heads for primary and secondary cameras, as well as fusion detection heads required for fusion detection.

[0010] Step 3: Model Training

[0011] The main task of model training is to mix the constructed data and input it into the model, calculate the loss function value, optimize the model parameters through backpropagation and gradient descent algorithms, and obtain the model weights. The model training phase includes training the feature extraction backbone network, training the single-path detection head, and training the fusion detection head.

[0012] Step 4: Reasoning and Recognition

[0013] The main task of inference recognition is to input the image to be detected into the trained model, including the feature extraction backbone network, single-path detection head and fusion detection head, and finally output the coordinates and category information of the target to be detected in the images of the main and secondary cameras.

[0014] The product recognition technology based on dual cameras provided by this invention can bring the following benefits:

[0015] 1. In this invention, images from two angles are fused and detected, which improves the accuracy of product recognition.

[0016] 2. In this invention, the feature extraction backbone network is trained using mixed data. The resulting feature extraction backbone network can extract features from two angles of the image simultaneously, reducing the training cost of the feature extraction backbone network when using traditional object detection algorithms. Attached Figure Description

[0017] Figure 1 This is a flowchart of an embodiment of the product recognition method based on dual cameras of the present invention; Detailed Implementation

[0018] This invention relates to the field of vending machine technology, specifically to product recognition in smart vending machines.

[0019] Step 1: Dataset Construction

[0020] Build the dataset needed for model training.

[0021] Step 1.1 Video Recording

[0022] The system connects to the engineering control system and computer of the smart vending machine via network cable, and uses OpenCV to record the shopping process footage observed simultaneously by the main and secondary cameras in a parallel manner.

[0023] Step 1.2 Video frame capture

[0024] The video is captured by setting the minimum number of frames between two videos of the same shopping process as the upper limit and setting the frame difference to 5, thereby ensuring that there is sufficient change in the position of the product between frames.

[0025] Step 1.3 Noise Reduction

[0026] The image was processed using a 3x3 median filter denoising operator to obtain a denoised shopping process image.

[0027] Step 1.4 Image annotation

[0028] Use the labeling software LabelImg to label the captured images. The labeling target is the item grabbed by the buyer, and the label includes the item's location and category information. The label box should include the outline of the item in the image.

[0029] Step 2: Model Building

[0030] A feature extraction backbone network is constructed, along with single-channel detection heads and fused detection heads for the main and secondary cameras. The specific implementation utilizes the mmdetection deep learning framework.

[0031] Step 2.1 Feature Extraction and Backbone Network Construction

[0032] The feature extraction backbone network adopts the Swin Transformer architecture, which consists of 4 layers. Each layer is composed of 2, 2, 6, and 2 cascaded basic modules, respectively. The channel dimensions of each layer are 96, 192, 384, and 768, respectively, and the number of attention heads are 3, 6, 12, and 24, respectively. The window size of the window attention involved in the basic modules is 7. All other parameters are determined by the default parameters in mmdetection-2.23.0.

[0033] Step 2.2 Construction of single-channel detection heads corresponding to the main and secondary cameras

[0034] The single-channel detection heads for the main and secondary cameras employ the DETRHead architecture. This detection head is an anchorless detection head based on multi-head attention, consisting of 6 encoding layers, 6 decoding layers, and a multilayer perceptron module. Each encoding layer comprises a multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module has 8 attention heads and a query dimension of 256, while the feedforward neural network module has a hidden layer dimension of 2048. Each decoding layer also comprises a multi-head attention module, a neural layer regularization module, a cross-attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. Both the multi-head attention module and the cross-attention module have 8 attention heads and a query dimension of 256, while the feedforward neural network module has a hidden layer dimension of 2048. The bounding box coordinates of the target to be detected are predicted by the multilayer perceptron module of the detection head. All other parameters are determined by the default parameters in mmdetection-2.23.0.

[0035] Step 2.3 Construction of the fusion detection head

[0036] The fusion detection head uses the DETRHead architecture, which is an anchorless detection head based on multi-head attention. It consists of 8 encoding layers, 8 decoding layers, and a linear layer module. The 8 encoding layers, except for the first layer which is composed of a fused multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series, each consist of a multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module has 8 attention heads, the query dimension is 256, and the hidden layer dimension of the feedforward neural network module is 2048. Each decoding layer consists of a multi-head attention module, a neural layer regularization module, a cross-attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module and the cross-attention module each have 8 attention heads, the query dimension is 256, and the hidden layer dimension of the feedforward neural network module is 2048. It should be noted that the fusion detection head also includes other parameters, all of which are determined by the default parameters in mmdetection-2.23.0. The category information of the target to be detected is predicted by the linear layer module.

[0037] Step 3: Model Training

[0038] The model training process uses a pre-built dataset to first train the feature extraction backbone network, then train the single-path detection head, and finally train the fusion detection head by adding a fusion detection head. The specific implementation uses the mmdetection deep learning framework.

[0039] Step 3.1 Feature Extraction Backbone Network Training

[0040] All training data collected by the main and secondary cameras is input into the single-branch object detection network for training. Then, the bounding box coordinates of the inferred target are input into the mean absolute error loss function to calculate the loss, and the target category confidence is input into the cross-entropy loss function to calculate the loss. The two losses are summed as the final loss. The model parameters are optimized using backpropagation and gradient descent to finally obtain the weights corresponding to the feature extraction backbone network, thus completing the training of the feature extraction backbone network. Here, the single-branch object detection network refers to the feature extraction backbone network connected in series with a single-path detection head.

[0041] Step 3.2 Training of single-channel detection heads corresponding to the main and secondary cameras

[0042] First, the feature extraction backbone network trained in step 3.1 is connected in series with the single-channel detector head corresponding to the main camera. After freezing the parameters of the trained feature extraction backbone network using the mmdetection framework, the image data captured by the main camera is input into the model. The bounding box coordinates of the target obtained through inference are input into the mean absolute error loss function to calculate the loss. The parameters of the single-channel detector head are optimized using backpropagation and gradient descent to obtain the weights of the single-channel detector head corresponding to the main camera, thus completing the training of the main camera branch detector head. Then, the same operation is performed on the data and single-channel detector head corresponding to the secondary camera to obtain the weights of the single-channel detector head corresponding to the secondary camera, thus completing the training of the secondary camera branch detector head.

[0043] Step 3.3 Training the fusion detection head

[0044] First, the parameters of the pre-trained feature extraction backbone network and single-path detection heads are frozen using the mmdetection framework. Then, the fusion detection head is configured, such as... Figure 1 As shown, the features output from the encoding parts of the single-channel detection heads corresponding to the main and secondary cameras are input into the fusion detection head. The features of the two branches are fused by performing a Hadamard product with the corresponding fusion weight matrix and then adding the results. The fused feature map is input into the decoding part, and the final linear layer outputs the target class confidence score after fusion detection. The target class confidence score predicted by the fusion detection head is input into the FocalLoss loss function, and the model parameters are optimized using backpropagation to obtain the weights corresponding to the fusion detection head, thus completing the training of the fusion detection head.

[0045] Step 4: Reasoning and Recognition

[0046] For the specified video information to be detected, a frame difference of 5 is set, and keyframes are extracted using the frame difference as the frame interval. The keyframes corresponding to the main and secondary cameras are input into the feature extraction backbone network, and the detection head outputs the category confidence of the detected targets. The single-channel detection head outputs the bounding box coordinates of the targets in the images corresponding to the main and secondary cameras. The specific implementation uses the mmdetection deep learning framework.

[0047] Step 4.1 Feature extraction, backbone network inference and recognition:

[0048] The feature extraction backbone network receives key frames captured simultaneously by the main and secondary cameras. The key frames are RGB images with a height of 480 and a width of 640. The images are input into the feature extraction backbone network for feature extraction. The feature extraction backbone network consists of 12 cascaded basic modules. The calculation process of each basic module is defined by formulas (1) to (4).

[0049]

[0050]

[0051]

[0052]

[0053] In the above formula, WMSA is the fixed window attention module in the Swin Transformer architecture, SWMSA is the sliding window attention module, MLP is the multilayer perceptron module, LN is the neural layer regularization module, l is the ordinal number of the basic module in the integer-referenced feature extraction backbone network, and 1≤l≤12. When l≠1, T l-1 For the features output by the previous module, when l=1, T l -1 For the segmented image patch, T l for The output T obtained after passing through the multilayer perceptron l+1 for The output obtained after passing through a multilayer perceptron. For the output of the fixed-window attention module, This is the output of the sliding window attention module. Assuming the batch size is 32, the sizes of the three sets of features output by the final feature extraction backbone network are 32*192*80*60, 32*384*40*30, and 32*768*20*15, respectively.

[0054] Step 4.2 Single-channel detection head inference and recognition:

[0055] The single-path detection head receives features output from the second, third, and fourth layers of the feature extraction backbone network, performs channel alignment on the features output from the feature extraction backbone network, and then uses them as input to the encoding and decoding part of the single-path detection head. The feature map output by the encoding and decoding structure is used as input to the concatenated MLP, and the output of the MLP is the predicted bounding box coordinates of the target. This completes the inference and recognition of the single-path detection head.

[0056] The feature size output by the feature extraction backbone network is the same as that described in step 4.1. The input received by the single-path detection head encoding structure is a feature map of size 6300*32*256. After position encoding of this feature map, it is recorded as the query of multi-head attention. The key and value of multi-head attention are the feature map without position encoding.

[0057] Step 4.3 Fusion detection head inference and recognition:

[0058] The fusion detection head receives the features output from the encoding parts of two single-path detection heads as input. The size of both features is 6300*32*256. The features are input into the encoding part of the fusion detection head. The multi-head attention module of the first encoding layer of the encoding part is the fusion multi-head attention module. The calculation process of one attention head of this module is defined by formulas (8) to (14), and the calculation of the fusion multi-head attention module is completed by formula (15). In the formula, m corresponds to the main camera branch, s corresponds to the secondary camera branch, and W is the corresponding learnable weight matrix. That is, the learnable weights of the Q matrix corresponding to the main camera branch, T m The output of the encoding section of the single-channel detection head corresponding to the main camera, Q m Then it is the result of the matrix product of the two, and the other letters appearing in formulas (5) to (10) have the same meaning.

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] The Q calculated using formulas (5) to (10) m K m V m , and Qs K s V s Matrix and learnable fusion weight matrix α of the same order k (k = Q, K, V) and β k (k = Q, K, V) perform the Hadamard product, which is defined by formulas (11) to (13), to calculate the fused matrix Q. f K f V f Where α i，j +β i，j ≤1, (α) i，j ∈α m ,β i，j ∈β m ). Where α k and β k The value is obtained during the training of the fusion detection head.

[0066]

[0067]

[0068]

[0069] The calculated Q f K f V f The matrix serves as the input to the attention mechanism, and the attention feature map fused with multi-head attention is calculated using formulas (14) and (15).

[0070]

[0071] MSA fusion =Concat(SA1,SA2,...,SA8)O (15)

[0072] In the above formula, d = 256. For K f The transpose of a matrix, SA t Let t represent the output of the t-th attention head, where t is an integer (1 ≤ t ≤ 8), and O is the multi-head attention weight matrix, whose values ​​are obtained during the training of the fused detection heads. MSA fusion To fuse the outputs of the multi-head attention module, Concat is a tensor concatenation operation. This involves combining the MSA... fusion After the remaining modules of the first layer are input, they continue to be input into the remaining 7 encoding layers and 8 decoding layers. Then, the linear layer of the fusion detection head predicts the category information of the target to be detected, thus completing the reasoning and recognition process of the fusion detection head.

[0073] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A product recognition method based on dual-channel cameras, characterized in that... Includes the following steps: Step 1: Dataset Construction Construct the dataset required for model training; Step 1.1 Video Recording The system connects to the engineering control and computer of the smart vending machine via network cable, and uses OpenCV to record the shopping process footage observed simultaneously by the main and secondary cameras in a parallel manner. Step 1.2 Video frame capture The video is captured with the minimum number of frames recorded from two videos of the same shopping process as the upper limit and a frame difference of 5, so as to ensure that there is enough change in the position of the product between frames. Step 1.3 Noise Reduction The acquired image was processed using a 3x3 median filter denoising operator to obtain a denoised shopping process image; Step 1.4 Image annotation Use the labeling software LabelImg to label the captured images. The labeling target is the item grabbed by the buyer. The label should include the location and type information of the item. The label box should include the outline of the product in the image. Step 2: Model Building A feature extraction backbone network is constructed, along with single-channel detection heads and fusion detection heads corresponding to the main and secondary cameras; the specific implementation uses the mmdetection deep learning framework. Step 2.1 Feature Extraction and Backbone Network Construction The feature extraction backbone network adopts the Swin Transformer architecture, which consists of 4 layers. Each layer is composed of 2, 2, 6, and 2 cascaded basic modules, respectively. The channel dimensions of each layer are 96, 192, 384, and 768, respectively, and the number of attention heads are 3, 6, 12, and 24, respectively. The window size of the window attention involved in the basic modules is 7. All other parameters are determined by the default parameters in mmdetection-2.23.

0. Step 2.2 Construction of single-channel detection heads corresponding to the main and secondary cameras The single-channel detection heads for the main and secondary cameras adopt the DETRHead architecture. This detection head is an anchorless detection head based on multi-head attention, consisting of 6 encoding layers, 6 decoding layers, and a multilayer perceptron module. Each encoding layer is composed of a multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module has 8 attention heads, a query dimension of 256, and the hidden layer dimension of the feedforward neural network module is 2048. Each decoding layer is composed of a multi-head attention module, a neural layer regularization module, a cross attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module and the cross attention module both have 8 attention heads, a query dimension of 256, and the hidden layer dimension of the feedforward neural network module is 2048. The bounding box coordinates of the target to be detected are predicted by the multilayer perceptron module of the detection head; other parameters are determined by the default parameters in mmdetection-2.23.

0. Step 2.3 Construction of the fusion detection head The fusion detection head uses the DETRHead architecture, which is an anchorless detection head based on multi-head attention. It consists of 8 encoding layers, 8 decoding layers, and a linear layer module. The 8 encoding layers, except for the first layer which is composed of a multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series, each consist of a multi-head attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module has 8 attention heads, the query dimension is 256, and the feedforward neural network... The hidden layer dimension of the network module is 2048; each decoding layer consists of a multi-head attention module, a neural layer regularization module, a cross attention module, a neural layer regularization module, a feedforward neural network module, and a neural layer regularization module connected in series. The multi-head attention module and the cross attention module each have 8 attention heads, the query dimension is 256, and the hidden layer dimension of the feedforward neural network module is 2048; the fusion detection head also includes other parameters, all of which are determined by the default parameters in mmdetection-2.23.0; the category information of the target to be detected is predicted by the linear layer module; Step 3: Model Training The model training uses a pre-built dataset to first train the feature extraction backbone network, then train the single-path detection head, and finally add a fusion detection head for fusion detection head training; the specific implementation uses the mmdetection deep learning framework. Step 3.1 Feature Extraction Backbone Network Training All training data collected by the main and secondary cameras are input into the single-branch object detection network for training. Then, the bounding box coordinates of the inferred target are input into the mean absolute error loss function to calculate the loss, and the target category confidence is input into the cross-entropy loss function to calculate the loss. The two losses are summed as the final loss. The model parameters are optimized using backpropagation and gradient descent. Finally, the weights corresponding to the feature extraction backbone network are obtained, and the training of the feature extraction backbone network is completed. Here, the single-branch object detection network refers to the feature extraction backbone network connected in series with a single detection head. Step 3.2 Training of single-channel detection heads corresponding to the main and secondary cameras First, the feature extraction backbone network trained in step 3.1 is connected in series with the single-channel detection head corresponding to the main camera. After freezing the parameters of the trained feature extraction backbone network using the mmdetection framework, the image data captured by the main camera is input into the model. The bounding box coordinates of the target obtained by inference are input into the mean absolute error loss function to calculate the loss. The parameters of the single-channel detection head are optimized using backpropagation and gradient descent to obtain the weights of the single-channel detection head corresponding to the main camera, thus completing the training of the main camera branch detection head. Then, the same operation is performed on the data and single-channel detection head corresponding to the secondary camera to obtain the weights of the single-channel detection head corresponding to the secondary camera, thus completing the training of the secondary camera branch detection head. Step 3.3 Training the fusion detection head First, the parameters of the trained feature extraction backbone network and single-path detection head are frozen using the mmdetection framework. The fusion detection head is then set up as shown in Figure 1. The features output from the encoding parts of the single-path detection heads corresponding to the main and secondary cameras are input into the fusion detection head. The features of the two branches are fused by performing a Hadamard product with the corresponding fusion weight matrix and adding the results. The fused feature map is then input into the decoding part, and the final linear layer outputs the target class confidence after fusion detection. The target category confidence predicted by the fusion detection head is input into the FocalLoss loss function, and the model parameters are optimized by backpropagation to obtain the weights corresponding to the fusion detection head, thus completing the training of the fusion detection head. Step 4: Reasoning and Recognition For the specified video information to be detected, the frame difference is set to 5, and key frames are extracted with the frame difference as the frame interval. The key frames corresponding to the main and secondary cameras are input into the feature extraction backbone network, and the detection head outputs the category confidence of the detected target. The single-channel detection head outputs the bounding box coordinates of the target in the scene corresponding to the main and secondary cameras. The specific implementation adopts the mmdetection deep learning framework. Step 4.1 Feature extraction, backbone network inference and recognition: The feature extraction backbone network receives key frames captured simultaneously by the main and secondary cameras. The key frames are RGB images with a height of 480 and a width of 640. The images are input into the feature extraction backbone network for feature extraction. The feature extraction backbone network consists of 12 interconnected basic modules. The calculation process of each basic module is defined by formulas (1) to (4). （1） （2） （3） （4） In the above formula This is the fixed-window attention module in the Swing Transformer architecture. For the sliding window attention module, For a multilayer perceptron module, LN stands for neural layer regularization module. Let be an integer representing the ordinal number of the basic module in the feature extraction backbone network, and ,when hour For the features output by the previous module, when hour, For dividing the image patch, for The output obtained after passing through a multilayer perceptron for The output obtained after passing through a multilayer perceptron; For the output of the fixed-window attention module, This is the output of the sliding window attention module; assuming the batch size is 32, the sizes of the three sets of features output by the final feature extraction backbone network are 32*192*80*60, 32*384*40*30 and 32*768*20*15, respectively. Step 4.2 Single-channel detection head inference and recognition: The single-channel detection head receives features output from the second, third, and fourth layers of the feature extraction backbone network, performs channel alignment on the features output from the feature extraction backbone network, and then uses them as input to the encoding and decoding part of the single-channel detection head. The feature map output by the encoding and decoding structure is used as input to the concatenated MLP, and the output of the MLP is the predicted bounding box coordinates of the target. This completes the inference and recognition of the single-channel detection head. The feature size output by the feature extraction backbone network is the same as that in step 4.

1. The input received by the single-path detection head encoding structure is a feature map of size 6300*32*256. After position encoding of this feature map, it is recorded as the query of multi-head attention. The key and value of multi-head attention are the feature map without position encoding. Step 4.3 Fusion detection head inference and recognition: The fusion detection head receives the features output from the encoding parts of two single-path detection heads as input. The size of both features is 6300*32*256. The features are input into the encoding part of the fusion detection head. The multi-head attention module of the first encoding layer of the encoding part is the fusion multi-head attention module. The calculation process of one attention head of this module is defined by formulas (5) to (14), and the calculation of the fusion multi-head attention module is completed by formula (15). Corresponding to the main camera branch, Corresponding secondary camera branch; This is the corresponding learnable weight matrix; That is, the main camera branch corresponds to Learnable weights of a matrix, The output of the encoding section of the single-channel detection head corresponding to the main camera. Then it is the result of the matrix product of the two, and the other letters appearing in formulas (5) to (10) have the same meaning; （5） （6） （7） （8） （9） （10） The results obtained from formulas (5) to (10) Matrix and learnable fusion weight matrix of the same order and Perform the Hadamard product, defined by formulas (11) to (13), to calculate the fused matrix. ;in , ;in and The value is obtained during the training of the fusion detection head; （11） （12） （13） The calculated The matrix serves as the input to the attention mechanism, and the attention feature map fused with multi-head attention is calculated using formulas (14) and (15). （14） （15） In the above formula =256, for The transpose of a matrix. Refers to the first The output of each attention head, Integer , This is the multi-head attention weight matrix, whose values ​​are obtained during the training of the fused detection heads. To integrate the outputs of the multi-head attention module, For tensor splicing operations; After the remaining modules of the first layer are input, they continue to be input into the remaining 7 encoding layers and 8 decoding layers. Then, the linear layer of the fusion detection head predicts the category information of the target to be detected, thus completing the reasoning and recognition process of the fusion detection head.

Images