A category-agnostic approach to automatic checkout product counting

Abstract

The present invention discloses a category-independent automatic checkout product counting method, comprising: using a counting divide-and-conquer strategy as a self-supervised method to generate a pre-trained model; utilizing an attention module and a domain adaptation module to address the fine-grained attributes and domain differences of retail product data; splicing the category feature map of the single product image and the feature map of the checkout image and inputting them into the counting module to locate and count the corresponding categories, while simultaneously performing an incremental learning experimental setting to verify the category independence of the method. The present invention fully utilizes point-level annotation information, counting based on the different heat map responses of single product images and checkout images of different categories. Simultaneously, a self-supervised method is used to initialize model parameters, fine-grained features are captured through the attention module, and a gradient reversal layer is applied for domain adaptation. A category-independent product counting method is proposed for the automatic checkout task, achieving good results in both conventional and incremental experimental settings.

Description

Technical Field

[0001] The present invention belongs to the field of automatic checkout of retail commodities, and in particular relates to a category-independent automatic checkout product counting method. Background Art

[0002] Automated checkout systems are increasingly being used in everyday life, such as automated vending machines and unmanned convenience stores. The essence of automated checkout is to accurately predict the presence and quantity of each product in any product mix. With automated checkout systems, customers simply place their selected items at the checkout counter, and the AI-based system identifies the item category and quantity and automatically processes the purchase. It then generates a shopping list, informing the customer of the quantity of each category and the required payment.

[0003] However, the automatic checkout task faces several challenges. One of the main challenges is the large scale of product categories. To address this issue, some existing methods [Wei XS, Cui Q, Yang L, et al.RPC:A large-scale retail product checkout dataset[J].arXiv preprint arXiv:1901.07249,2019.][LiC, Du D, Zhang L, et al.Data priming network for automatic check-out[C] / / Proceedings of the 27th ACM International Conference on Multimedia.2019:2152-2160.][Yang Y, Sheng L, Jiang X, et al.IncreACO:Incrementally Learned AutomaticCheck-out with Photorealistic Exemplar Augmentation[C] / / Proceedings of theIEEE / CVF Winter Conference on Applications of Computer Vision.2021:626-634.] use object detection methods with bounding box annotations to predict product categories and quantities. However, acquiring bounding box annotations is time-consuming and labor-intensive. Furthermore, the provided bounding boxes may not fully cover the target object being detected, especially in the presence of occlusions. Another challenge in the automated checkout task is the fine-grained nature of product categories. In automated checkout, multiple product categories may belong to the same meta-category. These products may be visually similar but differ in content. Third, unlike general object detection and recognition tasks, multi-category product counting must address domain gaps. Specifically, single-product images are captured under controlled conditions, while checkout images are captured on a checkout platform displaying a variety of products. Multi-category product counting algorithms must be able to adapt to the differences between the source and target domains. A final challenge in the automated checkout task is the continuous updating of products in realistic checkout scenarios. Therefore, it would be desirable to have a method that can continuously update existing models based on new product categories without having to retrain the model from scratch.

[0004] Although some existing methods [Wei Multimedia.2019:2152-2160.][Yang Y, Sheng L, Jiang X, et al. IncreACO: Incrementally Learned Automatic Check-out withPhotorealistic Exemplar Augmentation[C] / / Proceedings of the IEEE / CVF WinterConference on Applications of Computer Vision.2021:626-634.] Object detection methods with bounding box annotations are used to predict product categories and quantities, but these methods have the following shortcomings: 1) Bounding box annotations are expensive to obtain and may not fully cover the target objects to be detected; 2) The methods are not optimized for fine-grained attributes. Summary of the Invention

[0005] The object of the present invention is to provide a category-independent automatic checkout product counting method.

[0006] The technical solution for achieving the purpose of the present invention is: a category-independent automatic checkout product counting method, comprising the following steps:

[0007] Step 1: Use a counting divide-and-conquer strategy as a self-supervised method to generate a pre-trained model for initializing the counting module parameters;

[0008] Step 2: Use the attention module to capture fine-grained features through the location and detail information in the checkout image. Use the domain adaptation module to distinguish between the checkout image domain and the single product image domain. Use the gradient reversal layer to confuse the domain discriminator between the two domains.

[0009] In step 3, the category feature map of the single product image and the feature map of the checkout image are spliced ​​and input into the counting module to locate and count the corresponding categories. At the same time, an incremental learning experiment is set up to verify the category independence of the method.

[0010] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned category-independent automatic checkout product counting method is implemented.

[0011] A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the above-mentioned category-independent automatic checkout product counting method.

[0012] Compared with the existing technology, the present invention has the following significant advantages: (1) The present invention uses only point-level annotation information to locate and count each product in the checkout image, reducing the cost of obtaining annotation information. (2) The counting divide-and-conquer method is used as a self-supervised method to pre-train the model to initialize the counting module parameters, improving the accuracy of the network. (3) The network can achieve good results in both conventional experimental settings and incremental learning experimental settings, verifying the category independence of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of a category-independent automatic checkout product counting method of the present invention.

[0014] Figure 2 This is a visualization diagram of the process of a category-independent automatic checkout product counting method of the present invention.

[0015] Figure 3 This is a flowchart of an incremental learning experiment for a category-independent automatic checkout product counting method of the present invention. DETAILED DESCRIPTION

[0016] The present invention proposes a category-independent automatic checkout product counting method, comprising the following steps: inputting a checkout image into a self-supervised module constructed based on a divide-and-conquer approach, i.e., the sum of the counts of each part of the image should be equal to the global count sum, thereby generating a pre-trained model for initializing parameters of a subsequent counting module; extracting features from the single product image and the checkout image respectively, and broadcasting the feature vectors of the single product image selected according to a certain rule to the same size of the checkout image feature map after global average aggregation; simultaneously, the checkout image feature map is input into an attention module to enhance fine-grained features, and the module includes a layout attention submodule focusing on spatial information and a detail attention submodule focusing on channel information; the single product image feature map and the checkout image feature map are simultaneously input into a domain adaptation module, and domain discrimination is performed after passing through a gradient reversal layer to confuse the source domain and the target domain; simultaneously, the single product image feature map and the checkout image feature map are spliced ​​into a long feature map and input into a counting module, and counting is performed according to different response values ​​in the heat map to obtain the number of each type of instance in the checkout image. This paper makes full use of point-level annotation information and counts products based on the different heatmap responses of single product images and checkout images of different categories. It also uses a self-supervised method to initialize model parameters, captures fine-grained features through an attention module, and applies a gradient reversal layer for domain adaptation. A category-independent product counting method is proposed for the automatic checkout task, achieving good results in both conventional and incremental experimental settings.

[0017] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0018] Combine Figure 1 、 Figure 2 as well as Figure 3 , a category-independent automatic checkout product counting method, comprising the following steps:

[0019] Step 1: Use a counting divide-and-conquer strategy as a self-supervised method to generate a pre-trained model for initializing the counting module parameters;

[0020] The aforementioned self-supervised method uses a counting divide-and-conquer strategy to divide the checkout image into blocks and count them separately, with the sum of the local counts being the same as the global count. The self-supervised method is designed based on the technical divide-and-conquer strategy, evenly dividing the input checkout image into several blocks, counting each block and the complete checkout image separately, obtaining the local count of each block and the global count of the entire image, and optimizing the pre-trained model under the condition that the sum of the local counts is equal to the global count.

[0021] Step 2: The attention module uses location and detail information in the checkout image to capture fine-grained features. The domain adaptation module distinguishes between the checkout image domain and the product image domain. The gradient reversal layer confuses the domain discriminator between the two domains, addressing the fine-grained attributes and domain differences in retail product data.

[0022] For the i-th checkout image H and W represent the height and width of the image respectively. After the feature extraction module, the feature map can be obtained. π is the feature extraction network The parameters of this feature map are input into the attention module to obtain the enhanced feature map F′ c i :

[0023]

[0024] Att layout (F)=σ(conv([f GAP (F):f GMP (F)])),

[0025] Att detail (F)=σ(f MLP (f GAP (F))+f MLP (f GMP (F))).

[0026] Among them Att layout (·) and Att detail (·) represents the layout and detail attention submodules respectively, F is an arbitrary feature map, σ is the sigmoid function, conv(·) is a convolution operation with a kernel size of 7×7, [:] is a concatenation operation, and f GAP (·) and f GMP (·) are global average pooling and global maximum pooling operations, respectively, f MLP (·) is a multi-layer perceptron with one hidden layer.

[0027] In terms of single product images, N images of the same type of products are randomly sampled. s images, represented as K is the total number of product categories, r=1,…,N s , H and W represent the height and width of the image respectively. After extracting features, we get the feature vector Θ is the feature extraction network The parameters of the k-th category single product image are obtained after the broadcast operation. F′ s k The size of F′ c iStay consistent.

[0028] F′ c i and F′ s k The image is then fed into the domain adaptation module. It first passes through a gradient reversal layer, then the domain discriminator performs discrimination, ultimately confusing the checkout image domain with the product image domain, thus achieving domain adaptation.

[0029] In step 3, the category feature map of the single product image and the feature map of the checkout image are spliced ​​and input into the counting module to locate and count the corresponding categories. At the same time, an incremental learning experiment is set up to verify the category independence of the method.

[0030] The checkout image feature map F′ obtained in step 2 c i And single product image feature map F′ s k Enter the counting module to locate and count the k-th product in the checkout image.

[0031] F′ c i and F′ s k First, it is spliced ​​into a long feature map, and then sent to the hourglass network. According to the output heat map The k-th category products are located and counted by different thermal values:

[0032]

[0033] where Hg(·) is an hourglass network.

[0034] The experimental setting of incremental learning is as follows: randomly select some categories from the category as unknown categories, first train the complete network model with known category data, and then use the model to test the test data set containing all known categories and unknown categories and the test data set containing only known categories, and compare the results of the two, such as Figure 3 shown.

[0035] The effect of the present invention can be further illustrated by the following simulation experiments:

[0036] Simulation conditions

[0037] The simulation experiments use a retail product checkout dataset [Wei XS, Cui Q, Yang L, et al. RPC: A large-scale retail product checkout dataset [J]. arXiv preprint arXiv:1901.07249, 2019.]. This dataset contains 53,739 individual product images with a resolution of 2592 × 1944 and 30,000 checkout images (6,000 for validation and 24,000 for testing) with a resolution of 1800 × 1800. There are 200 product classes, each belonging to 17 major categories.

[0038] The simulation experiments were all completed in Python under the Linux operating system. The image resolution for both training and testing was set to 512×512, the batch size was set to 4, the SGD optimizer was used for training, the momentum was 0.9, the weight decay was 0.0001, the learning rate was initialized to 0.01, and the learning rate was increased to 0.1 times the original value at the 20th round. The total number of rounds was 80, and N C and N s They are 6000 and 8 respectively.

[0039] The present invention adopts four evaluation indicators, namely: check-out accuracy (cAcc), average counting distance (ACD), mean category counting distance (mCCD), and mean category intersection of union (mCIoU).

[0040] Simulation content

[0041] The present invention uses the retail commodity checkout image dataset RPC to verify the automatic checkout performance of the algorithm. To test the performance of the proposed algorithm, the proposed class-independent automatic checkout product counting method (self-supervised multi-class counting, S2MC2) was compared with RPC[Wei XS, Cui Q, Yang L, et al.RPC:Alarge-scale retail product checkout dataset[J].arXiv preprint arXiv:1901.07249,2019.], DPNet[Li C, Du D, Zhang L, et al.Data priming network for automatic check-out[C] / / Proceedings of the 27th ACM International Conference on Multimedia.2019:2152-2160.], and IncreACO[Yang Y, Sheng L, Jiang X, et al.IncreACO:Incrementally Learned Automatic Check-out with Photorealistic Exemplar Augmentation[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision.2021:626-634.] for comparison.

[0042] In addition, an incremental setting experiment was conducted. A small class from the 17 major classes was randomly selected as the unknown class, and the other 183 classes were used as known classes. The network of this method was trained with data from the known classes, and then tested with both known classes and complete data respectively, and compared with the previous method IncreACO that included incremental experiments.

[0043] Analysis of simulation experiment results

[0044] Table 1 compares the evaluation metrics of different automated checkout algorithms on the RPC dataset. Experimental results demonstrate that our algorithm outperforms other automated checkout algorithms, accurately locating and counting products in checkout images. Its cAcc performance is 1.61% higher than that of the previously superior DPNet algorithm.

[0045] Table 1 Comparison of automatic checkout evaluation indicators of RPC dataset

[0046]

[0047] Table 2 shows the comparison results of the evaluation indicators of the RPC dataset under the incremental setting experiment of different automatic checkout algorithms. The experimental data of other methods are all from IncreACO. (183+17) means incremental training using the LwF [Li Z, Hoiem D. Learning without forgetting [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40 (12): 2935-2947.] method. The experimental results show that the algorithm of the present invention is better than the IncreACO algorithm and does not show obvious catastrophic forgetting. The cAcc of the 183-class training tested on the 200-class dataset is only 2.23% lower than the cAcc tested on the 183-class dataset, while the results of IncreACO are 31.23% and 4.46% respectively.

[0048] Table 2 Comparison of evaluation indicators of the automatic checkout incremental setting experiment of the RPC dataset

[0049]

Claims

1. A category-independent automatic checkout product counting method, characterized in that: The following steps are involved: Step 1: Use a counting divide-and-conquer strategy as a self-supervised method to generate a pre-trained model for initializing the counting module parameters; Step 2: Use the attention module to capture fine-grained features through the location and detail information in the checkout image. Use the domain adaptation module to distinguish between the checkout image domain and the single product image domain. Use the gradient reversal layer to confuse the domain discriminator between the two domains. For the i-th checkout image H and W represent the height and width of the image respectively. After the feature extraction module, the feature map can be obtained. π is the feature extraction network The parameters of the feature map are input into the attention module to obtain the enhanced feature map Att layout (F)=σ(conv([f GAP (F):f GMP (F)])), Att detail (F)=σ(f MLP (f GAP (F))+f MLP (f GMP (F))). Among them Att layout (·) and Att detail (·) represents the layout and detail attention submodules respectively, F is an arbitrary feature map, σ is the sigmoid function, conv(·) is a convolution operation with a kernel size of 7×7, [:] is a concatenation operation, and f GAP (·) and f GMP (·) are the global average merging and global maximum merging operations, respectively, and f MLP (·) is a multilayer perceptron with one hidden layer; In terms of single product images, N images of the same type of products are randomly sampled. s images, represented as K is the total number of product categories, r=1,…,N s , H and W represent the height and width of the image respectively; After extracting features, we get the feature vector Θ is the feature extraction network The parameters of the k-th category single product image are obtained after the broadcast operation. The size and Stay consistent; Will and The data is then fed into the domain adaptation module. It first passes through a gradient reversal layer, and then the domain discriminator performs discrimination. Finally, the checkout image domain and the single product image domain are confused, thus achieving domain adaptation. In step 3, the product image feature map and the checkout image feature map are concatenated and fed into the counting module to locate and count the corresponding categories. At the same time, an incremental learning experiment is performed to verify the category independence of the method. Specifically: The checkout image feature map obtained in step 2 and single product image feature map Input counting module to locate and count the k-th product in the checkout image; and First, it is spliced ​​into a long feature map, and then sent to the hourglass network. According to the output heat map The k-th category products are located and counted by different thermal values: where Hg(·) is the hourglass network; The experimental setting of incremental learning is as follows: some categories are randomly selected from the categories as unknown categories, and the complete network model is first trained with the known category data. Then, the model is used to test the test data set containing all known categories and unknown categories and the test data set containing only known categories, and the results of the two are compared.

2. The category-independent automatic checkout product counting method according to claim 1, characterized in that: In step 1, the count divide-and-conquer strategy is used as a self-supervised method. The checkout image is divided into blocks and counted separately. The sum of the local counts is equal to the global count. The self-supervised method is designed based on the technical divide-and-conquer strategy. The input checkout image is evenly divided into several blocks. Each block and the complete checkout image are counted separately to obtain the local count of each block and the global count of the entire image. The pre-trained model is optimized under the condition that the sum of the local counts is equal to the global count.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the category-independent automatic checkout product counting method according to any one of claims 1-2 is implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the category-independent automatic checkout product counting method according to any one of claims 1 to 2 is implemented.

Images