Prototype-based classifier learning from single product examples

Abstract

The present invention discloses a prototype-based classifier learning method from a single product example, comprising the following steps: generating a prototype-based classifier from a single product example, including a background classifier; using the score of the background class to perform discriminative rearrangement; before generating a product candidate frame, performing multi-label learning, modeling the co-occurrence relationship of various types of products in the checkout image, and obtaining the corresponding loss to constrain the network. The present invention utilizes a prototype-based classifier to alleviate the domain difference between the single product example and the checkout image, develops a discriminative reordering module to enhance the discriminative ability of the classifier, and uses a multi-label recognition loss to model the co-occurrence relationship of products in the checkout image. The present invention can be widely used in the fields of automatic checkout, target detection, etc.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent retail automatic checkout, and in particular relates to a prototype-based classifier learning method from single product examples. Background Art

[0002] As an important area of ​​smart retail, automatic checkout has been continuously developing and popularizing in recent years. The basic design requirement of the automatic checkout system is to automatically identify and count the products purchased by customers. Therefore, how to accurately calculate the fine-grained subordinate categories and quantities of products in the checkout image becomes a key task of automatic checkout.

[0003] There are three main challenges in automatic checkout, namely, large-scale data of product images, domain gap between training and testing data, and fine-grained features of product categories. Specifically, single product images are taken under controlled conditions, while checkout images are taken on a checkout platform where various products are placed. Multi-category product counting algorithms must be able to adapt to the differences between the source and target domains. In the automatic checkout task, there may be multiple product categories belonging to the same meta-category. These products may be very similar in appearance, but their contents are not exactly the same. To deal with these problems, existing methods provide a baseline method based on object detection to achieve domain adaptation by synthesizing and rendering checkout images.

[0004] Although some existing automatic checkout methods [Wei Multimedia.2019:2152-2160.][Yang Y, Sheng L, Jiang X, et al. IncreACO: Incrementally LearnedAutomatic Check-out with Photorealistic Exemplar Augmentation[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision.2021:626-634.] Use object detection for recognition and counting, and perform domain adaptation by synthesizing and rendering images, but these methods have the following shortcomings: 1) The training time and storage costs are high, and a large amount of image data needs to be synthesized and rendered; 2) The accuracy is not high enough to be applied to actual scenarios. Summary of the invention

[0005] The object of the present invention is to provide a prototype-based classifier learning method from single product examples.

[0006] The technical solution to achieve the purpose of the present invention is: a prototype-based classifier learning method from a single product example, comprising the following steps:

[0007] Step 1: Generate prototype-based classifiers from single product examples, including background classifiers;

[0008] Step 2, use the background class scores for discriminative rearrangement;

[0009] Step 3: Before generating product candidate boxes, multi-label learning is performed to model the co-occurrence relationship of various products in the checkout images and obtain the corresponding loss to constrain the network.

[0010] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned prototype-based classifier learning method from a single product example is implemented.

[0011] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned prototype-based classifier learning method from single product examples.

[0012] Compared with the prior art, the present invention has the following significant advantages: (1) the present invention directly learns a prototype-based classifier from single product examples to complete the automatic checkout task, especially to solve the domain difference problem; (2) the discriminative ability of the prototype-based classifier is enhanced by performing a discriminative rearrangement of background class scores taking into account the fine-grained attributes of the product; and (3) multi-label learning is simultaneously used to model the product co-occurrence relationship in the checkout image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of the prototype-based classifier learning method from single product examples of the present invention.

[0014] Figure 2 This is a visualization diagram of the process of the prototype-based classifier learning method from a single product example of the present invention.

[0015] Figure 3 This is a schematic diagram of the rearrangement results of the prototype-based classifier learning method from single product examples in the present invention. DETAILED DESCRIPTION

[0016] The present invention proposes a prototype-based classifier learning method from a single product example, comprising the following steps: the prototype representation of each product category is first obtained from the single product example; based on the prototype, a prototype classifier containing a background classifier is generated, which can not only identify fine-grained product categories, but also distinguish product background candidate boxes from checkout images; in order to further improve the accuracy, a discriminative re-ranking is developed to adjust the prediction scores of product candidate boxes, bring stronger discriminative ability in classifier learning, and provide a reasonable sorting possibility by considering the fine-grained nature; in addition, a multi-label recognition loss is also equipped to model the co-occurrence relationship of products in checkout images. The present invention uses a prototype-based classifier to alleviate the domain difference between single product examples and checkout images, develops a discriminative re-ranking module to enhance the discriminative ability of the classifier, and uses a multi-label recognition loss to model the co-occurrence relationship of products in checkout images. The present invention can be widely used in fields such as automatic checkout and object detection.

[0017] Combination Figure 1 , Figure 2 as well as Figure 3 , a prototype-based classifier learning method from a single product example, specifically comprising the following steps:

[0018] Step 1: Generate a prototype-based classifier from a single product example, including a background classifier.

[0019] Collecting single product examples In the prototype-based classifier generator, H and W represent the height and width of the image, respectively. First, the feature vectors of all examples of the same class are averaged to obtain the prototype of the class. k is the corresponding category, is the total number of single product examples in the kth class, Θ is the feature extraction network f CNN (; Θ) parameters, and then average all class prototypes to get the background class prototype K is the total number of product categories, and all prototypes undergo a nonlinear mapping f generator (·) and we get K+1 classifiers, denoted as W = [w 1 ;…;w K ;w BG ], K is the total number of product categories; f generator (·) In the implementation, it is a multi-layer perceptron consisting of 2 fully connected layers and a ReLU activation function.

[0020] Step 2: Use the background class scores to perform discriminative re-ranking to improve the discriminative ability of each prototype-based classifier and provide a reasonable ranking possibility for the fine-grained properties of the products;

[0021] For checkout image I c The nth product candidate box, n = 1, ..., N, N is the total number of product candidate boxes generated, let m n Represents its feature vector, then its score for K+1 classes can be represented as vector s n :

[0022]

[0023] Considering the fine-grained nature of product categories, the appearance differences between some categories are very small, and a reasonable ranking possibility is (the true category is not the background class):

[0024]

[0025]

[0026]

[0027] where y nis the true category of the nth product candidate box, Ω=[1,…,K,BG], ∈ is a slack variable, α is a hyperparameter with a range of (0,1). n =BG, the score ranking does not change.

[0028] The present invention improves the discrimination ability of the prototype-based classifier by re-ranking the background class scores, and at the same time provides a reasonable ranking possibility for the fine-grained properties of the products.

[0029] Step 3: Before generating product candidate boxes, multi-label learning is performed to model the co-occurrence relationship of various products in the checkout images and obtain the corresponding loss to constrain the network.

[0030] For checkout image I c ∈R H×W , let the feature map of the last layer of the backbone network be F, and use a fully connected layer f FC (·; φ) for multi-label recognition, which can be expressed as:

[0031]

[0032] in is the multi-label recognition result, f GAP (·) and f GMP (·) are the global average pooling and global maximum pooling operations, respectively, and φ is the parameter of the fully connected layer.

[0033] The effect of the present invention can be further illustrated by the following simulation experiment:

[0034] Simulation conditions

[0035] The simulation experiment uses a retail product checkout image dataset (retail product checkout dataset) [Wei XS, Cui Q, Yang L, et al. RPC: A large-scale retail product checkout dataset [J]. arXiv preprint arXiv: 1901.07249, 2019.]. The dataset contains 53,739 single product images with a resolution of 2592 × 1944 and 30,000 checkout images (6,000 validation sets and 24,000 test sets) with a resolution of 1800 × 1800. There are 200 product categories, belonging to 17 major categories.

[0036] The simulation experiments were all completed in Python under the Linux operating system. The image resolution during training and testing was set to 800×800, the batch size was set to 2 and 1 respectively, the SGD optimizer was used during training, the momentum was 0.9, the weight decay was 0.0001, the learning rate was initialized to 0.0025, and the learning rate was changed to 0.1 times the original in the 24th and 33rd rounds, the total number of rounds was 36, and α was 0.97.

[0037] The present invention adopts six evaluation indicators, namely: check-out accuracy (cAcc), average counting distance (Average counting distance, ACD), mean category counting distance (Mean category counting distance, mCCD), mean category intersection of union (Mean category intersection of union, mCIoU), mean category PR (Precision-Recall) area under the curve (intersection-of-union threshold is 0.50) (mAP50), and mean intersection-of-union average category PR curve area (mmAP).

[0038] Simulation content

[0039] The present invention uses the retail commodity checkout image dataset RPC to verify the automatic checkout performance of the algorithm. To test the performance of the algorithm of the present invention, the proposed prototype-based classifier learning method from Single-Product exemplars (PSP) is compared with RPC[Wei XS, Cui Q, Yang L, et al.RPC:A large-scale retail product checkout dataset[J].arXivpreprint arXiv:1901.07249,2019.], DPNet[Li C, Du D, Zhang L, et al.Data priming network for automatic check-out[C] / / Proceedings of the 27th ACM International Conference on Multimedia.2019:2152-2160.], and IncreACO[Yang Y, Sheng L, Jiang X, et al.IncreACO:Incrementally Learned Automatic Check-out with PhotorealisticExemplar Augmentation[C] / / Proceedings of the IEEE / CVF Winter Conference onApplications of Computer Vision.2021:626-634.] for comparison.

[0040] Analysis of simulation experiment results

[0041] Table 1 shows the comparison results of evaluation indicators of RPC dataset under different automatic checkout algorithms. The experimental results show that the Faster RCNN+FPN target detection model in the algorithm application of the present invention is superior to other automatic checkout algorithms, and can accurately locate and count products in the checkout image. All indicators are comparable to the previous results, especially the main evaluation indicator cAcc is 5.89% higher than the previous better DPNet result.

[0042] Table 1 Comparison of automatic checkout evaluation indicators of RPC dataset

[0043]

Claims

1. A prototype-based classifier learning method from a single product example, characterized in that: The following steps are involved: Step 1: Generate prototype-based classifiers from single product examples, including background classifiers, as follows: In the prototype-based classifier generator, H and W represent the height and width of the image, respectively. First, the feature vectors of all examples of the same class are averaged to obtain the prototype of the class. k is the corresponding category, is the total number of single product examples in the kth class, Θ is the feature extraction network f CNN (; Θ) parameters, and then average all class prototypes to get the background class prototype K is the total number of product categories, and all prototypes undergo a nonlinear mapping f generator (·) and we get K+1 classifiers, denoted as W = [w1; ...; w K ;w BG ]; f generator (·) In the implementation, it is a multi-layer perceptron with 2 fully connected layers and a ReLU activation function; Step 2, use the background class scores for discriminative rearrangement; For checkout image I c ∈R H×W The nth product candidate box, n = 1, ..., N, N is the total number of product candidate boxes generated, let m n Represents its feature vector, then its score for K+1 classes can be represented as vector s n : A reasonable order is: where y n is the true category of the nth product candidate box, Ω = [1, ..., K, BG], ∈ is a slack variable, α is a hyperparameter with a range of (0, 1); when y n =BG, the score ranking will not be changed; Step 3: Before generating product candidate boxes, multi-label learning is performed to model the co-occurrence relationship of various products in the checkout images, and the corresponding loss is obtained to constrain the network; For checkout image I c , let the feature map of the last layer of the backbone network be F, and use a fully connected layer f FC (·; φ) for multi-label recognition, which can be expressed as: in is the multi-label recognition result, f GAP (·) and f GMP (·) are the global average pooling and global maximum pooling operations, respectively, and φ is the parameter of the fully connected layer.

2. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the prototype-based classifier learning method from single product examples as described in claim 1 is implemented.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the prototype-based classifier learning method from single product examples as claimed in claim 1 is implemented.

Images