A commodity recognition method based on image classification

Abstract

The application discloses a kind of commodity identification methods based on image classification, it is related to computer vision and commodity intelligent identification technical field, the method extracts visual token sequence to commodity image, constructs adjacent graph according to spatial position to commodity packaging text and obtains text node feature sequence using graph attention network coding, with visual token sequence as query, with text node feature sequence as key and value cross-modal fusion is carried out, obtains fusion feature vector, with the vector database of at least one appearance sub-prototype stored under each grouping is matched to the grouping according to commodity SKU identity, according to adaptive mahalanobis distance, judges commodity identity, automatically routes to active learning pool to unknown commodity and back to vector database, the method avoids long text truncation to cause distinguishing information loss, realizes commodity identity and appearance decoupling representation, and supports the incremental update of free retraining, and can be used for commodity identity identification in the scene of wisdom retail.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent product recognition technology, specifically to a product recognition method based on image classification. Background Technology

[0002] Image classification technology is widely used in the field of product recognition. It typically uses convolutional neural networks or visual Transformers to extract feature vectors from product images, and then outputs the probability distribution of the product's category through a classifier. To improve recognition accuracy, existing technologies further introduce optical character recognition technology to extract text content on product packaging, and use a multimodal image and text pre-trained model to map image features and text features to the same semantic space to achieve joint utilization of image and text information.

[0003] The closest prior art to this invention is a product classification method based on a multimodal image-text pre-trained model. This method uses an image encoder to extract global visual feature vectors of product images, uses an optical character recognition engine to extract all the text on the product packaging and concatenates them into a continuous string in sequence. The string is then input into a text encoder with a fixed upper limit of input length to obtain text feature vectors. The visual feature vectors and text feature vectors are then fused and input into a classification layer with a fixed output dimension. The category with the highest probability is taken as the recognition result.

[0004] The existing technologies have the following drawbacks: First, the input sequence length of the text encoder has a fixed upper limit. When the total amount of text on the product packaging exceeds this limit, the excess part is directly truncated and discarded, resulting in the loss of the distinguishing text information at the end of the product packaging, which is crucial for different specifications and flavors of the same brand, leading to fine-grained misjudgments. Second, the method of processing text into continuous strings does not preserve the spatial arrangement of each text box on the packaging surface and ignores the relative positional clues between text and images. Third, the number of output categories of the classification layer is fixed after the network structure is determined. When new categories are added to the product catalog, the classifier must be retrained. Fourth, the same product identity usually corresponds to only a single appearance feature distribution. After the product packaging is redesigned, it is easy to be misjudged as an unknown new product. Therefore, we propose a product recognition method based on image classification. Summary of the Invention

[0005] The purpose of this invention is to provide a product recognition method based on image classification.

[0006] The technical problem this invention aims to solve is that existing multimodal product classification methods are limited by fixed input length during text encoding, resulting in the truncation of long text on product packaging and the loss of key distinguishing information, leading to fine-grained product misjudgment. At the same time, existing methods fail to decouple product identity from appearance, making it easy to misjudge unknown new products after product packaging is redesigned. Furthermore, existing closed-set classifiers require full retraining when adding new categories to the product catalog, and lack a reliable rejection and automatic iteration mechanism for products that are not actually in the database.

[0007] To address the aforementioned technical problems, this invention provides a product recognition method based on image classification, comprising: acquiring a product image to be recognized; extracting visual features from the product image to obtain a visual token sequence; performing text detection and recognition on the product image to obtain multiple text boxes and the text content and position coordinates corresponding to each text box; constructing a spatial adjacency graph based on the position coordinates of each text box; encoding the spatial adjacency graph using a graph attention network to obtain a text node feature sequence; using the visual token sequence as a query vector and the text node feature sequence as a key vector and value vector, performing cross-modal cross-attention calculation to obtain a fused feature vector; and then... The feature vector is compared with the prototypes stored in the vector database. The vector database is grouped by product SKU identity, and each group stores at least one appearance sub-prototype. The prototype with the closest Mahalanobis distance to the fused feature vector is selected as the candidate prototype. It is determined whether the Mahalanobis distance between the fused feature vector and the candidate prototype exceeds the adaptive rejection boundary corresponding to the group to which the candidate prototype belongs. If it does not exceed the adaptive rejection boundary, the product identity recognition result corresponding to the group to which the candidate prototype belongs is output. If it exceeds the adaptive rejection boundary, the product image to be identified is determined to be an unknown product, and the fused feature vector is stored in the active learning pool.

[0008] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0009] 1. This invention organizes the text on product packaging into a graph structure according to spatial position and uses a graph attention network for encoding, avoiding the truncation problem caused by the fixed input length of traditional text encoders, realizing the lossless utilization of long text and its layout information, and improving the ability to distinguish fine-grained products;

[0010] 2. This invention achieves decoupled representation of product identity and appearance by using a dynamic prototype library structure that uses product SKU identity as the grouping unit and supports at least one appearance sub-prototype under each group. This allows product packaging to be correctly classified to the original SKU identity even after a redesign.

[0011] 3. This invention uses a vector database in conjunction with an incremental clustering mechanism to prevent the addition of new products from requiring retraining of the entire network, thus avoiding the catastrophic forgetting problem of closed-set classifiers. Through adaptive Mahalanobis distance rejection boundaries calculated separately for each group, coupled with a closed-loop mechanism that automatically routes unknown products to the active learning pool and feeds back into the vector database, the system can achieve continuous self-iteration of the product catalog without retraining the network. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall product identification main process in an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of the text graph attention encoding branch process in an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram of the dynamic updating of the vector library and the active learning closed-loop process in an embodiment of the present invention. Detailed Implementation

[0015] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0016] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0017] To make the technical solution and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. In this embodiment, SKU (Stock Keeping Unit) is used to uniquely identify the identity number of a certain product. The same SKU may correspond to multiple packaging appearances of the product at different times.

[0018] The product recognition method provided in this embodiment includes three parts: lossless fusion of image and text multimodal methods, dynamic multi-prototype representation decoupling identity and appearance, and open set recognition and active learning closed loop based on adaptive Mahalanobis distance. The three parts work together to form a complete technical solution.

[0019] I. Lossless Fusion of Image and Text Multimodal Modes:

[0020] After acquiring the product image to be identified, a lightweight visual Transformer network is used to extract features from the image and output a set of visual token sequences carrying local and global information of the image.

[0021] For the same product image, an optical character recognition engine is used to detect all text boxes on the product packaging, as well as the text content and bounding rectangle coordinates of each text box. All text boxes are treated as a set of nodes in a graph structure. For each node in the set, the coordinates of the center point of its bounding rectangle are taken as the spatial position of the node. The Euclidean distance between any two node spatial positions is calculated. For each node, several other nodes with the closest spatial distance are selected as its neighbor nodes. Edge connections between nodes are established to form a spatial adjacency graph. The text content corresponding to each node is encoded using a lightweight text encoder to obtain the initial feature vector of the node.

[0022] The initial feature vectors of all nodes and the constructed spatial adjacency graph are input into a two-layer graph attention network. In each layer, the attention coefficient between each node and each of its neighboring nodes is calculated. The calculation of the attention coefficient takes into account the node's own features, the features of its neighboring nodes, and the relative spatial position bias between the two nodes. Then, the features of the neighboring nodes are weighted and aggregated according to the attention coefficient to obtain the updated feature vector of the node. This update process is repeated layer by layer to finally obtain a text node feature sequence that incorporates the spatial context information of the packaging layout.

[0023] The formula for calculating the attention coefficient is as follows:

[0024] ;

[0025] in, Represents a node The initial feature vector, This represents the learnable linear transformation matrix in a graph attention network. Represents a learnable attention weight vector, with the symbol... This represents a vector concatenation operation. Represents a node With nodes The spatial position offset code between the two text boxes is calculated based on the difference in the center coordinates of the two text boxes. Represents a node The set of neighboring nodes, Represents a node For nodes The normalized attention coefficient;

[0026] The node feature update formula is as follows:

[0027] ;

[0028] in, Represents a node The updated feature vector obtained after graph attention aggregation in this layer Represents a nonlinear activation function;

[0029] After obtaining the text node feature sequence, the visual token sequence is used as the query matrix, and the text node feature sequence is used as the key matrix and value matrix respectively. The similarity score between the query matrix and the key matrix is ​​calculated, and the score is scaled and normalized to obtain the attention weight matrix. The attention weight matrix is ​​multiplied by the value matrix to obtain the visual enhancement feature guided by the text information. The enhancement feature is added to the original visual feature and then normalized to obtain the fused feature vector.

[0030] The formula for calculating cross-modal attention is as follows:

[0031] ;

[0032] in, This represents a query matrix consisting of sequences of visual tokens. This represents a key matrix composed of text node feature sequences. This represents a value matrix composed of text node feature sequences. This represents the dimension of the feature vector, used to scale the similarity score;

[0033] The formula for outputting the fused features is as follows:

[0034] ;

[0035] in, Represents original visual features. This represents the obtained fused feature vector. Presentation layer normalization processing.

[0036] II. Dynamic multi-prototype representation decoupled from identity and appearance:

[0037] During the training phase, the metric learning loss function is used to optimize the above fusion network so that samples of the same SKU are close to each other in the feature space and samples of different SKUs are separated from each other. During the initialization phase, the average value of the fusion features of all samples of each SKU in the training set is taken, and the average vector is used as the initial appearance prototype of the SKU and stored in the vector database with the SKU ID number as the label.

[0038] During the system operation phase, the fused features of newly identified samples are continuously accumulated and added to the historical sample set of the corresponding SKU, and the intra-class feature variance of the historical sample set of that SKU is periodically calculated. The intra-class feature variance calculation formula is as follows:

[0039] ;

[0040] in, Indicates the first Within-class feature variance of each SKU This indicates the current cumulative number of historical samples for this SKU. This indicates the first SKU under this SKU The fused feature vector of each sample, This represents the mean vector of the current prototype of the SKU;

[0041] When the intra-class feature variance of a certain SKU exceeds a preset threshold, it is determined that there are significantly different appearance distributions under that SKU. The K-means clustering algorithm is used to cluster all historical sample features of the SKU. The number of clusters is determined according to the silhouette coefficient. The center vector of each cluster is used as the appearance sub-prototype of the SKU. They are all written into the vector database with the same SKU identity number as the label, replacing the original single prototype. In the subsequent inference stage, all appearance sub-prototypes under the same SKU participate in the nearest neighbor search. As long as the test sample hits any sub-prototype under the SKU and meets the distance condition, it is determined to be the identity of the SKU. This decouples the product identity from the appearance.

[0042] III. Open Set Rejection and Active Learning Closed Loop Based on Adaptive Mahalanobis Distance:

[0043] During the training phase, the mean vector and covariance matrix of the fusion features are calculated for each SKU or its corresponding appearance sub-prototype sample cluster, and the Mahalanobis distance distribution of all samples of the SKU in the training set to its corresponding prototype is calculated. The 95th percentile of this distribution is taken as the adaptive rejection boundary of the SKU.

[0044] During the inference phase, after extracting and fusing features from the image of the product to be identified, the Mahalanobis distance between these features and all prototypes and appearance sub-prototypes in the vector database is calculated. The prototype with the closest distance is selected as the candidate identity. The pre-determined adaptive rejection boundary of the SKU to which the candidate prototype belongs is read. The Mahalanobis distance calculation formula is as follows:

[0045] ;

[0046] in, Indicates the feature to be measured To the The Mahalanobis distance of the prototype Indicates the first The mean vector of each prototype corresponds to a sample set. Indicates the first The covariance matrix of each prototype corresponding to a sample set This represents the inverse matrix of the covariance matrix;

[0047] If the Mahalanobis distance between the test sample and the candidate prototype does not exceed the rejection boundary, the test sample is determined to belong to the SKU identity corresponding to the candidate prototype, and the recognition result is output. If it exceeds the rejection boundary, the test sample is determined to be an unknown product and is automatically sent to the active learning pool. The samples in the active learning pool are used to generate new SKU prototypes or to split and update the appearance sub-prototypes of existing SKUs after verification processing. Verification processing includes manual verification processing and automatic incremental clustering processing. The update results are written to the vector database for subsequent reasoning and retrieval, thus forming a continuous self-iterative closed loop of the recognition system.

[0048] IV. Other implementation methods:

[0049] In other implementations, the text branch encoding method can also adopt a hierarchical text encoding structure that supports long sequence input. That is, the entire OCR text is first divided into several text segments according to its spatial order on the packaged image, each text segment is encoded, and then the encoding results of each segment are aggregated through pooling or a lightweight sequence model to obtain a text feature representation that can cover the entire text content without truncation.

[0050] In other implementations, the open set rejection step can also use a similarity score based on energy score or temperature scaling as the criterion, perform statistical distribution modeling on the similarity score between the test sample and the nearest neighbor prototype, and use the quantile of the distribution as the rejection boundary.

[0051] In other implementations, the dynamic prototype incremental clustering step can also use a sample density-based clustering algorithm to automatically determine the number of appearance sub-prototypes for historical samples of the same SKU, without having to pre-set the number of clusters.

[0052] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A product recognition method based on image classification, characterized in that, include: Obtain an image of the product to be identified, extract visual features from the product image, and obtain a visual token sequence; Text detection and recognition are performed on the product image to obtain multiple text boxes and the text content and position coordinates of each text box. A spatial adjacency graph is constructed based on the position coordinates of each text box, and a graph attention network is used to encode the spatial adjacency graph to obtain a text node feature sequence. Using the visual token sequence as the query vector and the text node feature sequence as the key vector and value vector, cross-modal cross-attention calculation is performed to obtain the fused feature vector. The fused feature vector is compared with the prototypes stored in the vector database. The vector database is grouped by product SKU, and each group stores at least one appearance sub-prototype. The prototype with the closest Mahalanobis distance to the fused feature vector is selected as the candidate prototype. Determine whether the Mahalanobis distance between the fused feature vector and the candidate prototype exceeds the adaptive rejection boundary corresponding to the group to which the candidate prototype belongs; If the adaptive rejection boundary is not exceeded, the product identity recognition result corresponding to the group to which the candidate prototype belongs is output; If the adaptive rejection boundary is exceeded, the product image to be identified is determined to be an unknown product, and the fused feature vector is stored in the active learning pool.

2. The product recognition method based on image classification according to claim 1, characterized in that: In the step of encoding the spatial adjacency graph, when calculating the attention coefficient between a node and its neighboring nodes, the characteristics of the current node itself, the characteristics of the neighboring nodes, and the relative spatial position offset encoding between the two nodes are considered simultaneously.

3. The product recognition method based on image classification according to claim 1, characterized in that, The cross-modal attention calculation includes: calculating the similarity score between the query vector and the key vector; scaling and normalizing the similarity score to obtain the attention weight; multiplying the attention weight by the value vector to obtain the visual enhancement feature; adding the visual enhancement feature to the original visual feature corresponding to the visual token sequence and then normalizing the result to obtain the fused feature vector.

4. The product recognition method based on image classification according to claim 1, characterized in that, The appearance sub-prototype corresponding to each group in the vector database is dynamically generated in the following way: continuously collect the fusion features of historical samples belonging to the same group, periodically calculate the intra-class feature variance of the historical samples of the group, and when the intra-class feature variance exceeds a preset threshold, perform clustering processing on the fusion features of all historical samples of the group, and store the cluster center vectors obtained by clustering as the appearance sub-prototype of the group in the vector database.

5. The product recognition method based on image classification according to claim 4, characterized in that: When performing clustering on the fusion features of all historical samples in this group, the K-means clustering algorithm is used, and the number of clusters is determined based on the silhouette coefficient.

6. The product recognition method based on image classification according to claim 1, characterized in that, The adaptive rejection boundary for each group is determined as follows: the Mahalanobis distance distribution from the fusion features of the training samples within the group to the corresponding prototype of the group is statistically analyzed, and the 95th percentile of the distribution is taken as the adaptive rejection boundary for the group.

7. The product recognition method based on image classification according to claim 1, characterized in that: The fusion features stored in the active learning pool are used to update the vector database after verification processing. The verification processing includes manual verification processing and automatic incremental clustering processing. The updated vector database includes newly added product identity groups and updated appearance sub-prototypes.

8. A product recognition device based on image classification, characterized in that, include: The visual feature extraction module is used to acquire an image of the product to be identified, extract visual features from the product image, and obtain a visual token sequence. The text graph encoding module is used to perform text detection and recognition on the product image, obtain multiple text boxes and the text content and position coordinates of each text box, construct a spatial adjacency graph based on the position coordinates of each text box, and encode the spatial adjacency graph using a graph attention network to obtain a text node feature sequence. The cross-modal fusion module is used to perform cross-modal cross-attention calculation using the visual token sequence as the query vector and the text node feature sequence as the key vector and value vector to obtain the fused feature vector; The prototype retrieval module is used to compare the fused feature vector with the prototypes stored in the vector database. The vector database is grouped by product SKU, and each group stores at least one appearance sub-prototype. The prototype with the closest Mahalanobis distance to the fused feature vector is selected as the candidate prototype. The rejection judgment module is used to determine whether the Mahalanobis distance between the fused feature vector and the candidate prototype exceeds the adaptive rejection boundary corresponding to the group to which the candidate prototype belongs. If it does not exceed the adaptive rejection boundary, the module outputs the product identity recognition result corresponding to the group to which the candidate prototype belongs. If it exceeds the adaptive rejection boundary, the module determines that the product image to be recognized is an unknown product and stores the fused feature vector in the active learning pool.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a product recognition method based on image classification as described in any one of claims 1 to 7.

Images