Method for classifying product images based on hierarchical convolutional neural network

Abstract

The application discloses a kind of based on layered convolutional neural network's commodity image classification method, comprising: step 1, only containing single commodity's original commodity image set is obtained and is preprocessed;Step 2, construct layered convolutional neural network model, including feature extraction module, hierarchical fusion classification module;Step 3, using supervised contrast learning optimization the feature extraction module to commodity image data set;Step 4, using classification loss and supervised contrast loss trains the layered convolutional neural network model, carries out commodity image label classification.The application can effectively solve the problem of insufficient use of data features and label relationships, thereby improving the accuracy of commodity image classification while improving the robustness of the commodity image classification model.

Description

Technical Field

[0001] This invention belongs to the field of computer image classification, specifically a product image classification method based on hierarchical convolutional neural networks. Background Technology

[0002] On e-commerce platforms, images are the most intuitive way to display products, and tags are the most concise and core way to describe them. Current technologies primarily classify products by collecting images. On the one hand, these technologies only consider coarse-grained tags for product images, neglecting fine-grained tag classification. However, in e-commerce platforms, fine-grained tags are more valuable in describing product images and better reflect product information. On the other hand, existing tag-supervised learning methods only use single features for training and do not fully utilize the relationships between tags. Therefore, the network needs to learn as many features as possible from the data, making the model highly demanding on data, with poor noise resistance, and insufficient generalization and transfer capabilities. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by proposing a product image classification method based on a hierarchical convolutional neural network. This method aims to simultaneously consider both coarse and fine-grained labels on product images. By fusing local and global features of product images and fully utilizing the correlation between different image labels and the hierarchical relationships within the same image label, a classification network with superior feature extraction and generalization capabilities is trained to achieve product image label classification. This enriches the product image label classification space, better extracts product image features, improves the accuracy of product image label classification, and is more suitable for existing e-commerce platforms.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The product image classification method based on hierarchical convolutional neural networks of this invention is characterized by the following steps:

[0006] Step 1. Obtain the original set of product images containing only a single product and preprocess them to obtain the product image set X = {X1, X2, ..., X}. i ,...,X n}, X i ∈R C×H×W Let represent the i-th product image, where C, H, and W represent the number of channels, height, and width of the image, respectively, 1≤i≤n, and n is the total number of product images;

[0007] Define coarse-grained label set Y = {Y j |j=1,2,…,J}, where Y j Let J represent the set of tags of type j, and J represent the types of tags; and yj,z denotes the jth label set Y j denotes the zth label in Y j denotes the jth label set Y j denotes the number of labels in Y

[0008] the ith product image X i is marked as j the ith product image X i is marked as denotes the ith product image X i is not marked as j denotes the ith product image X i is marked as j

[0009] Step 2. Constructing a hierarchical convolutional neural network model, including: a feature extraction module, a hierarchical fusion classification module;

[0010] The feature extraction module in step 2.1 is composed of 1 convolutional layer, 1 max-pooling layer, and several block blocks in sequence; each block block includes c two-dimensional convolutional layers, c batch normalization layers, and c RELU activation layers, and each two-dimensional convolutional layer is sequentially connected with 1 batch normalization layer and 1 RELU activation layer;

[0011] The product image set X = {X1, X2, …, X i , …, X n} is input into the hierarchical convolutional neural network model, and after being processed by the convolutional layer and the max-pooling layer in the feature extraction module, a product image local feature set is obtained, and after being processed by several block blocks, a product image global feature set is obtained, wherein, denotes the local feature of the ith product image X i denotes the global feature of the ith product image X i ;

[0012] The hierarchical fusion classification module in step 2.2 is composed of an average pooling layer and J fully connected layers, wherein the jth fully connected layer is used as a classifier of the jth label set, and contains Z j neurons, and the output of the zth neuron represents the classification probability value of the zth label y j in the jth label set Y j,z ; ​​​​

[0013] The local feature set F of the product image l and the global feature set F of the product image t The inputs are respectively fed into the average pooling layer for processing to obtain the local pooling feature set of the image. and image global pooling feature set in Represents the image X of the i-th product. i Local pooling characteristics, X represents the i-th product image. i Global pooling characteristics;

[0014] When j=1, global pooling features The j-th fully connected layer is used for processing to obtain the j-th label set Y. j Classification results X represents the i-th product image. i The label set Y of class j j The zth tag The classification probability value;

[0015] When j = 2, 3, ..., J, local pooling features will be used. Global pooling features The classification probability value of the (j-1)th class label set Meta-features are obtained after the fusion operation. Then, the image is input into the j-th fully connected layer for processing to obtain the i-th product image X. i The label set Y of class j j Classification probability value This allows for the local pooling of features. Global pooling features The classification probability value of the J-1th class label set After fusion, the image is input into the J-th fully connected layer for processing, and the i-th product image X is output. i In the J-class label set Y J Classification probability value

[0016] Step 2.3 Calculate the classification loss L for each type of label according to equation (1). cls :

[0017]

[0018] Step 3. Construct an optimization strategy for the feature extraction module in the hierarchical convolutional neural network model;

[0019] Step 3.1 For the i-th product image X i Random data augmentation is performed to obtain the augmented i-th product image X.i ′;

[0020] Step 3.2 merging the commodity image set X = {X1, X2, …, X i ,…,X n} and the enhanced commodity image set X' = {X1', X2', …, X i ′,…,X n ′} to obtain a new commodity image set X'' = {X1'', X2'', …, X s,e ,…,X s,e} and a label set Y'' = {y1'', y2'', …, y s,1 ,…,y s,2}; wherein Xe'' represents the e-th new commodity image, y s e'' represents the label of the e-th new commodity image Xe''. The label y ,e of the e-th new commodity image Xe'' is calculated according to formula (2).

[0021] The similarity S s,2n between the label y e of the e-th new commodity image Xe'' and all labels in the label set {y1'', y2'', …, y e ,…,y 2n} is calculated, thereby obtaining a label similarity matrix S = [S1, S2, …, S e,k ,…,S s,e ];

[0022]

[0023] In formula (2), s s,k represents the similarity between the label y of the e-th new commodity image Xe'' and the label y of the k-th new commodity image Xk'', represents the dot product operation, and || || represents the vector norm.

[0024] Step 3.3 inputting the new commodity image set X'' s into a feature extraction module for processing and outputting new commodity image local features and new commodity image global features Inputting the new commodity image global features F s,t into the average pooling layer in the hierarchical fusion classification module to obtain new image global pooling features F wherein Xe'' represents the e-th new commodity image, F e'' represents the local feature of the e-th new commodity image, F e'' represents the global feature of the e-th new commodity image, and F e'' represents the global pooling feature of the e-th new commodity image. ​​

[0025] Step 3.4: Construct the supervised contrastive loss function L for data training using equation (3). con :

[0026]

[0027] In equation (3), | represents the condition symbol, and β represents the tag similarity threshold. Represents a new set of product images X s Image of anchor The number of images with a label similarity greater than or equal to β. Represents the r-th global pooling feature The e-th global pooling feature with a similarity greater than or equal to the threshold β The dot product result, Represents the r-th global pooling feature Among the 2n global pooling features, excluding The sum of the dot products of its other global pooling features, where t is the dot product temperature parameter;

[0028] Step 4. The training of the hierarchical convolutional neural network model includes two stages:

[0029] Phase One Training:

[0030] Based on the product image set X = {X1, X2, ..., X...} i ,...,X n The hierarchical convolutional neural network is trained, and the classification loss function L is calculated. cls The hierarchical convolutional neural network model is trained by backpropagation. When the number of training epochs reaches a set threshold λ, the second stage of training is performed.

[0031] Second-stage training:

[0032] Based on the new product image set The feature extraction module in the hierarchical convolutional neural network model is first optimized using a contrastive learning method, and the contrastive learning loss L is calculated. con The feature extraction module in the hierarchical convolutional neural network model is used for backpropagation optimization, and then based on the product image set X = {X1, X2, ..., X...} i ,...,X n The hierarchical convolutional neural network model is trained, and the classification loss function L is calculated. cls The hierarchical convolutional neural network model is used for backpropagation training. After multiple iterations of training, the optimal hierarchical convolutional neural network model is obtained and used for product tag classification on e-commerce platforms.

[0033] The electronic equipment comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the commodity image classification method, and the processor is configured to execute the program stored in the memory.

[0034] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the commodity image classification method are executed.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1. The present application considers the coarse and fine granularity labels of commodity images, expands the label space of commodity images, and is more suitable for current e-commerce platform application requirements.

[0037] 2. The present application extracts local features and global features of commodity images, uses the hierarchical relationship within the label, fuses the local features, global features and the prediction value of the upper level label to obtain a new feature representation and then performs the next level label classification, thereby improving the accuracy of the commodity image classification network.

[0038] 3. The present application uses a supervised contrast learning method, fully utilizes the correlation between different image labels to obtain a feature extraction module with stronger effective commodity image feature extraction capability, further improves the network performance and improves the robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The present application is a whole flowchart;

[0040] Figure 2 The present application is a network overall structure diagram;

[0041] Figure 3 The present application is a block structure diagram in the network model;

[0042] Figure 4 The present application is a hierarchical fusion classification module structure diagram in the network model;

[0043] Figure 5 The present application is an original commodity image obtained;

[0044] Figure 6 The present application is a commodity image after data enhancement. DETAILED DESCRIPTION

[0045] In this embodiment, as shown in Figure 1 A commodity image classification method based on a hierarchical convolutional neural network is performed according to the following steps:

[0046] Step 1. Obtain the original set of product images containing only a single product and preprocess them to obtain the product image set X = {X1, X2, ..., X}. i ,…,X n}, X i ∈R C×H×W Let represent the i-th product image, where C, H, and W represent the number of channels, height, and width of the image, respectively, 1 ≤ i ≤ n, and n is the total number of product images.

[0047] Define a label set Y = {Y j |j=1,2,…,J}, where Y j Let J represent the set of tags of type j, and J represent the types of tags; and y j,z Y represents the tag set of class j. j The z-th tag, Z j Y represents the tag set of class j. j The number of tags in the text.

[0048] Let the i-th product image X i The label set Y of class j j The z-th tag in the middle is denoted as Then the i-th product image X i The corresponding tag set is denoted as Represents product image X i Not corresponding to the j-th type of tag set Y j The z-th tag in the middle, Represents product image X i Corresponding to the j-th type of tag set Y j The zth tag in the middle.

[0049] In specific implementation, in step 1, the acquired product images are uniformly preprocessed into R... 3×256×256 The tag set is divided into three categories based on size: category tags, attribute tags, and style tags. Category tags include T-shirts, coats, dresses, etc.; attribute tags include prints, chiffon, crew neck, short sleeves, etc.; and style tags include fashion, casual, retro, etc.

[0050] Step 2. Construct a hierarchical convolutional neural network model, with the following structure: Figure 2 As shown, it includes: a feature extraction module, a hierarchical fusion classification module, and processes the product image set;

[0051] Step 2.1 The feature extraction module consists of one convolutional layer, one max pooling layer, and several blocks. Each block contains c two-dimensional convolutional layers, c batch normalization layers, and c ReLU activation layers. Each two-dimensional convolutional layer is followed by one batch normalization layer and one ReLU activation layer.

[0052] The commodity image set X = {X1, X2, …, XN} is input into a hierarchical convolutional neural network model and processed by a convolutional layer and a max-pooling layer in a feature extraction module to obtain a commodity image local feature set F = {F1, F2, …, FN} i n} and a commodity image global feature set F = {F1, F2, …, FN} after processing by a plurality of block blocks. Wherein, F i l represents the local feature of the i-th commodity image X i , and F i represents the global feature of the i-th commodity image X .

[0053] In specific implementation, the feature extraction module is composed of 1 convolutional layer, 1 max-pooling layer and 5 blocks, wherein each block contains 3 two-dimensional convolutional layers, 3 batch normalization layers and 3 RELU activation layers, and the structure is as shown in Figure 3 Here, a more complex deep residual network structure can be selected to construct the feature extraction module according to the complexity of the commodity image.

[0054] The hierarchical fusion classification module in step 2.2 is composed of 1 average pooling layer and J fully connected layers, wherein the j-th fully connected layer is used as a classifier of the j-th label set and contains Z j neurons, and the output of the z-th neuron represents the classification probability value of the z-th label y j in the j-th label set Y j,z .

[0055] In specific implementation, the hierarchical fusion classification module is composed of 1 average pooling layer and 3 fully connected layers, and the structure is as shown in Figure 4 .

[0056] The commodity image local feature set F l and the commodity image global feature set F t are input into the average pooling layer for processing to obtain an image local pooling feature set F and an image global pooling feature set F . Wherein, F i represents the local pooling feature of the i-th commodity image X i , and F j represents the global pooling feature of the i-th commodity image X .

[0057] When j = 1, the global pooling feature F is input into the j-th fully connected layer for processing to obtain the classification result of the j-th label set Y j . Represents the image X of the i-th product. i The label set Y of class j j The zth tag The classification probability value;

[0058] When j = 2, 3, ..., J, local pooling features will be used. Global pooling features The classification probability value of the (j-1)th class label set Meta-features are obtained after the fusion operation. Then, the image is input into the j-th fully connected layer for processing to obtain the i-th product image X. i The label set Y of class j j Classification probability value This allows for the local pooling of features. Global pooling features The classification probability value of the J-1th class label set After fusion, the image is input into the J-th fully connected layer for processing, and the i-th product image X is output. i In the J-class label set Y J Classification probability value

[0059] Step 2.3 Calculate the classification loss L for each type of label according to equation (1). cls :

[0060]

[0061] Step 3. Construct an optimization strategy for the feature extraction module in the hierarchical convolutional neural network model;

[0062] In specific implementation, the present invention uses two datasets. One dataset is the original dataset that has been collected and preprocessed, denoted as dataset 1. The other dataset is the result of merging the original dataset and the enhanced dataset, denoted as dataset 2.

[0063] Step 3.1 For example Figure 5 The i-th product image X shown i Random data augmentation is performed to obtain the augmented i-th product image X. i ',like Figure 6 As shown.

[0064] Step 3.2 Create a product image set X = {X1, X2, ..., X} i ,…,X n} and the enhanced product image set X′={X1′,X2′,…,X i ′,…,X n After merging, a new set of product images is obtained. and label set wherein, denotes the e-th new product image, y s,e denotes the e-th new product image corresponding label.

[0065] The e-th new product image corresponding label is calculated according to formula (2) and label set The similarity S e of all labels in the new product image set X e , thereby obtaining the label similarity matrix S = [S1, S2, …, S 2n ];

[0066]

[0067] In formula (2), s e,k denotes the similarity of the e-th new product image corresponding label y s,e and the k-th new product image corresponding label y s,k , and denotes the dot product operation, and || || denotes the vector module.

[0068] Step 3.3 inputs the new product image set X s into the feature extraction module for processing, and outputs the new product image local feature and the new product image global feature The new product image global feature F s,t is input into the average pooling layer in the hierarchical fusion classification module, and then the new image global pooling feature F is obtained, wherein, denotes the local feature of the e-th new product image, denotes the global feature of the e-th new product image, denotes the global pooling feature of the e-th new product image.

[0069] Step 3.4 constructs a supervised contrast loss function L con of data training by using formula (3):

[0070]

[0071] In formula (3), | represents the conditional symbol, β represents the label similarity threshold, denotes the number of images in the new product image set X s whose label similarity with the anchor image is greater than or equal to β, denotes the r-th global pooling feature the dot product result of the e-th global pooling feature with the similarity greater than or equal to the threshold value β, denotes the r-th global pooling feature the sum of the dot products of the e-th global pooling feature with other global pooling features in the 2n global pooling features except for itself, and t is the dot product temperature parameter.

[0072] Step 4. Model training, the hierarchical convolutional neural network training process is divided into two stages, and the specific process is as follows:

[0073] First stage training:

[0074] Based on the commodity image set X = {X1, X2,..., X i ,...,X n}, the hierarchical convolutional neural network is trained, and the classification loss function L cls is calculated for backpropagation training of the hierarchical convolutional neural network. When the training times epoch reach the set threshold value λ, the second stage training is performed.

[0075] Second stage training:

[0076] Based on the new commodity image set The feature extraction module in the hierarchical convolutional neural network is optimized using the contrast learning method, and the contrast learning loss L con is calculated for backpropagation optimization of the feature extraction module in the hierarchical convolutional neural network. Based on the commodity image set X = {X1, X2,..., X i ,...,X n}, the hierarchical convolutional neural network is trained, and the classification loss function L cls is calculated for backpropagation training of the hierarchical convolutional neural network. After multiple iterations of training, a hierarchical convolutional neural network model with good classification effect is obtained, so that the trained hierarchical convolutional neural network model is used for commodity label classification on Taobao and other e-commerce platforms.

[0077] In specific implementation, the training threshold value λ = 10 is set. Here, the training threshold value can be set according to the first stage network training result.

[0078] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0079] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above method.

Claims

1. A product image classification method based on hierarchical convolutional neural networks, characterized in that, The procedure is as follows: Step 1. Obtain the original set of product images containing only a single product and preprocess them to obtain the product image set X = {X1, X2, ..., X}. i ,…,X n }, X i ∈R C×H×W Let represent the i-th product image, where C, H, and W represent the number of channels, height, and width of the image, respectively, 1≤i≤n, and n is the total number of product images; Define coarse-grained label set Y = {Y j |j=1,2,…,J}, where Y j Let J represent the set of tags of type j, and J represent the types of tags; and y j,z Y represents the tag set of the j-th class. j The z-th tag, Z j Y represents the tag set of class j. j The number of tags in; Let the i-th product image X i The label set Y of class j j The z-th tag in the middle is denoted as Then the i-th product image X i The corresponding tag set is denoted as X represents the i-th product image. i Not corresponding to the j-th type of tag set Y j The z-th tag in the middle, X represents the i-th product image. i Corresponding to the j-th type of tag set Y j The z-th tag; Step 2. Construct a hierarchical convolutional neural network model, including: a feature extraction module and a hierarchical fusion classification module; The feature extraction module described in step 2.1 consists of one convolutional layer, one max pooling layer, and several blocks. Each block contains c two-dimensional convolutional layers, c batch normalization layers, and c ReLU activation layers, and each two-dimensional convolutional layer is followed by one batch normalization layer and one ReLU activation layer. Let the product image set X = {X1, X2, ..., X...} i ,…,X n The input is fed into the hierarchical convolutional neural network model, and after processing by the convolutional layers and max pooling layers in the feature extraction module, a local feature set of the product image is obtained. After processing through several blocks, the global feature set of the product image is obtained. Among them, F i l X represents the i-th product image. i Local features, F i t X represents the i-th product image. i Global features; The hierarchical fusion classification module described in step 2.2 consists of an average pooling layer and J fully connected layers, wherein the j-th fully connected layer serves as the classifier for the j-th label set and contains Z... j There are z neurons, and the output of the z-th neuron represents the label set Y of the j-th class. j The zth tag y j,z The classification probability value; The local feature set F of the product image l and the global feature set F of the product image t The inputs are respectively fed into the average pooling layer for processing to obtain the local pooling feature set f of the image. avgpool (F l )={F1′ l ,…,F i ′ l ,…,F n ′ l } and image global pooling feature set Where F i ′ l X represents the i-th product image. i The local pooling feature, F i ′ t X represents the i-th product image. i Global pooling characteristics; When j=1, the global pooling feature F i ′ t The j-th fully connected layer is used for processing to obtain the j-th label set Y. j Classification results X represents the i-th product image. i The label set Y of class j j The zth tag The classification probability value; When j = 2, 3, ..., J, the local pooling feature F i ′ l Global pooling feature F i ′ t The classification probability value of the (j-1)th class label set Meta-features are obtained after the fusion operation. Then, the image is input into the j-th fully connected layer for processing to obtain the i-th product image X. i The label set Y of class j j Classification probability value Thus, the local pooling feature F i ′ l Global pooling feature F i ′ t The classification probability value of the J-1th class label set After fusion, the image is input into the J-th fully connected layer for processing, and the i-th product image X is output. i In the J-class label set Y J Classification probability value Step 2.3 Calculate the classification loss L for each type of label according to equation (1). cls : Step 3. Construct an optimization strategy for the feature extraction module in the hierarchical convolutional neural network model; Step 3.1 For the i-th product image X i Random data augmentation is performed to obtain the augmented i-th product image X. i ′; Step 3.2 Set the product image set X = {X1, X2, ..., X...} i ,…,X n } and the enhanced product image set X′={X′1,X′2,…,X′ i ,…,X′ n After merging, a new set of product images is obtained. and tag set {y s ,1 ,y s,2 ,…,y s,e ,…,y s,2n },in, Let y represent the image of the e-th new product. s,e This represents the image of the e-th new product. Tags; Calculate the image of the e-th new product according to formula (2). The corresponding label y s,e With tag set {y s,1 ,y s,2 ,…,y s,e ,…,y s,2n The similarity S of all tags in} e Thus, the label similarity matrix S = [S1, S2, ..., S...] is obtained. e ,…,S 2n ]; In equation (2), s e,k This represents the image of the e-th new product. The corresponding label y s,e With the kth new product image The corresponding label y s,k The similarity is represented by ⊙, where ⊙ denotes the dot product operation, and || denotes the magnitude of the vectors. Step 3.3 The new product image set X s The input is processed in the feature extraction module, and new local features of the product image are output. and new global features of product images The new global feature F of the product image s,t After being processed by the average pooling layer in the hierarchical fusion classification module, new global pooling features of the image are obtained. in, This represents the local features of the e-th new product image. Let F′ represent the global features of the e-th new product image. e s,t Represents the global pooling feature of the e-th new product image; Step 3.4: Construct the supervised contrastive loss function L for data training using equation (3). con : In equation (3), | represents the condition symbol, and β represents the tag similarity threshold. Represents a new set of product images X s Image of anchor The number of images with a label similarity greater than or equal to β. Represents the r-th global pooling feature The e-th global pooling feature with a similarity greater than or equal to the threshold β The dot product result, Represents the r-th global pooling feature Among the 2n global pooling features, excluding The sum of the dot products of its other global pooling features, where t is the dot product temperature parameter; Step 4. The training of the hierarchical convolutional neural network model includes two stages: Phase One Training: Based on the product image set X = {X1, X2, ..., X...} i ,…,X n The hierarchical convolutional neural network is trained, and the classification loss function L is calculated. cls The hierarchical convolutional neural network model is trained by backpropagation. When the number of training epochs reaches a set threshold λ, the second stage of training is performed. Second-stage training: Based on the new product image set The feature extraction module in the hierarchical convolutional neural network model is first optimized using a contrastive learning method, and the contrastive learning loss L is calculated. con The feature extraction module in the hierarchical convolutional neural network model is used for backpropagation optimization, and then based on the product image set X = {X1, X2, ..., X...} i ,…,X n The hierarchical convolutional neural network model is trained, and the classification loss function L is calculated. cls The hierarchical convolutional neural network model is used for backpropagation training. After multiple iterations of training, the optimal hierarchical convolutional neural network model is obtained and used for product tag classification on e-commerce platforms.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the product image classification method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when run by the processor, executes the steps of the commodity image classification method of claim 1.

Images