Method for training clothing classification model, clothing classification method and related device

Abstract

The embodiment of the present application relates to the field of image processing technology, and discloses a method for training a clothing classification model, a clothing classification method, and related devices, wherein a training set is obtained, and the training set includes multiple clothing images and text information. Each text information is encoded to obtain a text encoding feature. The first neural network is iteratively trained with each clothing image and the corresponding text encoding feature, and the first neural network after convergence is used as a clothing classification model. Based on the fact that the text information includes clothing descriptions, the first neural network is trained with multiple clothing images and corresponding text information. The first neural network can learn the image features reflected by the clothing images and the text features reflected by the text information. The text features can assist the first neural network in learning image features that are more representative in clothing classification, thereby making the image features extracted by the clothing classification model representative in clothing classification and improving the accuracy of clothing classification.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method for training a clothing classification model, a clothing classification method, and related devices. Background Art

[0002] With the continuous advancement of modern technology, the scale of online shopping is constantly increasing. Users can buy clothes on online shopping platforms through their mobile phones. Intelligent processing of clothing categories is currently an important part of improving user experience.

[0003] With the continuous development of neural networks, they have been widely used in the field of image classification. Therefore, researchers have applied neural networks to clothing classification. These proposed neural networks generally extract image features and use them for classification. However, given the vast variety of clothing categories, the similarities between clothing categories are often limited, often only differing in fabric, texture, and other aspects, making them difficult to distinguish. Summary of the Invention

[0004] The main technical problem solved by the embodiments of the present application is to provide a method for training a clothing classification model, a clothing classification method and related devices, wherein the clothing classification model trained by the training method can accurately classify clothing.

[0005] To solve the above technical problems, in a first aspect, an embodiment of the present application provides a method for training a clothing classification model, comprising:

[0006] Obtaining a training set, the training set including multiple clothing images and text information corresponding to the multiple clothing images, the text information including clothing descriptions, and each clothing image is respectively labeled with a real clothing category;

[0007] Encode each text information to obtain text encoding features;

[0008] Inputting the clothing image and the text encoding feature corresponding to the clothing image into a first neural network to obtain a first predicted clothing category corresponding to the clothing image;

[0009] According to the difference between the first predicted clothing category corresponding to the training set and the actual clothing category corresponding to the training set, the parameters of the first neural network are adjusted until convergence, and the converged first neural network is used as the clothing classification model.

[0010] In some embodiments, the aforementioned encoding of each text message to obtain text encoding features includes:

[0011] Use word embedding algorithm to vectorize each word in the text information to obtain a vector list;

[0012] Perform feature extraction on the vector list to obtain text encoding features.

[0013] In some embodiments, the aforementioned feature extraction of the vector list to obtain text encoding features includes:

[0014] The multi-layer perceptron module is used to extract features from the vector list to obtain text encoding features.

[0015] In some embodiments, the first neural network includes a convolution module, a fusion module, a fully connected layer, and a classification layer cascaded in sequence;

[0016] Inputting the clothing image and the text encoding features corresponding to the clothing image into the first neural network, obtaining a first predicted clothing category corresponding to the clothing image, including:

[0017] Input the clothing image into the convolution module for downsampling feature extraction to obtain the clothing feature map;

[0018] The clothing feature map and the text encoding feature are input into the fusion module for feature fusion to obtain a fused feature map;

[0019] After the fusion feature map passes through the fully connected layer and the classification layer, the first predicted clothing category is output.

[0020] In some embodiments, the fusion module performs feature fusion on the clothing feature map and the text encoding feature using the following formula;

[0021] F(x,e)=α*x+β

[0022] α=MLP1(e); β=MLP2(e)

[0023] Among them, x is the clothing feature map, e is a vector list, α is a text encoding feature corresponding to the clothing image, and β is another text encoding feature corresponding to the clothing image.

[0024] In some embodiments, the method further comprises:

[0025] Inputting the clothing image into a second neural network to obtain a second predicted clothing category corresponding to the clothing image, wherein the depth of the second neural network is less than the depth of the first neural network;

[0026] The loss function is used to calculate the loss, and the parameters of the first neural network and the second neural network are adjusted according to the loss until convergence, and the converged second neural network is used as the clothing classification model;

[0027] Among them, the loss function includes distillation loss and classification loss. The distillation loss reflects the difference between the intermediate feature map output by the first neural network and the intermediate feature map output by the second neural network. The classification loss reflects the difference between the first predicted clothing category, the second predicted clothing category and the actual clothing category.

[0028] In some embodiments, the loss function includes:

[0029]

[0030] in,

[0031]

[0032] in, is the distillation loss, L t-s is the classification loss, is the intermediate feature map corresponding to the first neural network, G s is the intermediate feature map corresponding to the second neural network, n L represents the number of intermediate feature maps of size L, represents the network parameters of the first neural network corresponding to the i-th feature map of size L, represents the network parameters of the second neural network corresponding to the i-th feature map of size L, n represents the total number of clothing categories, y i represents the probability value of the i-th category in the real clothing category, represents the probability value of the first neural network predicting the i-th category, Represents the probability value of the second neural network predicting the i-th category.

[0033] To solve the above technical problems, in a second aspect, an embodiment of the present application provides a clothing classification method, comprising:

[0034] A clothing classification model is used to identify clothing categories of clothing images to be classified to obtain corresponding clothing categories, wherein the clothing classification model is trained using the method for training a clothing classification model according to any one of claims 1 to 7.

[0035] To solve the above technical problems, in a third aspect, an embodiment of the present application provides an electronic device, including:

[0036] at least one processor, and

[0037] a memory communicatively coupled to at least one processor, wherein:

[0038] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0039] To solve the above technical problems, in a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the method of the first aspect.

[0040] Beneficial effects of the embodiments of the present application: Different from the prior art, the method for training a clothing classification model provided by the embodiments of the present application first obtains a training set, which includes multiple clothing images and text information corresponding to the multiple clothing images, each of which is labeled with a real clothing category. Each text information is encoded to obtain a text encoding feature. Then, a first neural network is iteratively trained using each clothing image and the corresponding text encoding feature, and the first neural network after convergence is used as a clothing classification model. In this solution, based on the text information including clothing descriptions, for example, the text information can be extracted from clothing tags or detailed descriptions of the clothing, the first neural network is trained using multiple clothing images and corresponding text information. The first neural network can learn the image features reflected by the clothing images and the text features reflected by the text information. The text features can assist the first neural network in learning image features that are more representative for clothing classification. As a result, the image features extracted by the clothing classification model are representative for clothing classification, which can improve the accuracy of clothing classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0042] Figure 1 This is a schematic diagram of an application scenario of a clothing classification system in some embodiments of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of an electronic device in some embodiments of the present application;

[0044] Figure 3 This is a flowchart of a method for training a clothing classification model in some embodiments of the present application;

[0045] Figure 4 for Figure 3 A schematic diagram of a sub-process of step S20 in the method shown;

[0046] Figure 5 for Figure 3 A schematic diagram of a sub-process of step S30 in the method shown;

[0047] Figure 6 This is a schematic diagram of training the first neural network in some embodiments of the present application;

[0048] Figure 7 This is a flowchart of a method for training a clothing classification model in some embodiments of the present application;

[0049] Figure 8 This is a schematic diagram of the training of the second neural network in some embodiments of the present application. DETAILED DESCRIPTION

[0050] The present application is described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but are not intended to limit the present application in any form. It should be noted that those skilled in the art may make several variations and improvements without departing from the scope of the present application. These all fall within the scope of protection of the present application.

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other and are all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish between the same items or similar items with basically the same functions and effects.

[0053] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the relevant listed items.

[0054] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0055] To facilitate understanding of the methods provided in the embodiments of the present application, the following nouns are first introduced:

[0056] (1) Neural Network

[0057] A neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. A neural network with many hidden layers is called a deep neural network (DNN). The operation of each layer in a neural network can be described by the mathematical expression y = a(W·x+b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / reduction; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. Bending. Operations 2 and 3 are performed by "W·x," operation 4 is performed by "+b," and operation 5 is performed by "a()." The word "space" is used here because the object being classified is not a single thing, but a category of things. Space refers to the collection of all individuals in this category. W is the weight matrix of each layer of the neural network. Each value in this matrix represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from input space to output space described above. That is, W in each layer of the neural network controls how space is transformed. The goal of training a neural network is to ultimately obtain the weight matrices for all layers of the trained neural network. Therefore, the process of training a neural network is essentially learning how to control spatial transformations, more specifically, learning the weight matrices.

[0058] It should be noted that in the embodiments of this application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, pooling layers, normalization layers, and deconvolution layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined so that the model error meets preset conditions or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0059] The convolutional layer is configured with multiple convolution kernels, each with a corresponding step size, to perform convolution operations on the image. The purpose of the convolution operation is to extract different features of the input image. The first convolution layer may only extract low-level features such as edges, lines, and corners. Deeper convolution layers can iteratively extract more complex features from these low-level features.

[0060] The deconvolution layer is used to map a low-dimensional space to a high-dimensional space while maintaining the connection relationship / pattern between them (the connection relationship here refers to the connection relationship during convolution). The deconvolution layer is configured with multiple convolution kernels, each with a corresponding step size, to perform deconvolution operations on the image. Generally, framework libraries used to design neural networks (such as the PyTorch library) have a built-in upsumple() function, which can be called to achieve low-dimensional to high-dimensional spatial mapping.

[0061] Pooling layers mimic the human visual system, reducing the dimensionality of data or representing images with higher-level features. Common pooling layer operations include max pooling, mean pooling, random pooling, median pooling, and combined pooling. Generally speaking, pooling layers are periodically inserted between convolutional layers in neural networks to achieve dimensionality reduction.

[0062] The normalization layer is used to normalize all neurons in the intermediate layer to prevent gradient explosion and gradient disappearance.

[0063] (2) Loss function

[0064] During neural network training, the goal is to ensure that the network's output is as close as possible to the desired predicted value. This is done by comparing the network's predictions with the desired target values. The weight matrices of each layer are then updated based on the difference between the two. (However, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the neural network.) For example, if the network's prediction is too high, the weight matrix is adjusted to predict a lower value. This adjustment is repeated until the network can predict the desired target value. Therefore, it is necessary to predefine how to compare the difference between the predicted and target values. This is known as the loss function, or objective function. These are important equations that measure the difference between the predicted and target values. For example, a higher loss function output (loss) indicates a greater difference, so neural network training becomes a process of minimizing this loss.

[0065] Before introducing the embodiments of the present application, a clothing classification method known to the inventor of the present application is briefly introduced to facilitate subsequent understanding of the embodiments of the present application.

[0066] Some clothing classification methods use clothing images to train a convolutional neural network to generate a clothing classification model, which is then used to extract image features and perform classification. This type of clothing classification model, trained using clothing images, cannot effectively handle the diversity of image samples and changes in the spatial structure of samples. Therefore, significant color differences or brightness variations, as well as occlusions and unusual changes in spatial structure, can cause a sharp drop in detection accuracy.

[0067] In some clothing classification methods, the attention mechanism is used to amplify the key vectors and weights of clothing image features, perform convolution and normalization on the image features, and use the spatial transformation network to transform the receptive field of the image features; the image features are input into the capsule network to extract the spatial correlation information of the image features and improve the generalization ability.

[0068] However, the technologies known to the inventors of this application, whether convolutional neural networks or capsule networks, are all about extracting image features. Simply using image feature information is not effective in dealing with such a wide variety of clothing features and clothing categories, and its accuracy is obviously insufficient.

[0069] In response to the above problems, embodiments of the present application provide a method for training a clothing classification model, a clothing classification method, an electronic device, and a storage medium, wherein the training method uses multiple clothing images and corresponding text information to train a first neural network, wherein the text information includes clothing descriptions, for example, the text information can be extracted from clothing tags or detailed descriptions of the clothing. The first neural network can learn the image features reflected by the clothing images and the text features reflected by the text information. The text features can assist the first neural network in learning image features that are more representative in clothing classification, thereby making the image features extracted by the trained clothing classification model representative in clothing classification and improving the accuracy of clothing classification.

[0070] The following describes exemplary applications of electronic devices for training clothing classification models or for clothing classification provided in embodiments of the present application. It can be understood that the electronic device can train clothing classification models and can also use the clothing classification models to classify clothing.

[0071] The electronic device provided in the embodiments of the present application can be a server, such as a server deployed in the cloud. When the server is used to train a clothing classification model, the neural network is iteratively trained using the training set and neural network provided by other devices or those skilled in the art to determine the final model parameters. The neural network then configures the final model parameters to obtain the clothing classification model. When the server is used for clothing classification, the built-in clothing classification model is called to perform corresponding computational processing on clothing images to be classified provided by other devices or users to obtain the clothing category corresponding to the clothing images to be classified.

[0072] The electronic devices provided in some embodiments of the present application can be various types of terminals, such as laptop computers, desktop computers, or mobile devices. When the terminal is used to train a clothing classification model, a person skilled in the art inputs a prepared training set into the terminal and designs a neural network on the terminal. The terminal uses the training set to iteratively train the neural network, determining the final model parameters. The neural network then configures the final model parameters to obtain a clothing classification model. When the terminal is used for clothing classification, the built-in clothing classification model is called to perform corresponding calculations on the clothing image to be classified input by the user to obtain the clothing category corresponding to the clothing image to be classified.

[0073] For example, see Figure 1 , Figure 1 2 is a schematic diagram of an application scenario of a clothing classification system provided in an embodiment of the present application. The terminal 10 is connected to the server 20 via a network, wherein the network may be a wide area network or a local area network, or a combination of the two.

[0074] Terminal 10 can be used to obtain a training set and construct a neural network. For example, a person skilled in the art can download a prepared training set to the terminal and build the neural network structure. It is understood that terminal 10 can also be used to obtain clothing images to be classified. For example, a user can input the clothing images to be classified through an input interface. After the input is completed, the terminal automatically obtains the clothing images to be classified. For example, terminal 10 has a camera and captures clothing images through the camera, or terminal 10 stores a clothing image library, and the user can select the clothing images to be classified from the clothing image library.

[0075] In some embodiments, the terminal 10 locally executes the method for training a clothing classification model provided in an embodiment of the present application to complete the training of the designed neural network using the training set, determine the final model parameters, and thus the neural network configures the final model parameters to obtain the clothing classification model. In some embodiments, the terminal 10 can also send the training set and the constructed neural network stored on the terminal by a person skilled in the art to the server 20 via the network. The server 20 receives the training set and the neural network, trains the designed neural network using the training set, determines the final model parameters, and then sends the final model parameters to the terminal 10. The terminal 10 saves the final model parameters, so that the neural network is configured to configure the final model parameters, and the clothing classification model can be obtained.

[0076] In some embodiments, the terminal 10 locally executes the clothing classification method provided in the embodiments of the present application to provide clothing classification services for users, calls the built-in clothing classification model, performs corresponding calculations on the clothing images and text information to be classified, and obtains the clothing category corresponding to the clothing image to be classified. In some embodiments, the terminal 10 can also send the clothing images and text information to be classified input by the user on the terminal to the server 20 through the network. The server 20 receives the clothing images and text information to be classified, calls the built-in clothing classification model to perform corresponding calculations on the clothing images and text information to be classified, obtains the clothing category corresponding to the clothing image to be classified, and then sends the clothing category to the terminal 10. After receiving the clothing category, the terminal 10 displays the clothing category on its own display interface to inform the user.

[0077] The structure of the electronic device in the embodiment of the present application is described below. Figure 2 5 is a schematic diagram of the structure of an electronic device 500 in an embodiment of the present application. The electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0078] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0079] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual displays. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, and other input buttons and controls.

[0080] The memory 550 includes volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in the embodiments of this application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices physically remote from the processor 510.

[0081] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0082] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0083] A network communication module 552 for reaching other computing devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, WiFi, and USB;

[0084] a display module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0085] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0086] It can be understood from the above that the method for training a clothing classification model and the clothing classification method provided in the embodiments of the present application can be implemented by various types of electronic devices with computing and processing capabilities, such as smart terminals and servers.

[0087] The following describes the method for training a clothing classification model provided by the embodiment of the present application in conjunction with the exemplary application and implementation of the server provided by the embodiment of the present application. Figure 3 , Figure 3 3 is a flowchart of a method for training a clothing classification model provided in an embodiment of the present application.

[0088] Please refer again Figure 3 The method S100 may specifically include the following steps:

[0089] S10: Obtain training set.

[0090] The training set includes multiple clothing images and text information corresponding to each of the clothing images. It is understood that each clothing image corresponds to one piece of text information. Each clothing image is labeled with a real clothing category. In some embodiments, the number of clothing images in the training set is in the tens of thousands, for example, 20,000, which facilitates training and obtaining an accurate general model. Those skilled in the art can determine the number of clothing images based on actual circumstances.

[0091] There are many ways to classify clothing. For example, by gender, it can be divided into men's clothing, women's clothing, and children's clothing; by combination, it can be divided into dresses, suits, jackets, vests, skirts, trousers, shorts, or shorts. It is understandable that the above clothing categories can be further subdivided by style. For example, dresses can be divided into commuting style dresses, casual style dresses, or sports style dresses. Examples are not given here one by one. It is understandable that those skilled in the art can determine the clothing categories that can be included in the training set according to recognition requirements, for example, including clothing images corresponding to 10 clothing categories, so that the trained clothing classification model can recognize these 10 clothing categories.

[0092] Each clothing image is labeled with its actual clothing category. It should be understood that this actual clothing category is the actual category of the clothing in the corresponding clothing image. The actual clothing category is equivalent to a label and can be encoded using hot encoding. Hot encoding is a common technique in this field and will not be discussed in detail here.

[0093] The text information includes a description of the clothing item, which may include the clothing item's style, fabric, design, collar, length, fit, sleeve length, thickness, or suitable season. In one embodiment, the text information may be extracted from clothing tags or detailed descriptions of the clothing item on a webpage. Based on the inclusion of the aforementioned clothing description in the text information, effective information can be provided for clothing classification.

[0094] It is understood that a training set including clothing images and text information can be collected in advance by a person skilled in the art on a terminal (e.g., a computer). For example, clothing images and corresponding detailed descriptions (i.e., text information) can be crawled from some clothing sales websites. After the training set is prepared, it is uploaded to the server via the terminal.

[0095] In some embodiments, the terminal or server can perform normalization on the clothing images in the training set, which is beneficial to improving the convergence speed and model accuracy of subsequent model training. Specifically, in some embodiments, the size of the clothing images can be set to 256*256, and the pixel value range of the clothing images can be converted from 0-255 to 0-1. In some embodiments, the following formula can be used for pixel value conversion:

[0096]

[0097] Among them, x i is the i-th pixel value of the image, max(x) is the maximum pixel value in the clothing image, min(x) is the minimum pixel value in the clothing image, and norm is the converted pixel value.

[0098] S20: Encode each text information to obtain text encoding features.

[0099] It's understandable that text information is data in a text format. To enable the neural network to learn the features that text information can reflect, the text information is encoded to obtain text encoding features. In other words, the text information is digitized, and the resulting text encoding features are digital data.

[0100] In some embodiments, see Figure 4 , the aforementioned step S20 specifically includes:

[0101] S21: Use word embedding algorithm to vectorize each word in the text information to obtain a vector list.

[0102] It is understandable that a word embedding algorithm is a method for converting words in a text into digital vectors. For example, a word embedding algorithm can be a word2vec algorithm. The word embedding algorithm is used to vectorize each word in the text information, that is, to convert the words into vectors (digital data). Thus, each word vector constitutes a vector list corresponding to the text information. It is understandable that in practice, the length of each text information is different, that is, the number of words included is different, and an appropriate value N can be set as the length of the vector list. If a text information does not reach this length, it can be filled with a word vector of all 0s. If a text information exceeds this length, it is truncated.

[0103] S22: Perform feature extraction on the vector list to obtain text encoding features.

[0104] After obtaining the vector list, in order to facilitate the neural network to learn the features in the vector list, feature extraction is performed on the vector list to obtain a one-dimensional text encoding feature. In some embodiments, a convolutional feature extraction network can be used to extract features and reduce the dimension of the vector list to obtain the text encoding feature. It can be understood that the convolutional feature extraction network includes a series of convolutional layers, pooling layers or normalization layers to achieve feature extraction and dimensionality reduction. The concepts of convolutional layers, pooling layers or normalization layers have been introduced in the above "(1) Neural Network" and will not be repeated here.

[0105] In some embodiments, the aforementioned step S22 specifically includes: using a multi-layer perceptron module to perform feature extraction on the vector list to obtain the text encoding feature.

[0106] The multilayer perceptron module consists of an input layer, multiple hidden layers, and an output layer. The input layer consists of N neurons, the hidden layer consists of Q neurons, and the output layer consists of K neurons. The operation of each layer can be described by a function expression, which is understandable because the function expression for each layer is different.

[0107] It's easy to understand that if the input vector list is represented by x, then the input layer feeds the hidden layer x, and the hidden layer output can be f(w1x+b1), where w1 is the weight, b1 is the bias, and the function f can be a common sigmoid function or tanh function. The transition from the hidden layer to the output layer is equivalent to a multi-class logistic regression, or softmax regression. Therefore, the output of the output layer is softmax(w2x1+b2), where x1 is the hidden layer output f(w1x+b1).

[0108] Therefore, the multi-layer perceptron module can be expressed as follows:

[0109]

[0110] Among them, G represents the softmax activation function, h represents the number of hidden layers, and W i and b i represents the weights and biases of the ith hidden layer. x represents the input vector list. W 1 and b 1 Represents the weight and bias of the input layer, S represents the activation function, and MLP(x) represents the text encoding feature.

[0111] In some embodiments, K may be 1024, so that the output layer outputs a one-dimensional vector with a length of 1024, that is, a text encoding feature with a length of 1024.

[0112] The activation functions used in each layer of the Multilayer Perceptron module introduce nonlinearities into the neurons, allowing the module to approximate any nonlinear function, thus enabling its application in a wider range of nonlinear models. The Multilayer Perceptron module has excellent feature extraction capabilities for discrete information, resulting in the extracted text encoding features that fully reflect the characteristics of the text.

[0113] S30: Inputting the clothing image and the text encoding features corresponding to the clothing image into a first neural network to obtain a first predicted clothing category corresponding to the clothing image.

[0114] The clothing images and the corresponding text encoding features are input into the first neural network. Based on the fact that the clothing images are labeled with real clothing categories, the first neural network will learn the relationship between the image features in the clothing images, the text features in the text encoding features, and the real clothing categories, and predict the clothing category corresponding to each clothing image, that is, obtain a first predicted clothing category.

[0115] In some embodiments, the first neural network includes a convolution module, a fusion module, a fully connected layer, and a classification layer that are cascaded in sequence. The convolution module includes multiple convolution layers, which are used to downsample the input clothing image to extract features; the fusion module is used to fuse at least two features. The fully connected layer is used to integrate and classify the input features and output a one-dimensional vector; the classification layer is used to convert the one-dimensional vector input by the fully connected layer into a probability vector with a value between 0 and 1, thereby achieving classification. It is worth noting that the convolution layer, the fully connected layer, and the classification layer are commonly used components in neural networks and are well known to those skilled in the art. They will not be introduced in detail here.

[0116] See also Figure 5 , the aforementioned step S30 specifically includes:

[0117] S31: Input the clothing image into the convolution module for downsampling feature extraction to obtain a clothing feature map.

[0118] In some embodiments, see Figure 6 The convolution module includes multiple convolution layers, each of which is followed by a pooling layer for feature extraction and dimensionality reduction. The sizes of the feature maps output by some convolution layers are 128*128*64, 64*64*128, 32*32*256, 16*16*256, and 8*8*512, respectively. It is understandable that the feature map output by the last convolution layer in the convolution module is the clothing feature map. In the aforementioned embodiment, the feature map of size 8*8*512 is the clothing feature map.

[0119] S32: The clothing feature map and the text encoding feature are input into the fusion module for feature fusion to obtain a fused feature map.

[0120] It is understood that the fused feature map obtained by fusion has both the image features reflected by the clothing feature map and the text features reflected by the text encoding features. Thus, the fused feature map enables the first neural network to learn image features that are more representative for clothing classification. In some embodiments, fusion methods can include feature concatenation, feature summation, and element-wise multiplication of corresponding features.

[0121] In some embodiments, the fusion module uses the following formula to perform feature fusion on the clothing feature map and the text encoding feature.

[0122] F(t,e)=α*t+β

[0123] α=MLP1(e); β=MLP2(e)

[0124] Among them, t is the clothing feature map, e is a vector list, α is a text encoding feature corresponding to the clothing image, β is another text encoding feature corresponding to the clothing image, and F(t,e) is the fusion feature map.

[0125] In this embodiment, α and β can be obtained by extracting features from a vector list using two different weight parameters in a multi-layer perceptron module. Linearly fusing the two text encoding features α and β with the clothing feature t, through multiplication and summation, maximizes the correlation between the corresponding features in the clothing feature map and the text encoding features, while maximizing the differences between different classes. This improves the fusion effect and makes the features of the fused feature map F(t,e) more representative.

[0126] S33: After the fusion feature map passes through the fully connected layer and the classification layer, the first predicted clothing category is output.

[0127] The fully connected layer integrates and classifies the input features, outputting a one-dimensional vector. The classification layer converts the one-dimensional vector input by the fully connected layer into a probability vector with values between 0 and 1, thereby achieving classification. Thus, after the fused feature map is processed by the fully connected layer, a one-dimensional vector is obtained, which is then converted into a probability vector with values between 0 and 1 after processing by the classification layer. It can be understood that this probability vector is a vector representation of the first predicted clothing category, where the elements in the probability vector represent the probabilities of the clothing in the clothing image belonging to each category. It can be understood that the category with the highest probability is the first predicted clothing category.

[0128] S40: According to the difference between the first predicted clothing category corresponding to the training set and the actual clothing category corresponding to the training set, the parameters of the first neural network are adjusted until convergence, and the converged first neural network is used as the clothing classification model.

[0129] It can be understood that the smaller the difference between the actual clothing category corresponding to each clothing image in the training set and the first predicted clothing category, the closer the first predicted clothing category is to the actual clothing category, indicating that the first neural network can accurately determine the clothing category corresponding to each clothing image. Therefore, the model parameters of the first neural network can be adjusted based on the difference between the first predicted clothing category corresponding to the training set and the actual clothing category corresponding to the training set, and the first neural network can be iteratively trained.

[0130] It is understandable that the convergence here may mean that under certain model parameters, the sum of the differences between the actual clothing categories in the training set and the first predicted clothing categories is less than a preset threshold or fluctuates within a certain range.

[0131] In some embodiments, the Adam algorithm is used to optimize model parameters. For example, the number of iterations is set to 100,000, the initial learning rate is set to 0.001, the weight decay of the learning rate is set to 0.0005, and the learning rate decays to 1 / 10 of the original value every 1,000 iterations. The learning rate and the difference between each real clothing category in the training set and the corresponding first predicted clothing category can be input into the Adam algorithm to obtain the adjusted model parameters output by the Adam algorithm. The adjusted model parameters are used for the next training until the training is completed, and the model parameters of the converged first neural network are output. Thus, the converged first neural network is used as the clothing classification model.

[0132] It's understandable that the training set includes multiple clothing images annotated with real clothing categories and corresponding text information, for example, 20,000 clothing images. The images cover a wide range of clothing types, encompassing most clothing types on the market. Therefore, the trained clothing classification model is a general model that can be widely used for clothing classification.

[0133] In this embodiment, based on the text information including clothing descriptions, for example, the text information can be extracted from clothing tags or detailed descriptions of the clothing, a first neural network is trained using multiple clothing images and corresponding text information. The first neural network can learn the image features reflected by the clothing images and the text features reflected by the text information. The text features can assist the first neural network in learning image features that are more representative in clothing classification, thereby making the image features extracted by the clothing classification model representative in clothing classification and improving the accuracy of clothing classification.

[0134] In some embodiments, see Figure 7 , the aforementioned method S100 further includes:

[0135] S50: Inputting the clothing image into a second neural network to obtain a second predicted clothing category corresponding to the clothing image, wherein the depth of the second neural network is less than the depth of the first neural network.

[0136] S60: Calculate the loss using a loss function, and adjust the parameters of the first neural network and the second neural network according to the loss until convergence, and use the converged second neural network as a clothing classification model.

[0137] Among them, the loss function includes distillation loss and classification loss. The distillation loss reflects the difference between the intermediate feature map output by the first neural network and the intermediate feature map output by the second neural network. The classification loss reflects the difference between the first predicted clothing category, the second predicted clothing category and the actual clothing category.

[0138] As can be seen above, after the clothing image is input into the first neural network, the convolution module in the first neural network extracts features from the clothing image. Some convolutional layers in the convolution module output intermediate feature maps of sizes 128*128*64, 64*64*128, 32*32*256, 16*16*256, and 8*8*512, respectively. The 8*8*512 intermediate feature map output by the last convolutional layer in the convolution module is the clothing feature map. The clothing feature map and the text encoding features are input into the fusion module for feature fusion to obtain a fused feature map. After the fused feature map passes through the fully connected layer and the classification layer, the first predicted clothing category is output.

[0139] See also Figure 8 In this embodiment, the depth of the second neural network is less than that of the first neural network. The second neural network also has a convolutional module, a fully connected layer, and a classification layer. Among them, some convolutional layers in the convolutional module of the second neural network also output feature maps of sizes of 128*128*64, 64*64*128, 32*32*256, 16*16*256, and 8*8*512. The last feature map is passed through the fully connected layer and classification layer in the second neural network to output the second predicted clothing category.

[0140] It is understandable that the first neural network with greater depth, with the assistance of text information, can learn more representative image features in clothing classification. Therefore, when the first neural network and the second neural network are trained together, the first neural network can converge faster and the classification prediction results are more accurate.

[0141] Since the loss function includes distillation loss and classification loss, the distillation loss reflects the difference between the intermediate feature map output by the first neural network and the intermediate feature map output by the second neural network, and the classification loss reflects the difference between the first predicted clothing category, the second predicted clothing category and the actual clothing category.

[0142] In some embodiments, the loss function includes:

[0143]

[0144] in,

[0145]

[0146] in, is the distillation loss, L t-s is the classification loss, is the intermediate feature map corresponding to the first neural network, G s is the intermediate feature map corresponding to the second neural network, n L represents the number of intermediate feature maps of size L, represents the network parameters of the first neural network corresponding to the i-th feature map of size L, represents the network parameters of the second neural network corresponding to the i-th feature map of size L, n represents the total number of clothing categories, y i represents the probability value of the i-th category in the real clothing category, represents the probability value of the first neural network predicting the i-th category, Represents the probability value of the second neural network predicting the i-th category.

[0147] During the training process, the first neural network transfers the attention weights on channel dimensions of different scales to the second neural network through distillation loss, so that the difference between the intermediate feature maps extracted by the second neural network and the intermediate feature maps extracted by the first neural network continues to narrow, thereby enabling the first neural network to guide the second neural network to extract image features that are more representative for clothing classification.

[0148] The classification loss includes the loss between the first predicted clothing category by the first neural network and the actual clothing category, the loss between the second predicted clothing category by the second neural network and the actual clothing category, and the loss between the first predicted clothing category and the second predicted clothing category. Because the first neural network can learn textual information, the first predicted clothing category is able to continuously approximate the actual clothing category. During training, based on the backpropagation of the loss, the loss continuously decreases, and the second predicted clothing category also becomes increasingly close to the actual clothing category.

[0149] Under the constraints of the loss function, the first neural network guides the second neural network to optimize towards a decreasing loss value. Under the guidance of the first neural network, the second neural network can learn image features that are more representative of clothing classification, thereby improving classification accuracy. When the loss fluctuates within a certain range or reaches a minimum, the first and second neural networks converge. The converged neural networks are able to extract image features that are more representative of clothing classification and accurately predict clothing categories. Therefore, the converged second neural network is used as the clothing classification model.

[0150] It's understandable that the lower depth of the second neural network means the clothing classification model requires less computing power to run, resulting in faster predictions. Furthermore, because the clothing classification model is trained under the guidance of the first neural network, it doesn't require text input to achieve the same or near-consistent classification accuracy as the first neural network.

[0151] After training a clothing classification model using the method for training a clothing classification model provided in the embodiments of this application, the clothing classification model can be applied to clothing classification. The clothing classification method provided in the embodiments of this application can be implemented by various types of electronic devices with computing processing capabilities, such as smart terminals and servers.

[0152] The following describes the clothing classification method provided in the embodiment of the present application in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0153] The clothing classification method uses a clothing classification model to identify clothing categories of clothing images to be classified to obtain corresponding clothing categories. The clothing classification model can be the converged first neural network or the converged second neural network in any of the above embodiments.

[0154] When the clothing classification model is the first neural network after convergence, the clothing image to be classified and the corresponding text information are input into the clothing classification model to obtain the clothing category. The clothing image to be classified includes clothes. The classification assistant (application software) built into the terminal (such as a smart phone) obtains the clothing image to be classified and the text information, wherein the clothing image to be classified can be taken by the terminal or input by the user. The text information can also be a clothing tag taken by the terminal, or crawled from a web page by the terminal, or input by the user. The clothing classification model built into the terminal extracts features from the clothing image to be classified, encodes the text information, fuses the encoded text encoding features with the extracted feature map, performs full connection and classification processing on the obtained fused feature map, and obtains the predicted clothing category.

[0155] It can be understood that the clothing classification model is trained by the method of training the clothing classification model in the above embodiment, and has the same structure and function as the clothing classification model in the above embodiment (the first neural network after convergence), which will not be described in detail here.

[0156] When the clothing classification model is a converged second neural network, the clothing image to be classified is input into the clothing classification model to obtain the clothing category. The clothing image to be classified includes clothing. A classification assistant (application software) built into a terminal (e.g., a smartphone) obtains the clothing image to be classified and text information. The clothing image to be classified can be captured by the terminal or input by the user. The clothing classification model built into the terminal performs feature extraction on the clothing image to be classified, then performs full connection and classification processing to obtain a predicted clothing category.

[0157] It can be understood that the clothing classification model is trained by the method of training the clothing classification model in the above embodiment, and has the same structure and function as the clothing classification model in the above embodiment (the second neural network after convergence), which will not be described in detail here.

[0158] The present application also provides a computer-readable storage medium, which stores computer-executable instructions for enabling an electronic device to execute the method for training a clothing classification model provided in the present application, for example, Figure 3-8 The method for training a clothing classification model shown, or the clothing classification method provided in an embodiment of the present application.

[0159] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0160] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0161] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0162] As an example, executable instructions may be deployed to be executed on one computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed in multiple locations and interconnected by a communication network.

[0163] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for training a clothing classification model or the clothing classification method as described in the aforementioned embodiment.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Based on the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present application as described above. For the sake of simplicity, they are not provided in detail. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a clothing classification model, characterized in that: include: Obtaining a training set, the training set comprising a plurality of clothing images and text information corresponding to the plurality of clothing images, the text information comprising clothing descriptions, and each of the clothing images being respectively annotated with a real clothing category; Encode each of the text information to obtain a vector list and text encoding features; Inputting the clothing image and the text encoding feature corresponding to the clothing image into a first neural network to obtain a first predicted clothing category corresponding to the clothing image; adjusting parameters of the first neural network according to a difference between a first predicted clothing category corresponding to the training set and a real clothing category corresponding to the training set until convergence, and using the converged first neural network as the clothing classification model; The first neural network includes a convolution module, a fusion module, a fully connected layer and a classification layer that are cascaded in sequence; Inputting the clothing image and the text encoding feature corresponding to the clothing image into the first neural network to obtain a first predicted clothing category corresponding to the clothing image includes: Inputting the clothing image into the convolution module for downsampling feature extraction to obtain a clothing feature map; The clothing feature map and the text encoding feature are input into the fusion module for feature fusion to obtain a fused feature map; After the fusion feature map passes through the fully connected layer and the classification layer, the first predicted clothing category is output; The fusion module uses the following formula to perform feature fusion on the clothing feature map and the text encoding feature; F(x,e)=α*x+β α=MLP1(e); β=MLP2(e) Wherein, x is the clothing feature map, e is the vector list, α is a text encoding feature corresponding to the clothing image, and β is another text encoding feature corresponding to the clothing image.

2. The method according to claim 1, characterized in that The encoding of each text information to obtain a vector list and text encoding features includes: Using a word embedding algorithm to vectorize each word in the text information to obtain a vector list; Feature extraction is performed on the vector list to obtain the text encoding feature.

3. The method according to claim 2, characterized in that Performing feature extraction on the vector list to obtain the text encoding features includes: A multi-layer perceptron module is used to perform feature extraction on the vector list to obtain the text encoding features.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Inputting the clothing image into a second neural network to obtain a second predicted clothing category corresponding to the clothing image, wherein the depth of the second neural network is less than the depth of the first neural network; Calculating the loss using a loss function, and adjusting the parameters of the first neural network and the second neural network according to the loss until convergence, and using the converged second neural network as the clothing classification model; The loss function includes distillation loss and classification loss, the distillation loss reflects the difference between the intermediate feature map output by the first neural network and the intermediate feature map output by the second neural network, and the classification loss reflects the difference between the first predicted clothing category, the second predicted clothing category and the real clothing category.

5. The method according to claim 4, characterized in that The loss function includes: in, in, is the distillation loss, L t-s is the classification loss, is the intermediate feature map corresponding to the first neural network, G s is the intermediate feature map corresponding to the second neural network, n L represents the number of intermediate feature maps of size L, represents the network parameters of the first neural network corresponding to the i-th feature map of size L, represents the network parameters of the second neural network corresponding to the i-th feature map of size L, n represents the total number of clothing categories, y i represents the probability value of the i-th category in the real clothing category, represents the probability value of the first neural network predicting the i-th category, Represents the probability value of the second neural network predicting the i-th category.

6. A clothing classification method, characterized in that: include: A clothing classification model is used to identify clothing categories of clothing images to be classified to obtain corresponding clothing categories, wherein the clothing classification model is trained using the method for training a clothing classification model according to any one of claims 1 to 5.

7. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the method according to any one of claims 1 to 6.

Images