A multi-modal CoT product catalog correction method

Abstract

The application discloses a kind of multi-modal CoT product catalog correction methods, it is characterized in that, including using the name of product, description, keyword etc. Text information and product picture train product multi-modal fusion model;Through the training optimization of multiple losses of pre-set multi-modal fusion model, the text and picture features of product are fused, and the multi-modal product vector is obtained;Product vector generates product category by OPT generation model;At the same time, the effect of the generated category is optimized by the constructed CoT label.The multi-modal catalog correction model proposed in the application can effectively generate all existing product catalogs without using artificial label information, thereby correcting the generated incorrect product catalogs, improving the precision of product catalog classification, and reducing most of the time consumption.

Description

Technical Field

[0001] This invention relates to the field of e-commerce and also to the field of big data analysis and application technology, specifically a method for revising a multimodal CoT product catalog. Background Technology

[0002] Since the introduction of self-attention models in 2017, significant progress has been made in deep learning for text and image classification, generation, and reasoning tasks. Currently, many pre-trained models based on self-attention mechanisms, such as BERT, ViT, and GPT, have emerged in text, image, and multimodal domains. These pre-trained models have demonstrated superiority and reliability in their respective fields.

[0003] Currently, multimodal tasks can be roughly classified into three categories. The first involves using complex image feature extraction and simple text encoding followed by simple feature fusion for downstream tasks. The drawback of this method is the slow image feature extraction speed, and the text encoding part needs to wait for image processing to complete before fusion. The second method uses simple image extraction and simple text encoding followed by simple feature fusion. This method has a simple model and is fast, but its performance is poor and training is difficult. The third method uses appropriate image feature extraction and text encoders for simple image fusion, with the Clip model being the most commonly used. Although this model performs well in problems such as image classification, its performance is insufficient for multimodal understanding tasks. The main reason for this is the overly simplistic fusion process between image and text features.

[0004] The current product catalog revision is largely studied as a classification task, and the classification methods can be roughly divided into two categories:

[0005] The first category is classification methods based on traditional machine learning, such as Rank-SVM and ML-KNN. These methods require manual feature design to train the classifier, and the quality of the model depends on the quality of the manual design. It is difficult to manually extract the corresponding feature information from the text and image information of a product.

[0006] The second category is classification methods based on deep learning, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory Networks (LSTMs). These early deep learning models performed well in specific domains. Conventional deep learning models rely heavily on the accuracy of the labels on the training data, which typically incurs significant human overhead. Summary of the Invention

[0007] The technical problem to be solved by this invention is to overcome the shortcomings of existing technologies and the deficiencies of current product catalog correction technologies. To solve the above-mentioned technical problem, this invention provides a multimodal CoT product catalog correction method, which combines the text and image features of products to minimize the amount of manually labeled data and improve the efficiency of product catalog correction. The method is characterized by the following steps:

[0008] Step 1: Extract the text and image information contained in the product to form the original text dataset and the original image dataset. For the original text dataset, use the first 6 layers of the Bert base model as the Bert encoder for text encoding. For the original image dataset, use the ViT model as the ViT encoder for image encoding. Use the last 6 layers of the Bert base model to fuse the two modalities to obtain a multimodal fusion model. Train and optimize the multimodal fusion model using various preset losses.

[0009] Step 2: For the multimodal model obtained from training and optimization in Step 1, design classification prompts in the samples and concatenate them with the original text to add them into the text information. Use the alignment layer to align the fusion vector with the input dimension of the OPT model, use the OPT model to generate a product catalog, and optimize the multimodal fusion model by the similarity between the generated product catalog and the real catalog.

[0010] Step 3: Use the trained multimodal fusion model and OPT model to construct a multimodal CoT generation model to generate a catalog of all products in the product library. If the generated product catalog does not match the original product catalog, the product catalog is corrected.

[0011] In step 1, the product's text information includes the product's name, description, keywords, and material information; the product's image information uses the image with the highest similarity score to the product name; and for the multimodal fusion model, Cross-Attention is used instead of the Self-Attention module in the Bert base model.

[0012] Step 1, the training and optimization of the multimodal fusion model, includes:

[0013] Step 1.1, for the multimodal fusion model, the preset loss includes three training tasks: Task 1 is the image-text contrast learning loss (ITC), where before multimodal fusion, text vectors are obtained through the Bert encoder and image vectors are obtained through the ViT encoder, and the matching degree between the text vector and the image vector is calculated to determine whether the product and the image belong to the same product; Task 2 is the mask reconstruction loss on the multimodal encoder (MLM), where the multimodal fusion model is used to fuse text feature vectors and image feature vectors to predict the masked words in the product text; Task 3 is the image-text matching loss (ITM), where the fused multimodal vectors are used to perform binary classification prediction to predict whether the product image and text match.

[0014] Step 1.2: Sum the losses of the three training tasks in Step 1.1, train and optimize the multimodal fusion model to obtain the required multimodal fusion model.

[0015] Step 2 further includes: the alignment and fusion vector and the input dimension of the OPT model are combined using a fully connected layer.

[0016] In step 2, the fully connected layer is an alignment layer, the OPT model is preferably the OPT-6.7B model, and the classification prompts include the classification reasons and features of positive samples.

[0017] Step 2 includes:

[0018] Step 2.1: Input the product text and product images with the added category prompts into the multimodal coding model proposed in Step 3 to obtain the corresponding product vectors. After inputting them into the alignment layer, a 4096-dimensional product vector is obtained. Finally, it is input into the OPT generation model to generate the product category.

[0019] Step 2.2: Calculate the similarity between the product catalog generated using the OPT model and the product catalogs with candidate tags. The cosine similarity formula is as follows:

[0020]

[0021] G represents the vector generated by the model, and L represents the label vector. After calculating the similarity between the generated vector and each label, a loss function is used to maximize the similarity between correct vector pairs and suppress the similarity between vector pairs with different labels. The loss function is:

[0022]

[0023] S iThis represents the similarity between positive vector pairs. By reducing the similarity between correct labels and increasing the similarity between incorrect labels, the model can be optimized. The parameters of the multimodal fusion model and the OPT model are frozen, and the parameters of the upgraded fully connected layer are optimized.

[0024] In step 3, the correction of the product catalog includes: calculating the similarity between the generated product catalog and the real catalog; if the similarity is lower than a preset threshold, the generated product catalog is used to replace it.

[0025] Step 1.2 includes the Bert encoder converting the input text T into an embedded sequence {W}. cls ,W1,...,W N}, where W cls The text information is cls, where N is the length of the text sequence. For image information, the ViT model preferably uses the ViT-B / 16 model as the ViT encoder, cutting the image into 16*16 blocks, totaling 196 blocks, and encoding them into {I cls ,I1,...,I 196}, where I cls This is the image CLS classification module; text information is encoded into a 768-dimensional vector by the BERT encoder, and image information is encoded into a 768-dimensional vector by the ViT encoder, as shown in the following formula:

[0026]

[0027] The two modalities are fused using the Bertbase model to obtain a 768-dimensional fusion vector S1, as shown in the following formula:

[0028]

[0029] In step 1.3, the image-text contrastive learning loss is used to learn a better single-modal representation before fusing the vectors, by using image I cls With text W cls The linear transformation maps to 256 dimensions. For each image vector and text vector, the similarity between the transposed image vector and the text vector is calculated to obtain s(I, W), and the similarity between the transposed text vector and the image vector is calculated to obtain s(W, I).

[0030] The image-text contrast loss Litc is calculated using the following formula:

[0031]

[0032] Where E represents variance, H represents cross-entropy, and yi2t(I) and yt2i(T) are 0 and 1 labels, used to indicate whether the actual image and text match.

[0033] The image-text matching loss is used to predict whether an image-text pair matches, and uses the multimodal fusion S... cls Vectors, as the fused representation of image-text pairs, are used for binary classification p through a fully connected (FC) layer followed by a softmax layer. itm The ITM loss Litm is calculated using the following formula:

[0034]

[0035] Among them, y itm These are genuine image and text binary categories;

[0036] The mask reconstruction loss includes randomly masking words with a 15% probability and replacing them with specially marked words, where T' represents the masked text, p msk (I,T') represents the predicted probability of the masked word. The minimum cross-entropy loss Lmlm is calculated using the following formula:

[0037]

[0038] Among them, y msk It is a one-hot word distribution, with a probability of 1 for the true word element; for each image in a mini-batch, a negative sample text is extracted from the same batch according to the contrast similarity distribution, where texts more similar to the image have a higher sampling chance. Similarly, a negative sample image is sampled for each text. The formula L for the summation of the loss is:

[0039] L = L itc +αL mlm +βL itm

[0040] Two hyperparameters, α and β, are used to control the ratio of the three types of losses and optimize the training of the model.

[0041] In step 3, replacing the product catalog with the generated product catalog includes: calculating the cosine similarity with the remaining product catalogs and selecting the product catalog with the highest similarity for replacement.

[0042] The beneficial effects achieved by this invention are: while using a pre-trained model with a self-attention mechanism to improve the product catalog correction effect, the CoT optimization model solves the dependence on manually labeled data, greatly reducing the time cost. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a method flow of an exemplary embodiment of the present invention;

[0044] Figure 2This is a flowchart illustrating the training loss calculation in an exemplary embodiment of the present invention. Detailed Implementation

[0045] This invention uses a deeper image feature extractor, a shallower text encoder, and multiple self-attention layers for fusion before proceeding to downstream tasks.

[0046] The embodiments of the present invention will be further explained and described below with reference to the accompanying drawings.

[0047] like Figure 1 The method flow shown in this exemplary embodiment of the present invention is a multimodal CoT product catalog revision method based on a self-attention mechanism, comprising the following steps:

[0048] Step 1 represents training the image fusion model. Product images and text are processed using a BERT encoder and a ViT encoder to obtain corresponding image vectors and text vectors. These image and text vectors are then fused using a fusion encoder to obtain a fused vector. Step 2 uses the trained multimodal fusion model to fuse product images and text into a fused product vector. This vector is then upscaled using an alignment layer and input into the product catalog of the OPT generation model. Step 3 uses the trained fusion and generation models to predict and generate a product catalog. The similarity between the generated and original product catalogs is calculated, and the catalog is then corrected accordingly.

[0049] The following is combined with Figure 2 The flowchart of training loss calculation in one exemplary embodiment further illustrates the present invention.

[0050] Step 1: Group the product text and images together, and use a fusion model to combine the two types of information from the product text and images. For example... Figure 2 As shown, product images are encoded into image vectors using a 12-layer ViT-B / 16 model as a ViT encoder, and product text is encoded into text vectors using the first six layers of a Bert-based model as a Bert encoder. Then, the image and text vectors are input into a fusion encoder consisting of the last six layers of a Bert-based model to obtain the fused product vector. To optimize the fusion model, we combine the Image-Text Contrast Learning Loss (ITC), Image-Text Matching Loss (ITM), and Mask Reconstruction Loss (MLM) functions for optimization.

[0051] Further expansion of step 1 yields:

[0052] Step 1.1: The Transformer model consists of an encoder and a decoder. The BERT model is trained using a self-attention encoder. The BERT base model uses a 12-layer self-attention encoder, with each layer consisting of a self-attention module and an MLP module. The BERT model is widely used in natural language processing. The ViT model and vision transformer are also trained based on the Transformer encoder. While the self-attention mechanism of the Transformer model is popular in natural language processing, the emergence of ViT for image processing has broken the dominance of convolutional neural networks in the field of vision. The ViT-B / 16 model is also a basic model of ViT, consisting of a 12-layer Transformer encoder module, which slices two-dimensional image information and stretches it to one-dimensional space.

[0053] For the fusion model, we initialized it using the last 6 layers of the Bert base model. However, in order to achieve better modality fusion, we replaced the Self-Attention module in the Bert base model with Cross-Attention.

[0054] Step 1.2, the Bert encoder converts the input text T into an embedded sequence {W}. cls ,W1,...,W N}, where W cls This is the cls tag for the product text, where N represents the text length. For product image I, since the ViT-B / 16 model is used, the image will be cut into 16*16 blocks, totaling 196 small blocks. These 196 image blocks are encoded using ViT into {I... cls ,I1,...,I 196}, where I cls It is the image cls classification module.

[0055] A 6-layer BERT encoder outputs a 768-dimensional vector. Similarly, a product image encoded using a ViT encoder will also be encoded into a 768-dimensional vector, as shown in the formula below:

[0056]

[0057] The two modalities are then fused using the last six layers of the BERT structure, resulting in a fused vector of 768 dimensions, as shown in the following formula:

[0058]

[0059] Step 1.3, Image-Text Contrast Learning Loss (ITC), aims to learn a better unimodal representation before fusing the image vectors. This is achieved by using image vectors I... cls With text vector W cls The linear transformation maps to 256 dimensions. For each image and text, we calculate the similarity s(I, W) between the transposed image vector and the text vector, and calculate the similarity s(W, I) between the transposed text vector and the image vector.

[0060] Using image-text similarity and text-image similarity, the image-text comparison loss is calculated using the following formula:

[0061]

[0062] Where yi2t(I) and yt2i(T) represent 0 and 1 labels, indicating whether the image and text match. The probability of a negative sample pair is 0, and the probability of a positive sample pair is 1. H represents the cross-entropy.

[0063] Image-text matching loss (ITM) aims to predict whether a pair of image-text pairs match. We use the S after multimodal fusion. cls Vectors are used as fused representations of image-text pairs, and binary classification p is performed through a fully connected (FC) layer followed by a softmax layer. itm The ITM loss is:

[0064]

[0065] Where y itm These are real image-text binary classification labels, where E represents variance and H represents cross-entropy.

[0066] Masked Reconstruction Loss (MLM) uses an image and contextual text to predict masked terms. We randomly mask terms with a 15% probability and replace them with specially labeled [MASK] terms. Let T' represent the masked text, p msk (I,T') represents the model's predicted probability for the masked term. The MLM formula minimizes the cross-entropy loss:

[0067]

[0068] Where y msk It is a one-hot word distribution, where the probability of a real word is 1.

[0069] For each image in a mini-batch, we extract a negative sample text from the same batch according to the contrast similarity distribution, where texts more similar to the image have a higher sampling chance. Similarly, we also sample a negative sample image for each text. The final complete loss formula is calculated as follows:

[0070] L = L itc +αL mlm +βL itm

[0071] We use two hyperparameters, α and β, to control the ratio of the three losses, preventing the values ​​of the losses from differing too much, thereby better optimizing the training of the model.

[0072] Step 2: Following the steps in Step 1, train the multimodal fusion model. Inputting product images and text information, we obtain multimodal product vectors. Our encoded product vectors have a dimension of 768, while the input vector of the OPT-6.7B model has a dimension of 4096. We need to train a full-connection layer to align the input of the OPT model, using this alignment layer to align the fused vector with the input dimension of the OPT model. The OPT model is an open-source generative model based on the Transformer Decoder from META. It has eight different parameter sizes, ranging from 125M, 350M to 175B. Considering the balance between performance and efficiency, we use the 6.7B model for the generation task.

[0073] Further expansion of step 2 yields:

[0074] Step 2.1 involves inputting the product text and images with added category prompts into the multimodal coding model proposed in Step 3 to obtain the corresponding product vectors. For example, Civic tires belong to the auto parts category, specifically tires and Honda vehicles. Category prompts are these types of hints and features that significantly aid in product classification. Afterward, these are input into the alignment layer to obtain 4096-dimensional product vectors, which are then finally input into the OPT generation model to generate the corresponding product catalog.

[0075] Step 2.2: To optimize the upgraded fully connected layer, we calculate the similarity between the product catalog generated by the OPT generative model and the real catalog of tag candidates. The real catalog is the category directory where the product should belong based on its own characteristics. The cosine similarity formula is as follows:

[0076]

[0077] G represents the vector generated by the model, and L represents the label vector. After calculating the similarity between the generated vectors and each label, a loss function is used to maximize the similarity between correct vector pairs and suppress the similarity between vector pairs with different labels. The loss function is:

[0078]

[0079] S iThis represents the similarity between positive vector pairs. By narrowing the similarity between correct labels and widening the similarity between incorrect labels, the model can be optimized. We freeze the parameters of the multimodal fusion model and the OPT model, and only optimize the parameters of the upgraded fully connected layer.

[0080] In step 2, the manually added product text information (CoT) is vectorized using BERT, and the image information is vectorized using ViT. The latter half of the BERT is then inserted and fused. The fused multimodal features are input into OPT to generate the corresponding product catalog. The similarity loss between the generated product catalog and the original tag catalog is calculated to optimize the generator.

[0081] Step 3: Revise the product catalog. As mentioned in Step 2, if the similarity between the product categories generated using OPT and the actual product categories is lower than a preset threshold, we need to revise the product categories. This is done by calculating the cosine similarity with the remaining product catalogs and selecting the catalog with the highest similarity to best match the product text and images.

[0082] To address the inaccuracies of single-modal product classification using only product text or images, this study employs a multimodal model for correction. It utilizes a BERT encoder and a ViT encoder to vectorize product text and images, followed by a fusion model to integrate the text and image modalities. A combination of ITC, ITM, and MLM losses is used, with α and β controlling the loss ratio. Optimization of parameters through scaling up and down improves the training of the fusion model. Product categories are generated using the OPT model, avoiding the use of a classification model to prevent unnecessary corrections due to similar product categories. CoT is used to expand product text, aiming to generate a more accurate product catalog. Simultaneously, an alignment layer is used to align the dimensions of the OPT generation model. For the generated product catalog, similarity is calculated against the original product catalog for more accurate correction.

[0083] The beneficial effects achieved by this invention are: while using a pre-trained model with a self-attention mechanism to improve the product catalog correction effect, the CoT optimization model solves the dependence on manually labeled data, greatly reducing the time cost of manual work.

[0084] This invention may also have other various embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, and these corresponding changes and modifications should all fall within the protection scope of the appended claims.

[0085] The above embodiments are not intended to limit the present invention in any way. Any other improvements and applications made to the above embodiments by equivalent transformations shall fall within the protection scope of the present invention.

Claims

1. A method for revising a multimodal CoT product catalog, characterized in that, Includes the following steps: Step 1: Extract the text and image information contained in the product to form the original text dataset and the original image dataset. For the original text dataset, use the first 6 layers of the Bert base model as the Bert encoder for text encoding. For the original image dataset, use the ViT model as the ViT encoder for image encoding. Use the last 6 layers of the Bert base model to fuse the two modalities to obtain a multimodal fusion model. Train and optimize the multimodal fusion model using various preset losses. The product's text information includes the product name, product description, product keywords, and material information; the product's image information uses the image with the highest similarity score to the product name; for the multimodal fusion model, Cross-Attention is used instead of the Self-Attention module in the Bert base model; Optimization of multimodal fusion model training includes: Step 1.1, for the multimodal fusion model, the preset multiple losses include three training tasks: Task 1 is the image-text comparison learning loss. Before multimodal fusion, text vectors are obtained through the Bert encoder, and image vectors are obtained through the ViT encoder. The matching degree between the text vector and the image vector is calculated to determine whether the product and the image belong to the same product. Task 2 is the mask reconstruction loss on the multimodal encoder. The multimodal fusion model is used to fuse text feature vectors and image feature vectors to predict the masked words in the product text. Task 3 is the image-text matching loss. The fused multimodal vectors are used to perform binary classification prediction to predict whether the product image and text match. Step 1.2: Sum the losses of the three training tasks in Step 1.1, train and optimize the multimodal fusion model to obtain the required multimodal fusion model; Step 1.2 includes the Bert encoder converting the input text T into an embedded sequence {W}. cls , W1, ...,W N }, where W cls The cls is the marker for text information, and N is the length of the text sequence. For image information, the ViT model is the ViT-B / 16 model; the image is cut into 16 segments.

16. A total of 256 image blocks, encoded into {I cls , I1, ..., I 256 }, where I cls This is the image CLS classification module; text information is encoded into a 768-dimensional vector by the BERT encoder, and image information is encoded into a 768-dimensional vector by the ViT encoder, as shown in the following formula: ; The two modalities are fused using the Bertbase model to obtain a 768-dimensional fusion vector S1, as shown in the following formula: ; In step 1.2, the image-text contrast learning loss is used to learn a better single-modal representation before fusing the vectors, by using image I cls With text W cls The linear transformation maps to 256 dimensions. For each image vector and text vector, the similarity between the transposed image vector and the text vector is calculated to obtain s(I, W), and the similarity between the transposed text vector and the image vector is calculated to obtain s(W, I). Calculate the image-text contrast loss L using the following formula. itc for: ; Where E represents variance, H represents cross-entropy, and y i2t (I) and y t2i (T) is a 0 or 1 label used to indicate whether the actual image and text match; The image-text matching loss is used to predict whether an image-text pair matches, and uses the multimodal fusion S... cls Vectors are used as the fused representation of image-text pairs. They are then processed through a fully connected layer followed by a softmax layer for binary classification (p). itm Calculate the ITM loss L using the following formula. itm for: ; Among them, y itm These are genuine image and text binary categories; The mask reconstruction loss includes randomly masking words with a 15% probability and replacing them with specially marked words. Let T' represent the masked text, p msk (I,T') represents the predicted probability of the masked word. The cross-entropy loss L is calculated using the following formula. mlm for ; Among them, y msk It is a one-hot word distribution, with a probability of 1 for the true word element; for each image in a mini-batch, a negative sample text is extracted from the same batch according to the contrast similarity distribution, where texts more similar to the image have a higher sampling chance. Similarly, a negative sample image is sampled for each text. The formula L for the summation of the loss is: ; Two hyperparameters, α and β, are used to control the ratio of the three types of losses and optimize the training of the model. Step 2: For the multimodal model obtained from training and optimization in Step 1, design classification prompts in the samples and concatenate them with the original text to add them into the text information. Use the alignment layer to align the fusion vector with the input dimension of the OPT model, use the OPT model to generate a product catalog, and optimize the multimodal fusion model by the similarity between the generated product catalog and the real catalog. Step 3: Use the trained multimodal fusion model and OPT model to construct a multimodal CoT generation model to generate a catalog of all products in the product library. If the generated product catalog does not match the original product catalog, the product catalog is corrected.

2. The multimodal CoT product catalog revision method as described in claim 1, characterized in that, Step 2 further includes: the alignment and fusion vector and the input dimension of the OPT model are combined using a fully connected layer.

3. The multimodal CoT product catalog revision method as described in claim 2, characterized in that, In step 2, the fully connected layer is an alignment layer, the OPT model is the OPT-6.7B model, and the classification hints include the classification reasons and features of positive samples.

4. The multimodal CoT product catalog revision method as described in claim 3, characterized in that, Step 2 includes: Step 2.1: Input the product text and product images with the added category prompts into the multimodal coding model proposed in Step 1 to obtain the corresponding product vectors. After inputting them into the alignment layer, a 4096-dimensional product vector is obtained. Finally, it is input into the OPT generation model to generate the product category. Step 2.2: Calculate the similarity between the product catalog generated using the OPT model and the product catalogs with candidate tags. The cosine similarity formula is as follows: ; G represents the vector generated by the model, and L represents the label vector. After calculating the similarity between the generated vector and each label, a loss function is used to maximize the similarity between correct vector pairs and suppress the similarity between vector pairs with different labels. The loss function is: ; S i This represents the similarity between positive vector pairs. By reducing the similarity between correct labels and increasing the similarity between incorrect labels, the model can be optimized. The parameters of the multimodal fusion model and the OPT model are frozen, and the parameters of the upgraded fully connected layer are optimized.

Images