E-commerce category fine-grained mining method based on multi-modal pre-training model
By using image segmentation and text enhancement methods based on multimodal pre-trained models, feature optimization is performed on e-commerce datasets, which solves the problem of insufficient fine-grained classification in e-commerce datasets and improves classification accuracy and dataset utilization efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2024-11-12
- Publication Date
- 2026-05-12
AI Technical Summary
Existing e-commerce datasets are insufficient to meet the needs of fine-grained classification, especially when classifying and searching products, where the dataset richness and hierarchy are low. Furthermore, multimodal pre-trained models suffer from background interference and insufficient textual information in e-commerce data, which affects classification performance.
We employ a multimodal pre-trained model-based approach, using a two-stage process of image segmentation enhancement and text enhancement, combined with CLIP and RAM large models, to optimize image and text features on the e-commerce dataset. We utilize InfoNCE loss to evaluate differences, generate reasonable labels, and perform fine-grained segmentation.
It improves the classification ability of multimodal pre-trained models on e-commerce datasets, reduces inter-class errors, realizes fine-grained data mining and information enhancement of e-commerce data, and improves classification accuracy.
Smart Images
Figure CN122023786A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to fine-grained mining of e-commerce categories, and in particular to a method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, belonging to the field of computer technology. Background Technology
[0002] In recent years, with the development of artificial intelligence-related technologies, machine learning and its related applications have achieved significant results in areas such as speech recognition, computer vision, and natural language processing. Among these, classifying objects based on descriptive information is a fundamental and crucial task in machine learning. Generally, classification can be categorized into coarse-grained classification, fine-grained classification, and instance-level classification based on the features of the objects themselves. For example, coarse-grained classification typically involves broad categories such as dogs, cars, and flowers, characterized by significant differences between classes. Fine-grained classification builds upon coarse-grained classification, further refining the subcategories within broad classes, such as dog breeds, car models, and flower varieties. Instance-level classification further distinguishes different individuals, achieving individual classification, such as facial recognition and its applications. With the deepening of artificial intelligence research, various research efforts on coarse-grained and instance sets have progressed rapidly. However, some existing datasets are insufficient to meet the research needs of artificial intelligence algorithms in specific scenarios in modern society, especially in fields like e-commerce where fine-grained classification is urgently needed.
[0003] Early general-purpose object classification datasets, such as COCO and VOC, contained only 80 and 20 categories, respectively. Generally, the granularity of these datasets was concentrated on broad categories like dogs, cars, and flowers. This made it difficult to meet the product classification needs in e-commerce, such as brand classification or product feature classification. Early product datasets were constructed similarly to general-purpose object datasets, often based on retail store scenarios. They contained relatively few product types and numbers, and the dataset richness and hierarchy required for product classification and searching were low. Therefore, coarse-grained classification could meet the needs of applied research in this scenario. With the development of the internet and information technology, e-commerce has grown exponentially, and the types and numbers of products have increased dramatically. Early classification levels are increasingly insufficient for practical applications, and the demand for fine-grained classification in the e-commerce industry is becoming increasingly urgent. Therefore, constructing fine-grained datasets specifically for e-commerce is becoming increasingly important for new scenarios.
[0004] Building an e-commerce dataset from scratch requires a series of complex processes, including data collection and cleaning. Especially considering the scale required for e-commerce datasets, this work consumes considerable time and manpower. Furthermore, working on a completely new dataset hinders continuity and necessitates higher algorithm design and training costs. Therefore, designing and utilizing relevant methods to fine-grainedly upgrade existing datasets can save costs and reduce the difficulty of application in downstream tasks. With the rapid development of internet technology, the information carriers of goods have become more diversified. Images, text, audio, and video are all media that can carry information. By delving deeper into the relationships and information between modalities, multimodal and related technologies can be used to design methods for further fine-grained category mining of existing e-commerce datasets, achieving the upgrade and expansion of existing datasets. This paper combines the multimodal pre-trained model CLIP with fine-grained category mining. An e-commerce multimodal pre-trained model trained on the large-scale e-commerce dataset MEP-3M learns semantically sound product image representations, and then transfers the multimodal pre-trained model to the fine-grained category mining work in e-commerce. This allows us to leverage knowledge from fields such as deep learning to further research and develop fine-grained classification datasets. Summary of the Invention
[0005] The technical problem to be solved by this invention is to provide a method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, thus providing a more accurate solution to the problem of fine-grained mining of e-commerce categories.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0007] A fine-grained method for e-commerce category mining based on a multimodal pre-trained model includes the following steps:
[0008] Step 1: Image segmentation is achieved based on the multimodal pre-trained model transfer method. The semantic segmentation model is trained using the "pre-training + fine-tuning" paradigm.
[0009] Step 2: Fine-tune the multimodal pre-trained model based on image segmentation enhancement. Perform image segmentation enhancement on the multimodal dataset MEP-3M, and fine-tune the multimodal pre-trained model CLIP on the MEP-3M-SEG dataset.
[0010] Step 3: A two-stage text fusion enhancement method based on a large model is used to process and enhance the text modality data in the e-commerce dataset, addressing the issues of missing or insufficient text modality data.
[0011] Step 4: Based on a two-stage text fusion enhancement method using a large model, text enhancement is performed on the multimodal dataset MEP-3M to obtain the text-enhanced multimodal dataset MEP-3M-TEXT. The multimodal pre-trained model CLIP is then fine-tuned on the MEP-3M-TEXT dataset.
[0012] Step 5: Implement fine-grained category partitioning of the dataset, extract and encode features for each modality in the e-commerce dataset; and implement fine-grained category partitioning based on feature encoding.
[0013] Step 6: Optimize category labeling. Utilize the RAM large model to generate labels for the text and image information from the new subcategories derived from the initial class division. Set an optimization function to find the most reasonable specific feature labels as new labels for category mining.
[0014] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:
[0015] 2-1. After training the semantic segmentation model using the "pre-training + fine-tuning" paradigm, image segmentation enhancement is performed on the multimodal dataset MEP-3M to obtain the multimodal dataset MEP-3M-SEG with image segmentation enhancement.
[0016] 2-2. Align the text labels of the e-commerce dataset. Compare and align the text modal information in the original multimodal product dataset with the information generated in the first step. Automatically filter the generated text labels based on their reasonableness and send the reasonable results to the subsequent steps for overall generation.
[0017] 2-3. Fine-tune the multimodal pre-trained model CLIP on the MEP-3M-SEG dataset.
[0018] 2-4. CLIP pre-training uses the InfoNCE loss method. It replaces the negative sample single-classification method with a negative sample multi-classification method, which can evaluate the degree of difference between the true and predicted values. Assume a given text set K = {k0, ..., k} N-1} and an image q, containing a positive sample (q, k) + Given N-1 negative samples, the InfoNCE loss is calculated as follows:
[0019]
[0020] Where F(·) and G(·) represent the image encoder and text encoder, respectively, which can output image features and text features, and τ is the temperature coefficient. The magnitude of τ can control the model's ability to distinguish negative samples.
[0021] As a preferred embodiment of the present invention, the specific process of step 3 is as follows:
[0022] 3-1. First, use a large model RAM to generate a basic label description for the product image, which supplements the e-commerce dataset with the image information of the product in the text modality. Specifically, it can include the color, shape and some feature information of the product image in the image.
[0023] 3-2. Align the text labels of the e-commerce dataset. Compare and align the text modal information in the original multimodal product dataset with the information generated in the first step. Automatically filter the generated text labels based on their reasonableness and send the reasonable results to the subsequent steps for overall generation.
[0024] 3-3. The obtained text information, including the original information of the sample text and the generated labels obtained in the first step, is input into the ChatGLM large model in an equivalent form to supplement the peripheral information of the product labels, expand the semantics of the text, and improve the utilization efficiency of the dataset in the text modality.
[0025] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:
[0026] 1. This invention addresses the interference information such as background in image sets of e-commerce data, which exists in fine-grained data mining for e-commerce categories. It designs a two-stage multimodal pre-training algorithm based on image segmentation enhancement for e-commerce data. The algorithm trains a corresponding multimodal pre-trained model based on the segmented and enhanced product images, improving the model's fine-grained mining capability. Furthermore, considering the lack of textual information features in existing e-commerce datasets for individual product samples, this invention designs a method using a large language model to enhance textual information features. Through enhanced textual information features, the classification ability of the multimodal pre-trained model on e-commerce datasets is improved.
[0027] 2. This invention pre-trains a multimodal model based on image segmentation-enhanced e-commerce data and text-enhanced e-commerce data. Through methods such as clustering, it reduces the inter-class error in fine-grained product categories, thereby realizing fine-grained data mining of e-commerce data. Attached Figure Description
[0028] Figure 1 This is a flowchart of the fine-grained mining method for e-commerce categories based on a multimodal pre-trained model, as described in this invention.
[0029] Figure 2 This is a schematic diagram of a multimodal pre-trained model fine-tuning method based on image segmentation enhancement.
[0030] Figure 3 This is a flowchart of a text-enhanced multimodal pre-trained model algorithm.
[0031] Figure 4 This is the workflow of a fine-grained product category mining algorithm based on a multimodal pre-trained model. Detailed Implementation
[0032] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0033] Background information interference exists in e-commerce image datasets. This interference affects the ability and effectiveness of multimodal pre-trained models in extracting product image information, thus reducing the feature matching and semantic alignment between image-text pairs in the multimodal pre-trained models, and impacting their performance in downstream tasks such as linear classification and image-text retrieval. Furthermore, e-commerce text datasets lack sufficient text information. Existing e-commerce datasets generally store product information primarily in the form of images, with relatively less text and other modal information. This leads to insufficient utilization of textual feature information in multimodal pre-trained models based on product image-text pairs when aligning features of different modalities for the same product. This, in turn, affects the performance of multimodal pre-trained models in downstream tasks. Currently, some existing multimodal datasets lack sufficient fine-grained product data. In e-commerce datasets such as MEP-3M and product1M, product data is classified in a coarse-grained manner, resulting in large inter-class discrepancies, low efficiency in utilizing image and text information, and difficulties in further fine-grained classification. There is an urgent need for suitable methods to improve the utilization efficiency of existing data and to complete the fine-grained segmentation and information enhancement of e-commerce data. Based on this problem, this invention proposes a fine-grained mining method for e-commerce categories based on a multimodal pre-trained model, which mines fine-grained content of e-commerce categories using this model.
[0034] Combination Figure 1 As shown, the present invention provides a fine-grained method for e-commerce category mining based on a multimodal pre-trained model, comprising the following steps:
[0035] 1. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 1 is as follows:
[0036] (1) The weights of the CLIP pre-trained image encoder are used to initialize the parameters of the backbone network of the image segmentation model, so that the image features of e-commerce data can be extracted better after the initialization is completed.
[0037] (2) During network training, the backbone network is frozen, the feature extraction part remains unchanged, and the decoder part is fine-tuned.
[0038] 2. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 2 is as follows:
[0039] (1) After training the semantic segmentation model using the "pre-training + fine-tuning" paradigm, image segmentation enhancement is performed on the multimodal dataset MEP-3M to obtain the multimodal dataset MEP-3M-SEG with image segmentation enhancement.
[0040] (2) Align the text labels of the e-commerce dataset, compare and align the text modal information in the original multimodal product dataset with the information generated in the first step, automatically filter the generated text labels based on their reasonableness, and send the reasonable results to the subsequent steps for overall generation.
[0041] (3) Fine-tuning the multimodal pre-trained model CLIP on the MEP-3M-SEG dataset.
[0042] (4) CLIP pre-training uses the InfoNCE loss method. It replaces the negative sample single-classification method with a negative sample multi-classification method, which can assess the degree of difference between the true and predicted values. Assume a given text set K = {k0, ..., k} N-1} and an image q, containing a positive sample (q, k) + Given N-1 negative samples, the InfoNCE loss is calculated as follows:
[0043]
[0044] Where F(·) and G(·) represent the image encoder and text encoder, respectively, which can output image features and text features, and τ is the temperature coefficient. The magnitude of τ can control the model's ability to distinguish negative samples.
[0045] 3. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 3 is as follows:
[0046] (1) First, a basic label description of the product image is generated using a large model RAM to supplement the product image information in the text modality of the e-commerce dataset. Specifically, it can include the product color, shape and some feature information about the product image in the image.
[0047] (2) Align the text labels of the e-commerce dataset, compare and align the text modal information in the original multimodal product dataset with the information generated in the first step, automatically filter the generated text labels based on their reasonableness, and send the reasonable results to the subsequent steps for overall generation.
[0048] (3) Input the obtained text information, including the original information of the sample text and the generated labels obtained in the first step, into the ChatGLM large model in an equivalent form to supplement the peripheral information of the product labels, expand the semantics of the text, and improve the utilization efficiency of the dataset in the text modality.
[0049] 4. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 4 is as follows:
[0050] (1) First, text enhancement is achieved by using the text fusion enhancement method to utilize the image data and text data in the multimodal e-commerce dataset;
[0051] (2) Use the enhanced text data and the initial image data to fine-tune the multimodal pre-trained model.
[0052] 5. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 5 is as follows:
[0053] (1) Use a self-supervised method to classify the feature codes of commodities;
[0054] (2) Classify according to image features and text features respectively, and then align according to the classification results to achieve fine-grained subclass division.
[0055] 6. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that the specific process of step 6 is as follows:
[0056] (1) Use the RAM large model to generate labels for the text and image information of the new subclasses divided from the initial class;
[0057] (2) Set up an optimization function to find the most reasonable unique feature label as the new label for category mining. The selection requirements for sub-category labels include two points: one is that the label information of products in the same sub-category should be as consistent as possible, and the other is that it should be as different as possible from other labels in the major category to which the sub-category belongs.
Claims
1. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, Includes the following steps: Step 1: Image segmentation is achieved based on the multimodal pre-trained model transfer method. The semantic segmentation model is trained using the "pre-training + fine-tuning" paradigm. Step 2: Fine-tune the multimodal pre-trained model based on image segmentation enhancement. Perform image segmentation enhancement on the multimodal dataset MEP-3M, and fine-tune the multimodal pre-trained model CLIP on the MEP-3M-SEG dataset. Step 3: A two-stage text fusion enhancement method based on a large model is used to process and enhance the text modality data in the e-commerce dataset, addressing the issues of missing or insufficient text modality data. Step 4: Based on a two-stage text fusion enhancement method using a large model, text enhancement is performed on the multimodal dataset MEP-3M to obtain the text-enhanced multimodal dataset MEP-3M-TEXT. The multimodal pre-trained model CLIP is then fine-tuned on the MEP-3M-TEXT dataset. Step 5: Implement fine-grained classification of the dataset categories, and perform feature extraction and feature encoding for each modality dataset in the e-commerce dataset; Fine-grained category segmentation is achieved based on feature encoding; Step 6: Optimize category labeling. Utilize the RAM large model to generate labels for the text and image information from the new subcategories derived from the initial class division. Set an optimization function to find the most reasonable specific feature labels as new labels for category mining.
2. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, The specific process of step 1 is as follows: 1-1. The weights of the CLIP pre-trained image encoder are used to initialize the parameters of the backbone network of the image segmentation model, so that the image features of e-commerce data can be extracted better after the initialization is completed. 1-2. During network training, the backbone network is frozen, the feature extraction part remains unchanged, and the decoder part is fine-tuned.
3. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, The specific process of step 2 is as follows: 2-1. After training the semantic segmentation model using the "pre-training + fine-tuning" paradigm, image segmentation enhancement is performed on the multimodal dataset MEP-3M to obtain the multimodal dataset MEP-3M-SEG with image segmentation enhancement. 2-2. Align the text labels of the e-commerce dataset. Compare and align the text modal information in the original multimodal product dataset with the information generated in the first step. Automatically filter the generated text labels based on their reasonableness and send the reasonable results to the subsequent steps for overall generation. 2-3. Fine-tune the multimodal pre-trained model CLIP on the MEP-3M-SEG dataset. 2-4. CLIP pre-training uses the InfoNCE loss method. Using a negative sample multi-class classification method instead of a negative sample single-class classification method can assess the degree of difference between the true and predicted values. Assume a given text set K = {k0, ..., k} N-1 } and an image q, containing a positive sample (q, k) + Given N-1 negative samples, the InfoNCE loss is calculated as follows: Where F(·) and G(·) represent the image encoder and text encoder, respectively, which can output image features and text features, and τ is the temperature coefficient. The magnitude of τ can control the model's ability to distinguish negative samples.
4. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, The specific process of step 3 is as follows: 3-1. First, use a large model RAM to generate a basic label description for the product image, which supplements the e-commerce dataset with the image information of the product in the text modality. Specifically, it can include the color, shape and some feature information of the product image in the image. 3-2. Align the text labels of the e-commerce dataset. Compare and align the text modal information in the original multimodal product dataset with the information generated in the first step. Automatically filter the generated text labels based on their reasonableness and send the reasonable results to the subsequent steps for overall generation. 3-3. The obtained text information, including the original information of the sample text and the generated labels obtained in the first step, is input into the ChatGLM large model in an equivalent form to supplement the peripheral information of the product labels, expand the semantics of the text, and improve the utilization efficiency of the dataset in the text modality.
5. A method for fine-grained category mining in e-commerce based on a multimodal pre-trained model, characterized in that, The specific process of step 4 is as follows: 4-1. First, use text fusion enhancement methods to achieve text enhancement using image and text data in the multimodal e-commerce dataset; 4-2. Use the enhanced text data and the initial image data to fine-tune the multimodal pre-trained model.
6. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, The specific process of step 5 is as follows: 5-1. Use a self-supervised method to classify the feature codes of goods; 5-2. Classify according to image features and text features respectively, and then align according to the classification results to achieve fine-grained subclass division.
7. A method for fine-grained mining of e-commerce categories based on a multimodal pre-trained model, characterized in that, The specific process of step 6 is as follows: 6-1. Use the RAM large model to generate labels for the text and image information of the new subclasses divided from the initial class; 6-2. Set up an optimization function to find the most reasonable unique feature tags as new tags for category mining. The selection requirements for subcategory tags include two points: firstly, the tag information of products in the same subcategory should be as consistent as possible; secondly, the tags should be as different as possible from other tags in the main category to which the subcategory belongs.