Commodity feature generation model, commodity identification and clustering method and commodity management system
By building a product feature generation model on an e-commerce platform, combining user information and large language models, the identification and management problems caused by differences in product title description are solved, and efficient product clustering and accurate feature generation are achieved.
Patent Information
- Application Number
- CN202411511415.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-25
AI Technical Summary
Due to the differences in product title descriptions, e-commerce platforms are difficult to accurately identify and manage the same or similar products, and the existing technology has high computing resources and low processing efficiency.
A large language model and machine learning algorithm are used to build a product feature generation model, and combined with user information for training. Product feature vectors are generated through text embedding vectors, entity eigenvectors, word embedding vectors and association matrix, and high-frequency products are filtered as verification sets to optimize model training.
It improves the clustering accuracy of the product feature generation model, reduces the computing power requirements for model training and operation, and improves processing efficiency.
Smart Images
Figure SMS_1 
Figure SMS_2 
Figure SMS_3
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of commodity management on an e-commerce platform, and in particular to a commodity feature generation model, a commodity identification and clustering method, and a commodity management system. Background Art
[0002] On comprehensive e-commerce platforms, there are a large number of products that have different names but are essentially the same product due to differences in title descriptions. For example: "Brand A Special Pure Milk Classic 250ml*12 Boxes Whole Case" and "Brand A Special Pure Milk 250mx12 Boxes*1 Case"; this name difference comes from the product titles that merchants can edit by themselves, which are not only different in terms of specification descriptions and packaging information, but can also contain different advertising slogans, product introductions, etc. This makes it difficult for e-commerce platforms to distinguish whether different product title descriptions represent the same or similar products, making it difficult to accurately display, push or manage products on e-commerce platforms.
[0003] Traditional methods for identifying products with the same name mainly rely on text segmentation and similarity calculation between titles, but it is difficult to achieve high-accuracy recognition when faced with complex product titles and diverse product information.
[0004] On the other hand, the leading e-commerce platforms often use comprehensive title, product details, user reviews and other text information, product diagrams, introduction videos and other pictures or video information to build highly complex similar product recognition models to ensure the recognition and clustering of the same products. However, this approach consumes a lot of computing resources, has low processing efficiency, and relies on the long-term operation and management of the e-commerce platform. Summary of the invention
[0005] The purpose of this application is to solve the problem of large computing resource consumption and low processing efficiency in the commodity management of e-commerce platforms in the prior art. Therefore, this application provides a commodity feature generation model, a commodity identification and clustering method, and a commodity management system. The commodity feature generation model is constructed using a large language model and a machine learning algorithm, and user information is introduced into the training process of the commodity feature generation model. The data feature dimension of the user information is used to make the clustering results of the commodity feature generation model more in line with user needs and improve the clustering accuracy of the commodity feature generation model. At the same time, the commodity feature generation model is mainly based on text operation, and targeted screening and fusion are performed on the original data, so that the computing power requirements during model training and operation are low and the processing efficiency is greatly improved.
[0006] The embodiment of the present application provides a commodity feature generation model, a commodity feature generation model including:
[0007] Original data acquisition, obtaining product information and user information of all products; the product information includes product titles and product feature data; the user information includes keywords searched by users and data on products clicked by users after searching keywords;
[0008] Data processing: using a large language model to generate a text embedding vector for the product title, cleaning, denoising and unifying the specifications of the product title and product feature data in the product information, and extracting structured entity features in the product title and product feature data to obtain an entity feature vector; using a large language model to generate a word embedding vector for the keyword, and constructing a first association matrix between the keyword and the corresponding clicked product data;
[0009] Constructing a validation set, screening out high-frequency products whose click product data volume is greater than a first threshold from all products, and determining clustering features corresponding to the high-frequency products;
[0010] The product feature generation model is trained, the text embedding vector, the entity feature vector, the word embedding vector and the first association matrix are input into the product feature generation model, and a product feature vector corresponding to the product is generated; and the high-frequency products and the corresponding clustering features are used for adjustment and optimization to complete the construction of the product feature generation model.
[0011] By adopting the above technical solution, the original data not only obtains product information, but also adds user information; and for the original data with increased complexity, targeted feature extraction is performed, so as to reduce the resource consumption of the machine learning model training process. At the same time, a multi-feature fusion construction method is adopted to ensure the accuracy of the model's generation of product feature vectors, so as to accurately distinguish different and identical products. Furthermore, high-frequency products are selected as validation sets. On the one hand, high-frequency products contain more manual recognition experience and features, and the accuracy of high-frequency product clustering is high. There are more related data between product information and user information, and the data of high-frequency products is relatively convenient to obtain and the data volume is large. As a result, the trained product feature generation model can achieve the expected effect.
[0012] In some embodiments, the first threshold is the amount of data ranked in the top 1% to 15% of the amount of clicked product data;
[0013] Determine the product titles of the high-frequency products, clean, denoise and unify the product titles of the high-frequency products, and then input the product titles and product feature data corresponding to the high-frequency products into a large language model to generate a text feature vector, and the text feature vector is the clustering feature.
[0014] By adopting the above technical solution and using a large language model to extract features from the product titles of high-frequency products and obtain a text feature vector, the text feature vector can contain the overall semantic information of the product title, thereby retaining more features to ensure the accuracy of subsequent model training.
[0015] In some embodiments, the product feature data includes at least one of a product core name, a product series, a product model, a product brand name, and a product specification parameter.
[0016] In some embodiments, the product information further includes product classification information, and the product classification information includes all products and corresponding product categories;
[0017] Determine the weights of different entity features in each product category, and construct an initial weight distribution table of product classification information and product entity features;
[0018] Based on the weight distribution table, weighted calculation is performed on the structured entity features in the commodity title and the commodity feature data, and based on the weighted entity features, an entity feature vector is calculated.
[0019] By adopting the above technical solution, the model can arrange the weight distribution in a targeted manner based on different product categories. At the same time, the weight distribution table can be set in advance based on prior experience, thereby greatly reducing the complexity of model training; the model only needs to fine-tune the weights of the entity features of different categories of goods in the weight distribution table during the training process to meet the accuracy requirements of the model.
[0020] In some embodiments, during the training process of the product feature generation model, the product feature generation model can adjust and optimize the weight distribution table.
[0021] The embodiment of the present application also provides a method for identifying and clustering commodities, including the commodity feature generation model described in any one of the above embodiments;
[0022] Obtaining first feature vectors of all commodities through the commodity feature generation model;
[0023] Based on the first feature vector, a clustering algorithm is used to calculate the similarity between different commodities, obtain clustering labels of similar and / or identical commodities, and construct a clustering result table;
[0024] The clustering result table includes clustering labels of commodities and corresponding commodity information.
[0025] By adopting the above technical solution, based on the product information, all products are generated with corresponding first feature vectors through the product feature generation model, so that the clustering algorithm can be used to directly calculate the similarity between different products by comparing the first feature vectors of the products to complete the identification and clustering of the products; finally, by constructing clustering labels and clustering result tables, the identification and clustering results are displayed to facilitate the subsequent management of the products.
[0026] In some embodiments, the clustering algorithm is set to a weighted KMeans algorithm, and the similarity is measured by weighted distance;
[0027] The calculation formula is:
[0028]
[0029] w i is the weight of the i-th feature dimension, x i and y are the values of the two products on the i-th feature dimension.
[0030] In some embodiments, in each cluster, the product name of the product with the highest appearance frequency or sales volume is used as the cluster label.
[0031] An embodiment of the present application also provides a product management system for an e-commerce platform, and the product identification and clustering method described in any of the above embodiments identifies and clusters all the products in the e-commerce platform.
[0032] By adopting the above technical solution, based on the product information, all products are generated with corresponding first feature vectors through the product feature generation model, so that the clustering algorithm can be used to directly calculate the similarity between different products by comparing the first feature vectors of the products to complete the identification and clustering of the products; finally, by constructing clustering labels and clustering result tables, the identification and clustering results are displayed to facilitate the subsequent management of the products. DETAILED DESCRIPTION
[0033] The following specific embodiments illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with some embodiments, this does not mean that the features of this application are limited to the implementation. On the contrary, the purpose of introducing the application in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide a deep understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0034] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0036] The present application embodiment provides a commodity feature generation model, including:
[0037] Original data acquisition, obtain product information and user information of all products; product information includes product title and product feature data; user information includes keywords searched by users and data on products clicked by users after searching keywords;
[0038] Data processing: Use a large language model to generate text embedding vectors for product titles, clean, denoise, and unify the specifications of product titles and product feature data in product information, and extract structured entity features from product titles and product feature data to obtain entity feature vectors; Use a large language model to generate word embedding vectors for keywords, and construct the first association matrix between keywords and corresponding clicked product data;
[0039] Construct a validation set, select high-frequency products whose click data volume is greater than the first threshold from all products, and determine the clustering features corresponding to the high-frequency products;
[0040] Train the product feature generation model, input the text embedding vector, entity feature vector, word embedding vector and the first association matrix into the product feature generation model, and generate the product feature vector corresponding to the product; and use high-frequency products and corresponding clustering features for adjustment and optimization to complete the construction of the product feature generation model.
[0041] Among them, the product feature generation model not only obtains product information, but also adds user information; and performs targeted feature extraction on the original data with increased complexity, so as to reduce the resource consumption of the machine learning model training process. At the same time, a multi-feature fusion construction method is adopted to ensure the accuracy of the model's generation of product feature vectors, so as to accurately distinguish different and identical products. Furthermore, high-frequency products are selected as validation sets. On the one hand, high-frequency products contain more manual recognition experience and features, and the accuracy of high-frequency product clustering is high. There are more related data between product information and user information, and the data of high-frequency products is relatively convenient to obtain and the data volume is large. As a result, the trained product feature generation model can achieve the expected effect.
[0042] It should be noted that the administrator of the e-commerce platform can directly obtain the product information of all products on the platform from the merchants listed on the platform. The amount of product information is very large and is provided by the merchants. The product information may include product titles, product feature data, product images, product demonstration videos, promotional posters, product prices, product categories, product inventory, etc. Among them, some product information can be filled in according to the requirements of the e-commerce platform (for example, the classification of products); some product information can be compiled and determined by the merchants themselves (for example, the title of the product link).
[0043] In addition, users' related operations on the e-commerce platform can also be recorded, such as users' search behavior using keywords, records of users clicking on links, users' evaluation information, users' purchase records, etc.
[0044] Therefore, in the process of acquiring raw data, it is necessary to screen and debug massive amounts of raw data in order to greatly reduce the computing resource requirements for training the product feature generation model and the efficiency of model operation while ensuring the accuracy of model feature extraction;
[0045] In one embodiment, when the original data is acquired, large amounts of data such as pictures and videos are removed, and only text data such as product titles and product feature data are extracted, so that the model can complete the generation of product feature vectors only through product titles and corresponding product feature data, thereby playing a role in preliminary distinction; however, since product titles and product feature data are often compiled by merchants themselves and have great uncertainty, it is difficult to directly distinguish products based on the text similarity of product titles; and e-commerce platforms often only conduct macro-collection of product feature data, and do not mark all products of a single merchant one by one, but only record the physical features of all products sold.
[0046] Therefore, in obtaining product information, additional user information is added, including the keywords searched by users and the data of products clicked by users after searching keywords, so as to use user behavior to link keywords with corresponding products. That is, after searching keywords, users will read product titles, view product descriptions and product images, and determine whether they match the searched keywords, thereby generating the behavior of clicking on products. This allows the original training data of the product feature generation model to contain additional distinguishing information, thereby greatly improving the accuracy of the training results of the product feature generation model.
[0047] Furthermore, user information can be converted into text data. The total amount of data is small and has a high correlation with product differentiation. Therefore, the computing power required for the product feature generation model training process can be reduced, saving the training and operation costs of the product feature generation model.
[0048] In one embodiment, the first threshold is the amount of data ranking in the top 1% to 15% of the amount of clicked product data;
[0049] Determine the product titles of high-frequency products, clean, denoise and unify the product titles of high-frequency products, and then input the product titles and product feature data corresponding to the high-frequency products into the large language model to generate text feature vectors, and the text feature vectors are clustering features. Using the large language model to extract features from the product titles of high-frequency products and obtain text feature vectors can make the text feature vectors contain the overall semantic information of the product title, thereby retaining more features to ensure the accuracy of subsequent model training.
[0050] It should be noted that user information can reflect the popularity of products, and popular products are easy to distinguish. Therefore, the first threshold is set to the top 1% to 15% of the clicked product data. This allows the model validation set to be constructed quickly and accurately, simplifying the difficulty of model training, reducing the difficulty of data acquisition and processing, and reducing the cost of model development.
[0051] In one embodiment, the commodity feature data includes at least one of a commodity core name, a commodity series, a commodity model, a commodity brand name, and a commodity specification parameter.
[0052] In other embodiments, the commodity feature data includes the commodity core name, commodity series, commodity model, commodity brand name and commodity specification parameters. In this case, if the commodity feature data are the same, it means that the two commodities are exactly the same.
[0053] It should be noted that in different categories of goods, the parameters in the product feature data determine the degree of product similarity to different degrees. For example, in cosmetics, the entity and brand of the product are the key features to distinguish the products, while in beverages, the entity, brand and regulations of the product are all the key features to distinguish the products.
[0054] Therefore, in one embodiment, the commodity information also includes commodity classification information, and the commodity classification information includes all commodities and corresponding commodity categories;
[0055] Determine the weights of different entity features in each product category, and construct an initial weight distribution table of product classification information and product entity features;
[0056] Based on the weight distribution table, weighted calculation is performed on the structured entity features in the product title and product feature data, and based on the weighted entity features, an entity feature vector is calculated.
[0057] The model can arrange weight distribution in a targeted manner based on different product categories. At the same time, the weight distribution table can be set in advance based on prior experience, which greatly reduces the complexity of model training. The model only needs to fine-tune the weights of the entity features of different categories of goods in the weight distribution table during training to meet the accuracy requirements of the model.
[0058] In one embodiment, the initial weight distribution table may be as shown in the following table:
[0059]
[0060]
[0061] In one embodiment, during the training process of the product feature generation model, the product feature generation model may adjust and optimize the weight distribution table.
[0062] In one embodiment, the commodity feature generation model can be applied to commodity identification and clustering. A commodity identification and clustering method based on the commodity feature generation model is described in detail below.
[0063] First, the first feature vector of all products is obtained through the product feature generation model;
[0064] Based on the first eigenvector, a clustering algorithm is used to calculate the similarity between different commodities, obtain clustering labels of similar and / or identical commodities, and construct a clustering result table;
[0065] The clustering result table includes the clustering labels of the products and the corresponding product information.
[0066] Therefore, based on the product information, the corresponding first feature vectors can be generated for all products through the product feature generation model, so that the clustering algorithm can be used to directly calculate the similarity between different products by comparing the first feature vectors of the products to complete the identification and clustering of the products; finally, by constructing clustering labels and clustering result tables, the identification and clustering results are displayed to facilitate the subsequent management of the products.
[0067] In one embodiment, the clustering algorithm is set to a weighted KMeans algorithm, and the similarity is measured by weighted distance;
[0068] The calculation formula is:
[0069]
[0070] w i is the weight of the i-th feature dimension, x i and i is the value of the two products on the i-th feature dimension.
[0071] In one embodiment, in each cluster, the product name of the product with the highest frequency of occurrence or sales volume is used as the cluster label.
[0072] In one usage scenario, the commodity identification and clustering method can be applied to the commodity management system of an e-commerce platform, thereby identifying and clustering all commodities in the e-commerce platform.
[0073] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and modifications of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and modifications.
Claims
1. A commodity feature generation model, characterized in that: include: Original data acquisition, obtaining product information and user information of all products; the product information includes product titles and product feature data; the user information includes keywords searched by users and data on products clicked by users after searching keywords; Data processing: using a large language model to generate a text embedding vector for the product title, cleaning, denoising and unifying the specifications of the product title and product feature data in the product information, and extracting structured entity features in the product title and product feature data to obtain an entity feature vector; using a large language model to generate a word embedding vector for the keyword, and constructing a first association matrix between the keyword and the corresponding clicked product data; Constructing a validation set, screening out high-frequency products whose click product data volume is greater than a first threshold from all products, and determining clustering features corresponding to the high-frequency products; The product feature generation model is trained, the text embedding vector, the entity feature vector, the word embedding vector and the first association matrix are input into the product feature generation model, and a product feature vector corresponding to the product is generated; and the high-frequency products and the corresponding clustering features are used for adjustment and optimization to complete the construction of the product feature generation model.
2. The commodity feature generation model according to claim 1, characterized in that: The first threshold is the amount of data ranking in the top 1% to 15% of the amount of clicked product data; Determine the product titles of the high-frequency products, clean, denoise and unify the product titles of the high-frequency products, and then input the product titles and product feature data corresponding to the high-frequency products into a large language model to generate a text feature vector, and the text feature vector is the clustering feature.
3. The commodity feature generation model according to claim 1, characterized in that: The commodity characteristic data includes at least one of a commodity core name, a commodity series, a commodity model, a commodity brand name, and a commodity specification parameter.
4. The commodity feature generation model according to claim 1, characterized in that: The commodity information also includes commodity classification information, which includes all commodities and corresponding commodity categories; Determine the weights of different entity features in each product category, and construct an initial weight distribution table of product classification information and product entity features; Based on the weight distribution table, weighted calculation is performed on the structured entity features in the commodity title and the commodity feature data, and an entity feature vector is calculated based on the weighted entity features.
5. The commodity feature generation model according to claim 4, characterized in that: During the training process of the commodity feature generation model, the commodity feature generation model can adjust and optimize the weight distribution table.
6. A method for identifying and clustering commodities, characterized in that: Comprising a commodity feature generation model as described in any one of claims 1 to 5; Obtaining first feature vectors of all commodities through the commodity feature generation model; Based on the first feature vector, a clustering algorithm is used to calculate the similarity between different commodities, obtain clustering labels of similar and / or identical commodities, and construct a clustering result table; The clustering result table includes clustering labels of commodities and corresponding commodity information.
7. The commodity feature generation model according to claim 6, characterized in that: The clustering algorithm is set to a weighted KMeans algorithm, and the similarity is measured by weighted distance; The calculation formula is: w i is the weight of the i-th feature dimension, x i and i is the value of the two products on the i-th feature dimension.
8. The commodity feature generation model according to claim 6, characterized in that: In each cluster, the product name of the product with the highest frequency or sales volume is used as the cluster label.
9. A product management system for an e-commerce platform, characterized in that: The commodity identification and clustering method as described in any one of claims 6 to 8 is used to identify and cluster all commodities in the e-commerce platform.