Commodity data multimodal cleaning method and its device, equipment, medium, and product

Abstract

The present application discloses a method for multimodal cleaning of commodity data and its device, equipment, medium, and product. The method comprises: determining a commodity data set, the commodity data set comprising a plurality of commodity data carrying real classification labels, the real classification labels being leaf nodes in a preset category tree; clustering the text feature information of each commodity data to determine the text clustering labels corresponding to the commodity data, the total number of text clustering labels being equal to the total number of leaf nodes; clustering the image feature information of each commodity data to determine the image clustering labels corresponding to the commodity data, the total number of image clustering labels being equal to the total number of leaf nodes; and performing data cleaning on the commodity data in the commodity data set according to the value information of the text clustering labels and image clustering labels of the same commodity data relative to the real classification labels. The present application can effectively clean massive commodity data in a multimodal manner.

Description

Technical Field

[0001] The present application relates to the field of e-commerce information technology, and in particular to a multimodal cleaning method for commodity data and its corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of deep learning, the scale of various neural network models is getting larger and larger, and the scale of training data is also getting larger and larger. As the scale expands, the noise data in the training data is also gradually increasing. A large amount of noise data will seriously affect the effect of the model. Traditional manual labeling is difficult to efficiently process such a large amount of data. If the data is not effectively cleaned, the noise data will seriously affect the training effect of the neural network model.

[0003] In the prior art, the training data required for the neural network model is cleaned by applying simple means to detect incomplete data, erroneous data, duplicate data, etc. This is very simple and crude, and does not fully consider the value factors of the data itself to the neural network model. Therefore, it has little effect on improving the value of the training data relative to the model.

[0004] The training data required for model training, especially data with supervisory labels, and whether their corresponding labels are accurate and effective will greatly affect the training effect of the model, and specifically affect the model's learning ability. Therefore, more attention should be paid to this in the data cleaning stage. The relevant solutions proposed by the industry for cleaning labeled training data are mainly based on clustering algorithms, which simply verify the clustering results and remove training data whose annotation information does not match the clustering results. Although this method improves the quality of the data to a considerable extent, it is not sophisticated enough and fails to combine with the model to clean the relevant data. Especially for multimodal models, simple clustering may not be able to effectively achieve data cleaning.

[0005] The need for data cleaning is particularly evident in the e-commerce field. In the e-commerce field, the product data corresponding to the massive amount of goods generally have label information corresponding to them one by one. However, when the label information of these product data comes from different sources or is generated with different standards, how to effectively label these product data becomes a greater problem.

[0006] In summary, how to clean the training data consisting of product data so that it can adapt to the needs of the neural network model and become effective training data is worth exploring in the e-commerce field. Summary of the invention

[0007] The primary purpose of the present application is to solve at least one of the above problems and to provide a multimodal cleaning method for commodity data and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0008] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0009] A multimodal cleaning method for commodity data provided to meet one of the purposes of this application includes the following steps:

[0010] Determine a commodity data set, where the commodity data set includes a plurality of commodity data carrying real classification labels, where the real classification labels are leaf nodes in a preset category tree;

[0011] Clustering is performed based on the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes;

[0012] Clustering is performed based on the image feature information of each product data to determine the image clustering label corresponding to the product data, and the total number of image clustering labels is equal to the total number of leaf nodes;

[0013] Data cleaning is performed on the commodity data in the commodity data set according to value information of text clustering labels and image clustering labels of the same commodity data relative to real classification labels.

[0014] In a further embodiment, clustering is performed according to the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes, including the following steps:

[0015] Using a pre-trained text feature extraction model to extract text feature information of the product title of each product data in the product data set;

[0016] The number of categories required for clustering is set according to the number of leaf nodes, and the text feature vectors of the commodity data are used for clustering to obtain a plurality of text-based commodity data clusters corresponding to the number of categories, each commodity data cluster containing a plurality of correspondingly clustered commodity data;

[0017] For each text-based commodity data cluster, the maximum number of true classification labels of the commodity data therein is counted, and the maximum number of true classification labels is determined as the text clustering label of each commodity data in the commodity data cluster.

[0018] In a further embodiment, clustering is performed according to the image feature information of each commodity data to determine the image clustering label corresponding to the commodity data, and the total number of image clustering labels is equal to the total number of leaf nodes, including the following steps:

[0019] Using a pre-trained image feature extraction model to extract image feature information of the product image of each product data in the product data set;

[0020] The number of categories required for clustering is set according to the number of leaf nodes, and the image feature vectors of the commodity data are used for clustering to obtain a plurality of commodity data clusters based on the images corresponding to the number of categories, each of which contains a plurality of commodity data clustered accordingly;

[0021] For each image-based product data cluster, the maximum number of true classification labels owned by the product data therein is counted, and the maximum number of true classification labels is determined as the image clustering label of each product data in the product data cluster.

[0022] In a further embodiment, data cleaning is performed on the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels, including the following steps:

[0023] Make consistency judgment on the text clustering labels and image clustering labels of the same product data;

[0024] When it is determined that the text clustering label and the image clustering label of the same product data are consistent, it is determined that the two clustering labels have contribution value relative to the real classification label, and the real classification label is reset to the text clustering label or the image clustering label;

[0025] When it is determined that the text clustering label and the image clustering label of the same product data are inconsistent, and one of the text clustering label and the image clustering label is consistent with the real classification label, a text clustering label or an image clustering label with contribution value is determined according to a preset condition, so as to reset the real classification label of the product data that meets the preset condition;

[0026] The product data whose true classification labels have not been reset are deleted from the product dataset to achieve cleaning.

[0027] In a specific embodiment, determining a text clustering label or an image clustering label with contribution value according to a preset condition to reset the real classification label of the commodity data meeting the preset condition includes the following steps:

[0028] Count the average length of product titles in the product data set;

[0029] In the case where the text clustering label of the product data is consistent with the real classification label but the image clustering label is inconsistent with the real classification label, the length of the product title of the product data exceeds the average length as a preset condition. When the preset condition is met, it is confirmed that the text clustering label has contribution value, and the real classification label of the product data is reset with it;

[0030] For the case where the image clustering label of the product data is consistent with the true classification label but the text clustering label is inconsistent with the true classification label, the length of the product title of the product data is less than the average length as a preset condition. When the preset condition is met, it is confirmed that the image clustering label has contribution value, so as to reset the true classification label of the product data.

[0031] In an extended embodiment, the multimodal cleaning method for commodity data of the present application further includes the following subsequent steps:

[0032] After the data cleaning is completed, the text feature information and the image feature information of each commodity object in the commodity data set are concatenated to obtain the corresponding image and text feature information;

[0033] The classifier is iteratively trained using the graphic feature information of each commodity data in the commodity data set, and the classification label of each commodity data is supervised by the real classification label of the commodity data during the iterative training process to predict the classification label, and the weight of the classifier is gradient updated according to the loss value between the predicted classification label and the real classification label until the classifier is trained to a convergence state.

[0034] A multimodal cleaning device for commodity data provided to meet one of the purposes of the present application includes: a data determination module, a text clustering module, an image clustering module, and a data cleaning module, wherein the data determination module is used to determine a commodity data set, the commodity data set includes multiple commodity data carrying real classification labels, and the real classification labels are leaf nodes in a preset category tree; the text clustering module is used to cluster and determine the text clustering label corresponding to the commodity data according to the text feature information of each commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes; the image clustering module is used to cluster and determine the image clustering label corresponding to the commodity data according to the image feature information of each commodity data, and the total number of image clustering labels is equal to the total number of leaf nodes; the data cleaning module is used to perform data cleaning on the commodity data in the commodity data set according to the value information of the text clustering label and the image clustering label of the same commodity data relative to the real classification label.

[0035] In a further embodiment, the text clustering module includes: a text extraction submodule, which is used to extract text feature information of the product title of each product data in the product data set by using a pre-trained text feature extraction model; a text clustering submodule, which is used to set the number of categories required for clustering according to the number of leaf nodes, cluster the text feature vectors of the product data, and obtain multiple text-based product data clusters corresponding to the number of categories, each product data cluster containing multiple correspondingly clustered product data; a text annotation submodule, which is used to count the maximum number of real classification labels possessed by the product data in each text-based product data cluster, and determine the maximum number of real classification labels as the text clustering labels of each product data in the product data cluster.

[0036] In a further embodiment, the image clustering module includes: an image extraction submodule, which is used to extract image feature information of the product image of each product data in the product data set by using a pre-trained image feature extraction model; an image clustering submodule, which is used to set the number of categories required for clustering according to the number of leaf nodes, cluster the image feature vectors of the product data, and obtain multiple image-based product data clusters corresponding to the number of categories, each product data cluster containing multiple correspondingly clustered product data; an image labeling submodule, which is used to count the maximum number of real classification labels of the product data in each image-based product data cluster, and determine the maximum number of real classification labels as the image clustering labels of each product data in the product data cluster.

[0037] In a further embodiment, the data cleaning module includes: a comparison judgment submodule, which is used to judge the consistency of the text clustering label and the image clustering label of the same product data; a priority reset submodule, which is used to determine that the two clustering labels have contribution value relative to the real classification label when the text clustering label and the image clustering label of the same product data are consistent, and reset the real classification label to the text clustering label or the image clustering label; a comprehensive reset submodule, which is used to determine that the text clustering label and the image clustering label of the same product data are inconsistent, and one of the text clustering label and the image clustering label is consistent with the real classification label, and determine the text clustering label or the image clustering label with contribution value according to preset conditions to reset the real classification label of the product data that meets the preset conditions; a purification cleaning submodule, which is used to delete the product data whose real classification labels have not been reset from the product data set to achieve cleaning.

[0038] In a specific embodiment, the comprehensive reset submodule includes: a title length statistics unit, which is used to count the average length of product titles of product data in the product data set; a text label reset unit, which is used to reset the real classification label of the product data by taking the length of the product title of the product data exceeding the average length as a preset condition when the text clustering label of the product data is consistent with the real classification label but the image clustering label is inconsistent with the real classification label; and a picture label reset unit, which is used to reset the real classification label of the product data by taking the length of the product title of the product data being less than the average length as a preset condition when the preset condition is met.

[0039] In an extended embodiment, the multimodal cleaning device for commodity data of the present application also includes: a feature splicing module, which is used to perform feature splicing on the text feature information and image feature information of each commodity object in the commodity data set after data cleaning is completed, so as to obtain its corresponding graphic and text feature information; an iterative training module, which is used to use the graphic and text feature information of each commodity data in the commodity data set to perform iterative training on the classifier, and to supervise the predicted classification label of the commodity data during the iterative training process with the real classification label, and to perform gradient update on the weight of the classifier according to the loss value between the predicted classification label and the real classification label until the classifier is trained to a convergence state.

[0040] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the multimodal cleaning method for commodity data described in the present application.

[0041] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the multimodal cleaning method for commodity data in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0042] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0043] Compared with the prior art, the advantages of this application are as follows:

[0044] First, when cleaning the commodity data in the commodity data set, the present application adapts to the situation where the commodity data contains multimodal information. Taking into account that multimodal information can be required for multimodal training, deep semantic information is extracted for the specific data of each modality in the commodity data, and the clustering labels corresponding to each modality are determined according to the deep semantic information of each modality, such as the text clustering labels and image clustering labels described above. Then, the value information of each modality clustering label is examined relative to the real classification labels originally annotated with the commodity data, and the clustering labels that contribute to the adjustment of the real classification labels are reset to the latest real classification labels of the corresponding commodity data, thereby realizing the correction of the annotation of the corresponding commodity data. At the same time, other commodity data that lack value can also be cleaned out of the commodity data set, making the commodity data set a higher quality training data set.

[0045] Secondly, the commodity data set obtained after data cleaning of multiple modalities in this application can be used as a training data set for a neural network model that performs classification tasks. The annotation information of the classification labels is more accurate, suitable for more efficient training of new classifier instances, which can make the training process of the classifier instance converge more quickly and improve the prediction accuracy of the classifier;

[0046] In addition, for the e-commerce field, since the information of product data is multi-source and complex, this application implements clustering from two dimensions: text and pictures, and then performs data cleaning on this basis, taking into account both the complexity of product data and the computational efficiency of the cleaning process. Therefore, the technical solution of this application can provide significant contribution value in the preparation of training data sets in the e-commerce field. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a typical embodiment of the multimodal cleaning method for commodity data of the present application;

[0049] Figure 2 A flowchart of a process for obtaining text feature information of commodity data in an embodiment of the present application;

[0050] Figure 3 A flowchart of the process of obtaining image feature information of commodity data in an embodiment of the present application;

[0051] Figure 4 A flowchart of a data cleaning process according to an embodiment of the present application is shown below;

[0052] Figure 5This is a principle block diagram of the process of re-determining the real classification label according to the average length of the product title in an embodiment of the present application;

[0053] Figure 6 This is a flow chart of the process of training a classifier using a commodity data set after data cleaning in an embodiment of the present application;

[0054] Figure 7 This is a principle block diagram of the commodity data multi-modal cleaning device of this application;

[0055] Figure 8 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0057] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0059] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0060] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0061] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0062] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0063] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0064] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0065] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0066] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0067] A multimodal cleaning method for commodity data of the present application can be programmed as a computer program product and deployed in a service cluster for execution, so that the method can be executed by accessing an interface opened after the computer program product is running and performing human-computer interaction with the computer program product through a graphical user interface.

[0068] An application scenario exemplified in the present application is an application scenario related to classification task training in the e-commerce field. Due to the need for certain standard classification in the e-commerce field based on the product data corresponding to the product object, including but not limited to product title, product image, product details, product attributes, etc., a product data set composed of prepared product data is used to train the classifier. Therefore, these product data sets can be cleaned with the help of the technical solution of the present application to determine an effective purified training set, and the classifier trained with the purified training set is determined as the classifier required for production, so that the classifier can accurately classify the product data in the e-commerce platform according to the said standard.

[0069] The classification implemented by the classifier can be, for example, the classification of product safety attributes, the classification of products mapped to the category tree of the e-commerce platform, etc. Different classification tasks can be determined according to different classification standards and corresponding training can be performed. Accordingly, the classification labels in the product data set are also classification labels corresponding to the classification standards, so that the classifier can implement supervised learning based on the classification labels and acquire the corresponding classification capabilities after learning. Those skilled in the art should be aware of this.

[0070] See also Figure 1 In a typical embodiment, the multimodal cleaning method for commodity data of the present application comprises the following steps:

[0071] Step S1100: determine a commodity data set, wherein the commodity data set includes a plurality of commodity data carrying real classification labels, and the real classification labels are leaf nodes in a preset category tree:

[0072] First, a commodity data set is prepared, which contains commodity data required for training the exemplary neural network model of this application, and the commodity data all carry classification labels with pre-established corresponding mapping relationships, which are referred to as real classification labels. The real classification labels are suitable for providing the trained neural network model with the supervision labels required for supervised learning, so as to supervise the training process of the neural network model and promote the neural network model to be trained to a convergence state.

[0073] The product data described above, for the purpose of adapting to the exemplary application scenarios of this application, includes specific data in multiple modalities such as product titles and product images. The various types of information content included can be determined based on the specific data on which the neural network model is based. For example, it can also include product details, product attributes and other data.

[0074] The commodity data may be obtained from a commodity database of an e-commerce platform, or from a website that is open and free to obtain data on the Internet, and may be flexibly selected by technicians in this field.

[0075] The real classification label of the commodity data in the commodity data set can be determined as the classification label of the commodity data in the preset category tree constructed by the source e-commerce platform website. The category tree generally includes multiple levels. In addition to the root node, each level belongs to a parent node, which may have multiple child nodes, among which the child nodes at the bottom of the category tree constitute leaf nodes. Each commodity data will be pre-assigned a classification path. The classification path starts from the root node of the category tree through various levels of intermediate nodes to the terminal leaf node, and thus includes multiple levels of nodes. That is, each commodity data will have a corresponding leaf node. Since the leaf nodes in the category tree generally have unique characteristics, the leaf nodes of the commodity data can be used as their real classification labels.

[0076] The real classification label of the commodity data can be derived from the source e-commerce platform website of the commodity data, or it can be generated by a technician in this field using a pre-trained classification model to adapt to the needs of this application. Such a classification model is suitable for classification based on the deep semantic information of the commodity title and / or commodity image of the commodity data, and classifies the commodity data into a leaf node of a preset category tree. Naturally, the classification performed here is only a preliminary classification performed for the purpose of implementing this application to obtain the real classification label of the commodity data. The classification model referred to here is a neural network model. For example, Bert, Electra, Albert, Roberta, etc. can be used to extract deep semantic information from the commodity title, and Resnet, EfficientNet, etc. can be used to extract deep semantic information from the commodity image, and then a classifier is used to classify each leaf node of the category tree according to these deep semantic information, so as to determine the real classification label corresponding to the commodity data.

[0077] For the convenience of explanation, in this application, the commodity data set is concentrated, and the real classification label carried by the commodity data is marked as T label .

[0078] Step S1200: clustering is performed according to the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes:

[0079] In this application, in order to meet the needs of clustering the commodity data set before data cleaning, it is necessary to provide deep semantic information of the corresponding commodity data for clustering. Taking into account the diversity of specific data contained in commodity data, this application divides it into two categories according to the information mode of the data. One category is text type data in commodity data, including but not limited to commodity titles, commodity details, commodity attributes, etc.; the other category is image type data in commodity data, including but not limited to commodity pictures, advertising pictures, comment pictures, etc., where commodity pictures also include default pictures of commodities that represent the commodity data by default and other detail pictures displayed on the details page. Data that adapt to these two modes can be processed by different processing methods when clustering.

[0080] This step is mainly responsible for clustering the text type data in the commodity data. These text type data can be all the text type data in the commodity data, or one of them can be selected. For example, for the convenience of explanation, this typical embodiment only selects the commodity title in the commodity data as the source data required for clustering.

[0081] When clustering the product titles, it is performed based on the deep semantic information of the product titles. Therefore, it is necessary to extract the deep semantic information of the product titles of each product data, that is, its text feature information, which is usually expressed in vector form. The extraction of the text feature information can be implemented using a pre-trained text feature extraction model, including but not limited to Bert, Electra, Albert, Roberta and other optional neural network models, which can be used to extract the deep semantic information of the product titles in the product data and obtain their corresponding text feature information. After obtaining the text feature information corresponding to the product title of each product data, it can be associated and stored in the product data set as an extended data of the corresponding product data for reference during clustering.

[0082] After obtaining the text feature information of each commodity data, each commodity data can be clustered based on the text feature information with the help of any known clustering algorithm, including but not limited to k-means clustering algorithm, mean shift clustering algorithm, DBSCAN clustering algorithm, expectation maximization (EM) clustering using Gaussian mixture model (GMM), hierarchical clustering algorithm, spectral clustering algorithm and other clustering algorithms known to those skilled in the art. In this embodiment, the spectral clustering algorithm is recommended. Spectral clustering is a widely used clustering algorithm. Compared with the traditional K-Means algorithm, spectral clustering has stronger adaptability to data distribution and excellent clustering effect. At the same time, the amount of clustering calculation is much smaller, and what is more valuable is that it is not complicated to implement. The basic principle of various clustering algorithms is to calculate the similarity information, or distance information, between the deep semantic information of various commodity data, and treat commodity data with close or relatively similar distances as similar data, and treat commodity data with distant or less similar distances as different types of data, thereby determining the corresponding multiple classifications.

[0083] Since in this application, the product data already has the real classification labels corresponding to its leaf nodes, the total number of leaf nodes has determined the number of categories into which all the product data in the product data set are classified. For the purpose of data cleaning, the number of clusters when performing text-based clustering should also be equal to the total number of leaf nodes. Therefore, when clustering, the number of clusters can be preset as the total number of leaf nodes where the product data in the product data set is distributed.

[0084] When the clustering algorithm is used to cluster the commodity data in the commodity data set based on the text feature information, multiple clustering labels corresponding to the total number of leaf nodes can be obtained. Here, the text clustering label is marked as V for the convenience of subsequent explanation. label It is not difficult to understand that each product data in the product data set can obtain a corresponding text clustering label after being clustered based on the text of the product title.

[0085] Generally speaking, according to statistical principles, for commodity data under the same real classification label, the text clustering labels of most commodity data may be consistent with the real classification label, while the text clustering labels of a small number of commodity data may be other real classification labels. On the other hand, for commodity data under a text clustering label, the real classification labels of most commodity data may be consistent with the text clustering label, but it is not ruled out that the real classification labels of a small number of commodity data are inconsistent with the text clustering label and are relatively scattered. This difference also illustrates the necessity of data cleaning for commodity data in commodity datasets.

[0086] Step S1300: cluster the image feature information of each commodity data to determine the image clustering label corresponding to the commodity data, and the total number of image clustering labels is equal to the total number of leaf nodes:

[0087] Similar to the previous step of clustering based on the text feature information of the product titles of the product data, in this step, another modality of information, namely, product images, is clustered.

[0088] When clustering the product images, it is performed based on the deep semantic information of the product images. Therefore, it is necessary to extract the deep semantic information, that is, the image feature information, of the product images of each product data, which is usually expressed in vector form. The extraction of the image feature information can be implemented using a pre-trained image feature extraction model, including but not limited to optional neural network models such as Resnet and EfficientNet, which can be used to extract the deep semantic information of the product images in the product data and obtain their corresponding image feature information. After obtaining the image feature information corresponding to the product title of each product data, it can be associated and stored in the product data set as an extension data of the corresponding product data for reference during clustering.

[0089] After obtaining the image feature information of each commodity data, each commodity data can be clustered based on the image feature information with the help of any known clustering algorithm, including but not limited to k-means clustering algorithm, mean shift clustering algorithm, DBSCAN clustering algorithm, expectation maximization (EM) clustering using Gaussian mixture model (GMM), hierarchical clustering algorithm, spectral clustering algorithm and other clustering algorithms known to those skilled in the art. In this embodiment, the spectral clustering algorithm is recommended for the same reason. The basic principles and applications of various clustering algorithms are described in detail in the previous step, which will not be repeated here.

[0090] Similar to the previous step, since the product data already has the real classification labels corresponding to its leaf nodes, the total number of leaf nodes has determined the number of categories into which all product data in the product data set are classified. For data cleaning purposes, the number of clusters when clustering based on images should also be equal to the total number of leaf nodes. Therefore, when clustering, the number of clusters can be preset as the total number of leaf nodes where the product data in the product data set is distributed.

[0091] When the clustering algorithm is used to cluster the commodity data in the commodity data set based on the image feature information, multiple clustering labels corresponding to the total number of leaf nodes can be obtained. Here, it is the image clustering label, which is marked as C for the convenience of subsequent explanation. label It is not difficult to understand that each product data in the product data set can obtain a corresponding image clustering label after being clustered based on the product images.

[0092] Similarly, according to statistical principles, for commodity data under the same real classification label, the image clustering labels of most commodity data may be consistent with the real classification label, while the image clustering labels of a small number of commodity data may be other real classification labels. On the other hand, for commodity data under an image clustering label, the real classification labels of most commodity data may be consistent with the image clustering label, but it is not ruled out that the real classification labels of a small number of commodity data are inconsistent with the image clustering label and are relatively scattered. This difference also illustrates the necessity of data cleaning for commodity data in commodity datasets.

[0093] Step S1400: Perform data cleaning on the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels:

[0094] As mentioned above, after obtaining text clustering labels and image clustering labels based on clustering of text and images in product data, both may be consistent or inconsistent with the real classification labels of the product data itself. Furthermore, the text clustering labels and image clustering labels in the same product data may also be consistent or inconsistent. In this application, the consistency between these different labels is utilized to mine the value information of the text clustering labels and image clustering labels relative to their real classification labels for each product data, and then data cleaning is performed on the product data based on this value information. The main cleaning methods include deleting the product data or correcting its real classification label.

[0095] The basic principle of the value information is to examine the credibility of the text clustering labels and / or image clustering labels relative to the real classification labels. If the credibility of the former is higher than that of the latter, indicating that the clustering result is more credible, the former can be replaced by the latter, thereby correcting the real classification labels of the commodity data. If the credibility of the former is lower than that of the latter, it indicates that the effective classification labels of the commodity data cannot be determined with high credibility by integrating various information. In this case, the entire commodity data can be deleted from the commodity data set.

[0096] The mining of the value information can be achieved by setting preset conditions according to preset strategies and comparing the consistency of the text clustering labels, image clustering labels and real classification labels of the commodity data. For example, V label =C label =T labelWhen this precondition is met, the text clustering labels and image clustering labels are considered to have contribution value relative to the real classification labels, so the original real classification labels can be retained, or the text clustering labels and image clustering labels can be used to replace the real classification labels of the product data (the three are essentially the same). On the contrary, if V label ≠C label ≠T label When this preset condition is met, it can be considered that the text clustering labels and image clustering labels have no contribution value relative to the original real classification labels, and the classification labels of the product data cannot be re-determined, so the product data can be deleted from the product data set.

[0097] In addition to the above exemplary methods, in other embodiments, when setting the preset conditions, other factors can be introduced to sort out the complex relationships between the labels to refine the analysis of the value information, such as introducing the length information of the product title to measure whether the text clustering label has a higher value than the image clustering label. In this regard, the subsequent embodiments of this application will further reveal this, and this embodiment will not be discussed for the time being. In short, based on the disclosure of this embodiment, those skilled in the art should be able to determine which label has which value information for the product data, including contribution value or no contribution value, based on the relationship between the text clustering label and the image clustering label and the real classification label. Then, cleaning decisions can be made based on this value information, and corresponding means can be used to achieve the cleaning of the corresponding product data.

[0098] In the commodity data set after data cleaning, some commodity data that are easy to cause classification confusion are deleted during the cleaning process, and the real classification labels of some commodity data that are obviously incorrectly labeled are corrected. Therefore, the real classification labels of the commodity data constitute more effective supervision labels. When the commodity data set is used as the training data set of the classification model corresponding to the classification task, the commodity data with improved accuracy and increased proportion of effective samples can prompt the corresponding classification model to be trained to a convergence state more quickly and efficiently, and the trained classification model can obtain higher classification prediction accuracy.

[0099] Through this typical embodiment, it can be seen that the present application has many advantages, such as:

[0100] First, when cleaning the commodity data in the commodity data set, the present application adapts to the situation where the commodity data contains multimodal information. Taking into account that multimodal information can be required for multimodal training, deep semantic information is extracted for the specific data of each modality in the commodity data, and the clustering labels corresponding to each modality are determined according to the deep semantic information of each modality, such as the text clustering labels and image clustering labels described above. Then, the value information of each modality clustering label is examined relative to the real classification labels originally annotated with the commodity data, and the clustering labels that contribute to the adjustment of the real classification labels are reset to the latest real classification labels of the corresponding commodity data, thereby realizing the correction of the annotation of the corresponding commodity data. At the same time, other commodity data that lack value can also be cleaned out of the commodity data set, making the commodity data set a higher quality training data set.

[0101] Secondly, the commodity data set obtained after data cleaning of multiple modalities in this application can be used as a training data set for a neural network model that performs classification tasks. The annotation information of the classification labels is more accurate, suitable for more efficient training of new classifier instances, which can make the training process of the classifier instance converge more quickly and improve the prediction accuracy of the classifier;

[0102] In addition, for the e-commerce field, since the information of product data is multi-source and complex, this application implements clustering from two dimensions: text and pictures, and then performs data cleaning on this basis, taking into account both the complexity of product data and the computational efficiency of the cleaning process. Therefore, the technical solution of this application can provide significant contribution value in the preparation of training data sets in the e-commerce field.

[0103] See also Figure 2 In a further embodiment, the step S1200, clustering the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, the total number of text clustering labels is equal to the total number of leaf nodes, comprises the following steps:

[0104] Step S1210: Use a pre-trained text feature extraction model to extract text feature information of the product title of each product data in the product data set:

[0105] In this embodiment, in order to obtain the text feature information of the commodity data in the commodity data set, the commodity title in the commodity data can be preferably used as the extraction object, and the neural network model for extracting the text feature information of the commodity title can be preferably implemented by a pre-trained Bert model. After the commodity titles of each commodity data in the commodity data set are input into the Bert model, the corresponding text feature information is obtained one by one, and the text feature information is associated with the corresponding commodity data and stored.

[0106] Step S1220: setting the number of categories required for clustering according to the number of leaf nodes, clustering the text feature vectors of the commodity data, and obtaining a plurality of text-based commodity data clusters corresponding to the number of categories, each commodity data cluster containing a plurality of correspondingly clustered commodity data:

[0107] In this embodiment, a spectral clustering algorithm is used, and the number of categories required for clustering is set to be the same as the total number K of leaf nodes, and then the distance information is calculated for the text feature vector of each commodity data in the commodity data set, and then K commodity data clusters are obtained accordingly, and each commodity data cluster contains a plurality of commodity data that are clustered accordingly. That is to say, all commodity data in the commodity data set are dispersed and divided into K commodity data clusters after being clustered based on text feature information. It can be understood that all commodity data in the same commodity data cluster, in theory, are classified into the same category based on a certain feature.

[0108] Step S1230: For each text-based commodity data cluster, count the maximum number of real classification labels owned by the commodity data therein, and determine the maximum number of real classification labels as the text clustering label of each commodity data in the commodity data cluster:

[0109] In order to determine the clustering label corresponding to each commodity data cluster, that is, the text clustering label, it can be achieved by examining the common characteristics of many commodity data in the same commodity data cluster. Specifically, the real classification labels carried by the commodity data in the same commodity data cluster can be counted, among which there is a real classification label with the largest statistical number. Under this real classification label, there are the most commodity data in the current commodity data cluster. Therefore, the text clustering label for this part of the commodity data is naturally set to the real classification label with the largest statistical number. In order to unify the clustering labels of the current commodity data cluster, for other commodity data carrying different real classification labels in the current commodity data cluster, based on the default of the common characteristics of the clustering, its text clustering label is set to the real classification label with the largest statistical number. In other words, the real classification label with the largest statistical data in a commodity data cluster is also set as the text clustering label of all commodity data in the commodity data cluster. Based on this, the marking of the text clustering labels of the commodity data of each commodity data cluster can be completed.

[0110] This embodiment elaborates on the process of clustering commodity data based on text feature information using a preset algorithm to generate text clustering labels for each commodity data. It can be seen that labeling commodity data with text clustering labels through a clustering algorithm reflects the common characteristics of commodity data based on text type data, which can constitute a decision-making basis required for data cleaning and provide decision-making value for the data cleaning process.

[0111] See also Figure 3 In a further embodiment, the step S1300, clustering the image feature information of each commodity data to determine the image clustering label corresponding to the commodity data, wherein the total number of image clustering labels is equal to the total number of leaf nodes, comprises the following steps:

[0112] Step S1310: Use a pre-trained image feature extraction model to extract image feature information of the product image of each product data in the product data set:

[0113] In this embodiment, in order to obtain the image feature information of the commodity data in the commodity data set, the commodity images in the commodity data, especially the default images, can be preferably used as the extraction object, and the neural network model for extracting the image feature information of the commodity images can be preferably implemented by a pre-trained Resnet model. After the commodity images of each commodity data in the commodity data set are input into the Resnet model, the corresponding image feature information is obtained one by one, and the image feature information is associated with the corresponding commodity data and stored.

[0114] Step S1320: Set the number of categories required for clustering according to the number of leaf nodes, perform clustering based on the image feature vectors of the commodity data, and obtain a plurality of commodity data clusters based on the images corresponding to the number of categories, each commodity data cluster containing a plurality of commodity data clustered accordingly:

[0115] In this embodiment, a spectral clustering algorithm is used, and the number of categories required for clustering is set to be the same as the total number K of leaf nodes, and then the distance information is calculated for the image feature vector of each commodity data in the commodity data set, and then K commodity data clusters are obtained accordingly, and each commodity data cluster contains a plurality of commodity data that are clustered accordingly. That is to say, all commodity data in the commodity data set are dispersed and divided into K commodity data clusters after being clustered based on the image feature information. It can be understood that all commodity data in the same commodity data cluster, in theory, are classified into the same category based on a certain feature.

[0116] Step S1330: For each image-based product data cluster, count the maximum number of real classification labels owned by the product data therein, and determine the maximum number of real classification labels as the image clustering label of each product data in the product data cluster:

[0117] In order to determine the clustering label corresponding to each commodity data cluster, that is, the image clustering label, it can be achieved by examining the common characteristics of many commodity data in the same commodity data cluster. Specifically, the real classification labels carried by the commodity data in the same commodity data cluster can be counted, among which there is a real classification label with the largest statistical number. Under this real classification label, there are the most commodity data in the current commodity data cluster. Therefore, the image clustering label for this part of the commodity data is naturally set to the real classification label with the largest statistical number. In order to unify the clustering labels of the current commodity data cluster, for other commodity data carrying different real classification labels in the current commodity data cluster, based on the default of the common characteristics of the clustering, its image clustering label is set to the real classification label with the largest statistical number. In other words, the real classification label with the largest statistical data in a commodity data cluster is also set as the image clustering label of all commodity data in the commodity data cluster. Based on this, the marking of the image clustering labels of the commodity data of each commodity data cluster can be completed.

[0118] This embodiment describes in detail the process of clustering commodity data using a preset algorithm based on image feature information to generate image clustering labels for each commodity data. It can be seen that labeling commodity data with image clustering labels through a clustering algorithm reflects the common characteristics of commodity data based on image type data, which can constitute a decision-making basis required for data cleaning and provide decision-making value for the data cleaning process.

[0119] See also Figure 4 In a further embodiment, the step S1400, performing data cleaning on the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels, comprises the following steps:

[0120] Step S1410: Check the consistency of the text clustering label and the image clustering label of the same commodity data:

[0121] In order to determine the value information between the text clustering labels and image clustering labels of the product data relative to the true classification labels, we can first compare the consistency of the text clustering labels and the image clustering labels to see whether they belong to the same label.

[0122] Step S1420: When it is determined that the text clustering label and the image clustering label of the same commodity data are consistent, it is determined that the two clustering labels have contribution value relative to the real classification label, and the real classification label is reset to the text clustering label or the image clustering label:

[0123] When the text clustering labels V of the same product data label and the image cluster label C labelWhen they are consistent, that is, V label =C label , it means that the text feature information and the image feature information of the product data both point to the same label at the same time, which means that the directionality of the two labels has a high degree of credibility. Therefore, it shows that its value information indicates that the two cluster labels have contribution value relative to the correction of the true classification label. In this case, regardless of whether the true label of the product data is consistent with the text cluster label or the image cluster label, the true classification label of the product data can be reset to the text cluster label V. label Or image cluster label C label , that is, T label =V label or T label =C label It can be understood that the reset here includes the reset of V label =C label =T label Under the circumstances of resetting the real classification labels of product data, it also includes V label =C label ≠T label In the alternative embodiment, for V label =C label =T label In this case, since the reset will not cause a substantial change in the real classification label, the reset operation may not be performed, which is also completely equivalent to the present embodiment.

[0124] Step S1430: When it is determined that the text clustering label and the image clustering label of the same commodity data are inconsistent, and one of the text clustering label and the image clustering label is consistent with the real classification label, a text clustering label or an image clustering label with contribution value is determined according to a preset condition, and the real classification label of the commodity data that meets the preset condition is reset with the text clustering label or the image clustering label:

[0125] When the text clustering labels V of the same product data label and the image cluster label C label When they are inconsistent, V label ≠C label, it means that the two types of feature information of the text mode and the image mode of the commodity data cannot lead to the conclusion pointing to the same label. Therefore, in order to avoid mistaken cleaning, in this embodiment, it is necessary to further examine the consistency relationship between the text clustering label or the image clustering label and the real classification label, that is, to determine whether the text clustering label is consistent with the real classification label, and to determine whether the image clustering label is consistent with the real classification label. If either of the two is true, in theory, those skilled in the art can determine whether to reset the text clustering label or the image clustering label to the real classification label of the commodity data based on the credibility of the corresponding commodity title and commodity image in the commodity data.

[0126] The criterion for the credibility of the product title and product image can be implemented as a preset condition by those skilled in the art. For example, when the text clustering label is consistent with the real classification label, that is, T label =V label , and the image clustering label is inconsistent with the true classification label, that is, T label ≠C label , the preset condition can be implemented as judging whether the product title of the corresponding product data reaches a preset length. When the product title reaches the preset length, it can be confirmed that its text clustering label contains valuable information with contribution value. Therefore, the text clustering label can be used to reset the real classification label, otherwise the product data is regarded as an object to be deleted. For another example, when the image clustering label is consistent with the real classification label, that is, T label =C label , while the text clustering label is inconsistent with the true classification label, that is, T label ≠V label The preset condition can be implemented as judging whether the product image of the corresponding product data reaches a preset resolution. When the product image reaches the preset resolution, it can be confirmed that its image clustering label contains valuable information with contribution value. Therefore, the image clustering label can be used to reset the real classification label, otherwise the product data is regarded as an object to be deleted.

[0127] Step S1440: Delete the commodity data whose real classification labels have not been reset from the commodity data set to achieve cleaning:

[0128] After the above processing, for the case where the value information contains contribution value, the real classification labels of the corresponding commodity data have been reset (including the situation equivalent to being reset), among which the real classification labels of the commodity data whose real classification labels have been reset have been corrected, and all other commodity data in the commodity data set whose real classification labels have not been reset are theoretically data that are not helpful for training. Therefore, this part of the data can be completely deleted from the commodity data set to complete data cleaning and process the commodity data set into a purified training set.

[0129] In this embodiment, a more detailed analysis is further performed based on the consistency of the text clustering labels, image clustering labels, and true classification labels of the commodity data, and the granularity of data cleaning is refined. By correctly analyzing the relationship between these labels, the true classification labels of some commodity data are corrected, and some useless commodity data are deleted, a detailed cleaning of the commodity data set is achieved, and the quality of the commodity data in the commodity data set is further improved, which has a significant contribution to improving the prediction accuracy of the classification model.

[0130] See also Figure 5 In a specific embodiment, in step S1430, determining a text clustering label or an image clustering label with contribution value according to a preset condition to reset the real classification label of the commodity data that meets the preset condition includes the following steps:

[0131] Step S1431, counting the average length of the product titles of the product data in the product data set:

[0132] In this embodiment, when determining whether the value information of a text clustering label or an image clustering label relative to a real classification label contains contribution value according to preset conditions, only a single information basis can be used to implement the preset conditions, simplify the logic of analyzing the value information, and improve the decision-making efficiency of data cleaning.

[0133] To this end, this embodiment uses the length information of the product title as the basis for judging the preset conditions. Specifically, the average of the product titles of the product data in the product data set can be first calculated, that is, the character length of each product title is taken, and the sum of these character lengths is divided by the total amount of product data in the product data set to determine it as the average length of the product title.

[0134] Step S1432: When the text clustering label of the commodity data is consistent with the real classification label but the image clustering label is inconsistent with the real classification label, the length of the commodity title of the commodity data exceeds the average length as a preset condition. When the preset condition is met, it is confirmed that the text clustering label has contribution value, and the real classification label of the commodity data is reset with it:

[0135] In this step, when the text clustering label of the commodity data is consistent with the real classification label, but the image clustering label is inconsistent with the real classification label, it is determined whether the length of the commodity title of the commodity data exceeds the average length. When it exceeds the average length, it means that the information contained in the commodity title is relatively rich. Therefore, it is regarded that the preset condition is met. Thus, when the image clustering label is inconsistent with the real classification label and the commodity title corresponding to the text clustering label contains rich information, the credibility of the text clustering label is recognized. Therefore, it is confirmed that the text clustering label has contribution value. Therefore, the real classification label of the commodity data is reset with the text clustering label.

[0136] Step S1433: In the case where the image clustering label of the product data is consistent with the real classification label but the text clustering label is inconsistent with the real classification label, a preset condition is set that the length of the product title of the product data is less than the average length. When the preset condition is met, it is confirmed that the image clustering label has contribution value, and the real classification label of the product data is reset with it:

[0137] In this step, for the case where the image clustering label of the product data is consistent with the real classification label, but the text clustering label is inconsistent with the real classification label, it is determined whether the length of the product title of the product data is less than the average length. When it is less than the average length, it means that the product title contains less information, and the information provided by the product image is more reliable. Therefore, the preset condition is deemed to be met. Therefore, when the text clustering label is inconsistent with the real classification label and the information contained in the product title corresponding to the text clustering label is not reliable enough, the credibility of the image clustering label is recognized. Therefore, it is confirmed that the image clustering label has contribution value. Therefore, the real classification label of the product data is reset with the image clustering label.

[0138] In this embodiment, the average length of the product title of the product data is used as the control threshold in the preset conditions to determine the validity of the value information of the text clustering label and the image clustering label, unifying the implementation logic of the preset conditions in different situations, simplifying the judgment strategy for the value information, and improving the decision efficiency. Since the length of the product title itself also contains the degree of information contribution, making decisions on related labels based on its average length helps to effectively determine whether the product data needs to update its real classification label during the data cleaning process, thereby making a great contribution to improving the labeling quality of the product data and filtering invalid product data.

[0139] See also Figure 6 In an extended embodiment, the multimodal cleaning method for commodity data of the present application further includes the following subsequent steps:

[0140] Step S1500: perform feature splicing on the text feature information and the image feature information of each commodity object in the commodity data set after data cleaning to obtain its corresponding image and text feature information:

[0141] The commodity data set after data cleaning in this application can be directly used as a training data set for a classifier that performs classification tasks. After being processed by the processes disclosed in the aforementioned embodiments of this application, each commodity data has obtained corresponding text feature information and image feature information. Therefore, these deep semantic information can be directly used to train the classifier.

[0142] To this end, the text feature information and image feature information of each product data can be normalized into high-dimensional vectors of the same dimension, and then the two can be concatenated to obtain the corresponding text and image feature information. This image and text feature information integrates the deep semantic information of the product title and product image of the product data, and is the result of representation learning of the product title and product image. Therefore, after full connection, it can be used as the basis for the classifier to perform classification.

[0143] Step S1600: Iteratively train the classifier using the graphic feature information of each commodity data in the commodity data set, and use the real classification label of each commodity data to supervise the predicted classification label during the iterative training process of the commodity data, and implement gradient update on the weight of the classifier according to the loss value between the predicted classification label and the real classification label until the classifier is trained to a convergence state:

[0144] When training the classifier, the classifier is iteratively trained using each commodity data in the commodity data set in turn. In each iteration, the classifier calculates the classification probability of each specific classification label mapped to the classification space based on the image and text feature information, and determines the classification label with the largest classification probability as the predicted classification label. Then, the real classification label of each commodity data after data cleaning is used as the supervision label, and the loss value of the predicted classification label and the real classification label is calculated. Then, the weight of the classifier is gradient updated according to the loss value. Among them, the classification space used by the classifier is actually the classification space composed of the sum of the real classification labels in the commodity data set, and is also the classification space composed of the leaf nodes. The loss function used by the classifier to calculate the loss value can be a cross entropy loss function.

[0145] In the process of iterative training, after executing a training for each product data and calculating the loss value, calculate whether the loss value is close to zero or whether it reaches the preset threshold. When this judgment condition is met, the classifier is considered to be trained to a convergent state and the training can be terminated; otherwise, the classifier is considered to have not converged, and the next product data can be called to continue iterative training of the classifier, so that the classifier continues to approach convergence.

[0146] In this embodiment, the commodity data set cleaned by this application is used as a training data set to train the classifier. Since the commodity data set is purified data, the classifier trained by it will have higher prediction accuracy, and since the real classification labels of each commodity data are also corrected, the training efficiency of the classifier is also significantly improved. When the classifier trained to a convergent state is put into production, it can determine the classification label corresponding to the commodity data with high accuracy based on the text feature information of the commodity title and the image feature information of the commodity image in the commodity data, thereby realizing intelligent automatic classification of the commodity data.

[0147] See also Figure 7 A multimodal cleaning device for commodity data adapted to one of the purposes of the present application comprises: a data determination module 1100, a text clustering module 1200, an image clustering module 1300, and a data cleaning module 1400, wherein the data determination module 1100 is used to determine a commodity data set, the commodity data set comprising a plurality of commodity data carrying real classification labels, the real classification labels being leaf nodes in a preset category tree; the text clustering module 1200 is used to cluster and determine the text clustering labels corresponding to the commodity data according to the text feature information of each commodity data, the total number of text clustering labels being equal to the total number of leaf nodes; the image clustering module 1300 is used to cluster and determine the image clustering labels corresponding to the commodity data according to the image feature information of each commodity data, the total number of image clustering labels being equal to the total number of leaf nodes; the data cleaning module 1400 is used to perform data cleaning on the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels.

[0148] In a further embodiment, the text clustering module 1200 includes: a text extraction submodule, which is used to extract text feature information of the product title of each product data in the product data set by using a pre-trained text feature extraction model; a text clustering submodule, which is used to set the number of categories required for clustering according to the number of leaf nodes, cluster the text feature vectors of the product data, and obtain multiple text-based product data clusters corresponding to the number of categories, each product data cluster containing multiple correspondingly clustered product data; a text annotation submodule, which is used to count the maximum number of real classification labels possessed by the product data in each text-based product data cluster, and determine the maximum number of real classification labels as the text clustering labels of each product data in the product data cluster.

[0149] In a further embodiment, the image clustering module 1300 includes: an image extraction submodule, which is used to extract image feature information of the product image of each product data in the product data set by using a pre-trained image feature extraction model; an image clustering submodule, which is used to set the number of categories required for clustering according to the number of leaf nodes, cluster the image feature vectors of the product data, and obtain multiple image-based product data clusters corresponding to the number of categories, each product data cluster containing multiple correspondingly clustered product data; an image annotation submodule, which is used to count the maximum number of real classification labels of the product data in each image-based product data cluster, and determine the maximum number of real classification labels as the image clustering labels of each product data in the product data cluster.

[0150] In a further embodiment, the data cleaning module 1400 includes: a comparison judgment submodule, which is used to judge the consistency of the text clustering label and the image clustering label of the same product data; a priority reset submodule, which is used to determine that when the text clustering label and the image clustering label of the same product data are consistent, the two clustering labels have contribution value relative to the real classification label, and reset the real classification label to the text clustering label or the image clustering label; a comprehensive reset submodule, which is used to determine that when the text clustering label and the image clustering label of the same product data are inconsistent, and one of the text clustering label and the image clustering label is consistent with the real classification label, determine the text clustering label or the image clustering label with contribution value according to preset conditions, and reset the real classification label of the product data that meets the preset conditions; a purification cleaning submodule, which is used to delete the product data whose real classification labels have not been reset from the product data set to achieve cleaning.

[0151] In a specific embodiment, the comprehensive reset submodule includes: a title length statistics unit, which is used to count the average length of product titles of product data in the product data set; a text label reset unit, which is used to reset the real classification label of the product data by taking the length of the product title of the product data exceeding the average length as a preset condition when the text clustering label of the product data is consistent with the real classification label but the image clustering label is inconsistent with the real classification label; and a picture label reset unit, which is used to reset the real classification label of the product data by taking the length of the product title of the product data being less than the average length as a preset condition when the preset condition is met.

[0152] In an extended embodiment, the multimodal cleaning device for commodity data of the present application also includes: a feature splicing module, which is used to perform feature splicing on the text feature information and image feature information of each commodity object in the commodity data set after data cleaning is completed, so as to obtain its corresponding graphic and text feature information; an iterative training module, which is used to use the graphic and text feature information of each commodity data in the commodity data set to perform iterative training on the classifier, and to supervise the predicted classification label of the commodity data during the iterative training process with the real classification label, and to perform gradient update on the weight of the classifier according to the loss value between the predicted classification label and the real classification label until the classifier is trained to a convergence state.

[0153] In order to solve the above technical problems, the present application also provides a computer device. Figure 8 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a multimodal cleaning method for commodity data. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the multimodal cleaning method for commodity data of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] In this embodiment, the processor is used to execute Figure 7 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the commodity data multimodal cleaning device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0155] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multimodal cleaning method for commodity data of any embodiment of the present application.

[0156] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] In summary, the present application can perform effective data cleaning on massive commodity data based on multiple modalities to obtain high-quality training sets, so that the classifier of the classification task trained by it can obtain a classification prediction effect with higher accuracy.

[0159] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0160] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal cleaning method for commodity data, characterized in that: The steps include: Determine a commodity data set, where the commodity data set includes a plurality of commodity data carrying real classification labels, where the real classification labels are leaf nodes in a preset category tree; Clustering is performed based on the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes; Clustering is performed based on the image feature information of each product data to determine the image clustering label corresponding to the product data, and the total number of image clustering labels is equal to the total number of leaf nodes; Data cleaning is performed on the commodity data in the commodity data set according to value information of text clustering labels and image clustering labels of the same commodity data relative to real classification labels.

2. The method for multimodal cleaning of commodity data according to claim 1, characterized in that: Clustering is performed according to the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes, including the following steps: Using a pre-trained text feature extraction model to extract text feature information of the product title of each product data in the product data set; The number of categories required for clustering is set according to the number of leaf nodes, and the text feature vectors of the commodity data are used for clustering to obtain a plurality of text-based commodity data clusters corresponding to the number of categories, each commodity data cluster containing a plurality of correspondingly clustered commodity data; For each text-based commodity data cluster, the maximum number of true classification labels of the commodity data therein is counted, and the maximum number of true classification labels is determined as the text clustering label of each commodity data in the commodity data cluster.

3. The method for multimodal cleaning of commodity data according to claim 1, characterized in that: Clustering is performed according to the image feature information of each product data to determine the image clustering label corresponding to the product data. The total number of image clustering labels is equal to the total number of leaf nodes, including the following steps: Using a pre-trained image feature extraction model to extract image feature information of the product image of each product data in the product data set; The number of categories required for clustering is set according to the number of leaf nodes, and the image feature vectors of the commodity data are used for clustering to obtain a plurality of commodity data clusters based on the images corresponding to the number of categories, each commodity data cluster containing a plurality of commodity data clustered accordingly; For each image-based product data cluster, the maximum number of true classification labels owned by the product data therein is counted, and the maximum number of true classification labels is determined as the image clustering label of each product data in the product data cluster.

4. The method for multimodal cleaning of commodity data according to claim 1, characterized in that: Data cleaning is performed on the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels, including the following steps: Make consistency judgment on the text clustering labels and image clustering labels of the same product data; When it is determined that the text clustering label and the image clustering label of the same product data are consistent, it is determined that the two clustering labels have contribution value relative to the true classification label, and the true classification label is reset to the text clustering label or the image clustering label; When it is determined that the text clustering label and the image clustering label of the same product data are inconsistent, and one of the text clustering label and the image clustering label is consistent with the real classification label, a text clustering label or an image clustering label with contribution value is determined according to a preset condition, so as to reset the real classification label of the product data that meets the preset condition; The product data whose true classification labels have not been reset are deleted from the product dataset to achieve cleaning.

5. The method for multimodal cleaning of commodity data according to claim 4, characterized in that: Determining text cluster labels or image cluster labels with contribution value according to preset conditions to reset the real classification labels of commodity data that meets the preset conditions includes the following steps: Count the average length of product titles in the product data set; In the case where the text clustering label of the product data is consistent with the real classification label but the image clustering label is inconsistent with the real classification label, the length of the product title of the product data exceeds the average length as a preset condition. When the preset condition is met, it is confirmed that the text clustering label has contribution value, and the real classification label of the product data is reset with it; For the case where the image clustering label of the product data is consistent with the true classification label but the text clustering label is inconsistent with the true classification label, the length of the product title of the product data is less than the average length as a preset condition. When the preset condition is met, it is confirmed that the image clustering label has contribution value, so as to reset the true classification label of the product data.

6. The method for multimodal cleaning of commodity data according to any one of claims 1 to 5, characterized in that: The method also includes the following subsequent steps: After the data cleaning is completed, the text feature information and the image feature information of each commodity object in the commodity data set are concatenated to obtain the corresponding image and text feature information; The classifier is iteratively trained using the graphic feature information of each commodity data in the commodity data set, and the classification label of each commodity data is supervised by the real classification label of the commodity data during the iterative training process to predict the classification label, and the weight of the classifier is gradient updated according to the loss value between the predicted classification label and the real classification label until the classifier is trained to a convergence state.

7. A multimodal cleaning method for commodity data, characterized in that: The steps include: A data determination module, used to determine a commodity data set, wherein the commodity data set includes a plurality of commodity data carrying real classification labels, and the real classification labels are leaf nodes in a preset category tree; A text clustering module is used to cluster the text feature information of each commodity data to determine the text clustering label corresponding to the commodity data, and the total number of text clustering labels is equal to the total number of leaf nodes; An image clustering module is used to cluster the image feature information of each commodity data to determine the image clustering label corresponding to the commodity data, and the total number of image clustering labels is equal to the total number of leaf nodes; The data cleaning module is used to clean the commodity data in the commodity data set according to the value information of the text clustering labels and the image clustering labels of the same commodity data relative to the real classification labels.

8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images