Product data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a product data processing method and device, computer equipment, a storage medium and a computer program product. The method provided by the embodiment of the invention can be applied to the vehicle-mounted field. The method comprises the following steps: acquiring text information for describing a standardized product unit; constructing a product description statement of the standardized product unit based on the text information; extracting statement embedding characteristics of the product description statements by comparing and learning a statement embedding model; searching a similar vector retrieval library for a target similar vector which meets a similar condition with the statement embedding feature; and when the target similar vector is found, determining that the standardized product unit is a repeated product. According to the method, the accuracy of feature extraction can be effectively improved, the similar vectors of the similar targets are searched in the similar vector retrieval library, and the accuracy of repeated judgment of standardized product units is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a product data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology and Internet technology, product management methods have emerged. At present, product management methods can generally complete the electronic and informationization of product information through standardized product units (SPU) or standard stock keeping units (SKU), thereby realizing product management. The characteristics of a product can be described by multiple "attribute | attribute value pairs". Products with exactly the same "attribute | attribute value pairs" can be abstracted into a standardized product unit. On the other hand, these "attributes | attribute value pairs" are solidified in standardized product units and gradually standardized. Based on the product information structure of standardized product units, a variety of applications can be realized, such as the integration of product information with news, comments, and other standardized product units.

[0003] At present, deduplication of standardized product units can be achieved by comparing key attributes and categories. Specifically, a text representation of the standardized product unit can be constructed, and then the embedded feature vector of the text can be extracted through machine learning. The repeated standardized product units can then be identified by comparing similarities. However, this identification method will result in uneven distribution of sentence vectors, sparse distribution of low-frequency sentence vectors, and dense distribution of high-frequency sentence vectors, which will affect the accuracy of product data deduplication. Summary of the invention

[0004] Based on this, it is necessary to provide a product data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of product data deduplication in response to the above technical problems.

[0005] In a first aspect, the present application provides a product data processing method, comprising:

[0006] Get text information used to describe standardized product units;

[0007] constructing a product description statement of the standardized product unit based on the text information;

[0008] Extracting sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and supervised training is performed on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training through the positive samples and the negative samples;

[0009] Searching for a target similarity vector that satisfies a similarity condition with the sentence embedding feature in a similarity vector retrieval library;

[0010] When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

[0011] In a second aspect, the present application further provides a product data processing device, comprising:

[0012] An information acquisition module, used for acquiring text information for describing a standardized product unit;

[0013] A description statement construction module, used to construct a product description statement of the standardized product unit based on the text information;

[0014] An embedding feature extraction module is used to extract sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and uses the positive samples and the negative samples to perform supervised training on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training;

[0015] A feature query module, used to search a similar vector retrieval library for a target similar vector that satisfies a similarity condition with the sentence embedding feature;

[0016] The data processing module is used to determine that the standardized product unit is a duplicate product when the target similarity vector is found.

[0017] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Get text information used to describe standardized product units;

[0019] constructing a product description statement of the standardized product unit based on the text information;

[0020] Extracting sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and supervised training is performed on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training through the positive samples and the negative samples;

[0021] Searching for a target similarity vector that satisfies a similarity condition with the sentence embedding feature in a similarity vector retrieval library;

[0022] When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0024] Get text information used to describe standardized product units;

[0025] constructing a product description statement of the standardized product unit based on the text information;

[0026] Extracting sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and supervised training is performed on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training through the positive samples and the negative samples;

[0027] Searching for a target similarity vector that satisfies a similarity condition with the sentence embedding feature in a similarity vector retrieval library;

[0028] When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

[0029] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0030] Get text information used to describe standardized product units;

[0031] constructing a product description statement of the standardized product unit based on the text information;

[0032] Extracting sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and supervised training is performed on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training through the positive samples and the negative samples;

[0033] Searching for a target similarity vector that satisfies a similarity condition with the sentence embedding feature in a similarity vector retrieval library;

[0034] When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

[0035] The above-mentioned product data processing method, device, computer equipment, storage medium and computer program product first obtain text information used to describe the standardized product unit; and construct the product description sentence of the standardized product unit based on the text information. Thus, the basic data for the duplicate identification of the standardized product unit is constructed, and then the sentence embedding features of the product description sentence are extracted through the contrastive learning sentence embedding model. The contrastive learning sentence embedding model is specifically constructed by constructing positive samples and negative samples from the product description sentences in the historical data, and the unsupervised contrastive learning sentence embedding model obtained by unsupervised training is obtained by supervised training through the positive samples and negative samples; that is, the unsupervised contrastive learning sentence embedding model is supplemented by supervised training, so as to improve the accuracy of sentence embedding feature extraction. Then, the target similarity vector that meets the similarity condition with the sentence embedding feature is searched in the similarity vector retrieval library; when the target similarity vector is found, the current standardized product unit is determined to be a duplicate product, and the recognition process of the standardized product unit duplication is completed. In the present application, after constructing the product description sentences of standardized product units, the sentence embedding features of the product description sentences are extracted based on the contrastive learning sentence embedding model. The contrastive learning sentence embedding model obtained by combining unsupervised training with supervised training can make the sentence embedding features of the product description sentences similar and evenly distributed in the high-dimensional space, effectively improving the accuracy of feature extraction, and then searching for similar target similar vectors in the similarity vector retrieval library to ensure the accuracy of duplicate judgment of standardized product units. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 An application environment diagram of a product data processing method in an embodiment;

[0038] Figure 2 is a flowchart of a product data processing method in one embodiment;

[0039] Figure 3 is a schematic diagram of the composition of a standardized product unit in one embodiment;

[0040] Figure 4 A schematic diagram of a process of unsupervised training of a model in one embodiment;

[0041] Figure 5 A schematic diagram of a supervised training process of a model in one embodiment;

[0042] Figure 6 A schematic diagram of random masking of sample data in one embodiment;

[0043] Figure 7 is a flowchart of a product data processing method in one embodiment;

[0044] Figure 8 is a flowchart of a product data processing method in another embodiment;

[0045] Fig. 9 is a structural block diagram of a product data processing device in one embodiment;

[0046] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The product data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. When the terminal 102 needs to perform deduplication processing on the standardized product unit, so as to perform data analysis related to the standardized product unit, the product data processing of the present application can be used for deduplication processing, so as to classify the same products together and improve the accuracy of the product data analysis. Specifically, the terminal 102 can provide the relevant data of all standardized product units that need to be deduplicated to the server 104, and the server 104 can extract the sentence embedding features of the standardized product unit and establish a similar vector retrieval library. Then, each time the sentence embedding features are extracted, they can be compared with the sentence embedding features stored in the similar vector retrieval library to determine whether it belongs to a repeated standardized product unit. First, the server 104 can obtain text information for describing the standardized product unit; construct a product description sentence of the standardized product unit based on the text information; extract the sentence embedding features of the product description sentence through the contrastive learning sentence embedding model, the contrastive learning sentence embedding model constructs positive samples and negative samples from the product description sentences in the historical data, and supervises the unsupervised contrastive learning sentence embedding model obtained by unsupervised training through the positive samples and negative samples; search for the target similarity vector that meets the similarity condition with the sentence embedding feature in the similarity vector retrieval library; when the target similarity vector is found, the standardized product unit is determined to be a duplicate product. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster consisting of multiple servers.

[0049] In an exemplary embodiment, Figure 2 As shown, a product data processing method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 209. Among them:

[0050] Step 201, obtaining text information for describing a standardized product unit.

[0051] Among them, the Standard Product Unit (SPU) is the smallest unit of commodity information aggregation, a set of reusable and easily searchable standardized information, which describes the characteristics of a product. That is, commodities with the same attribute values ​​and characteristics can be called an SPU. Figure 3 The standardized product unit shown is composed of name, brand, key attributes (effects, scenarios), other attributes (aliases), standard product words, and brand. The uniqueness of the standardized product unit is determined by the brand, standard product words, category, and key attributes. The text information is the text content corresponding to each type of information in the standardized product unit, such as what is the name of the standardized product unit, what is the brand, etc.

[0052] Exemplarily, when the user of terminal 102 wishes to perform deduplication processing on the specified standardized product unit to avoid repeated entry of the standardized product unit, the product data processing method of the present application can be used to identify whether the standardized product unit to be entered is repeated. First, terminal 102 can submit the text information of the standardized product unit to server 104, and server 104 identifies whether the standardized product unit to be entered is a duplicate of the existing data based on the text information submitted by terminal 102. If the current standardized product unit is determined to be a duplicate, the current information entry process can be skipped directly. In one embodiment, the solution of the present application is applicable to the heat management of product units. At this time, the heat of the same standardized product unit can be added by repeatedly identifying the standardized product unit, thereby preventing heat recognition errors.

[0053] Step 203: construct a product description statement of the standardized product unit based on the text information.

[0054] The product description sentence refers to the text content used to describe the standardized product unit. The relevant features of the standardized product unit can be obtained through the product description sentence, and the text features extracted based on the product description sentence can also effectively represent the standardized product unit.

[0055] Exemplarily, after obtaining the text information, some text data in the text information can effectively reflect the product characteristics of the standardized product unit, such as the brand of the product and the key attributes of the product, while some text data cannot reflect the product characteristics, such as the alias of the product. Therefore, it is necessary to extract this part of the text information that can reflect the product characteristics from the text information to construct a product description statement, thereby ensuring the accuracy of deduplication processing based on the product description statement. In a specific embodiment, the text information contains unique information and non-unique information, wherein the unique information is used to indicate the uniqueness of the standardized product unit, and the non-unique information is used for auxiliary explanation. Figure 3The standardized product unit shown is composed of name, brand, key attributes (effects, scenarios), other attributes (aliases), standard product words, and brand. The unique information includes text information such as brand, standard product words, categories, and key attributes. At this time, the unique information can be identified and these text information can be combined to construct a product description statement.

[0056] Step 205, extracting sentence embedding features of product description sentences through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and uses the positive samples and negative samples to perform supervised training on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training.

[0057] Among them, contrastive learning focuses on learning the common features between similar instances and distinguishing the differences between non-similar instances. Compared with generative learning, contrastive learning does not need to pay attention to the cumbersome details of the instance, but only needs to learn to distinguish the data in the feature space at the abstract semantic level, so the model and its optimization become simpler and have stronger generalization ability. The goal of contrastive learning is to learn an encoder that encodes similar data similarly and makes the encoding results of different types of data as different as possible. The scheme of the present application trains a contrastive learning sentence embedding model by contrastive learning, and extracts the embedding features of the product description sentences of the input model through the contrastive learning sentence embedding model. Embedding refers to mapping a large sparse vector to a low-dimensional space that retains semantic relationships, and the embedded features are the obtained feature vectors that can retain the semantic relationships in the original product description sentences. Positive samples refer to sample data that can extract the same sentence embedding features as the input product description sentence after being processed by the model, while negative samples refer to sample data that can extract the embedding features of different sentences from the input product description sentence after being processed by the model. Unsupervised training refers to training the model with unlabeled data, while supervised training refers to training the model with labeled data.

[0058] Exemplarily, the present application can complete the feature encoding processing of the product description sentence by pre-training the contrastive learning sentence embedding model, and obtain the sentence embedding feature through the contrastive learning sentence embedding model obtained by training. For the contrastive learning sentence embedding model, the unsupervised contrastive learning sentence embedding model can be obtained by unsupervised training of the product description sentences in the historical data. Then construct positive samples and negative samples through the product description sentences in the historical data. Supervised training is performed on the unsupervised contrastive learning sentence embedding model based on the annotated positive and negative samples, so as to supplement the unsupervised contrastive learning sentence embedding model and obtain a contrastive learning sentence embedding model that can effectively and accurately extract sentence embedding features. In a specific embodiment, the contrastive learning sentence embedding model can be specifically implemented by the SimCSE (Simple Contrastive Learning of Sentence Embeddings, simple contrastive learning of sentence embeddings) model. Through the contrastive learning method of SimCSE, the semantic expression of the sentence embedding feature can be similar and evenly distributed in the high-dimensional space, thereby ensuring the accuracy of the deduplication processing of the standardized product unit.

[0059] Step 207: Search the similarity vector retrieval library for a target similarity vector that satisfies the similarity condition with the sentence embedding feature.

[0060] Among them, the similarity vector retrieval library refers to a database used to store various types of sentence embedding features. In the process of product data processing, the sentence embedding features of each standardized product unit can be extracted first, and then the extracted and non-repetitive sentence embedding features can be stored in the similarity vector retrieval library. Specifically, during the storage process, the newly extracted sentence embedding features can be compared with the sentence embedding features stored in the similarity vector retrieval library to determine whether there is a target similarity vector in the similarity vector retrieval library that meets the similarity condition with the sentence embedding feature, and then determine whether the newly extracted sentence embedding feature is a repeated sentence embedding feature, and then determine whether the current standardized product unit is a repeated standardized product unit. Satisfying the similarity condition can specifically be that the similarity of the vector is greater than the similarity threshold. By calculating the similarity between the sentence embedding feature and the embedded feature stored in the similarity vector retrieval library, it can be screened whether there is a target similarity vector in the similarity vector retrieval library that meets the similarity condition.

[0061] Exemplarily, the sentence embedding features extracted from the standardized product units can be stored by using a similarity vector retrieval library. When repeated standardized product unit identification is required, the sentence embedding features extracted from the current standardized product unit can be compared and analyzed with the sentence embedding features of other standardized product units stored in the similarity vector retrieval library to find out whether there is a target similarity vector that meets the similarity condition with the current sentence embedding feature. In a specific embodiment, the similarity vector retrieval library can be implemented by the faiss (Facebook AI SimilaritySearch) database, and the faiss database can efficiently search for similar feature vectors by establishing a faiss index, thereby improving the retrieval efficiency in the product data processing process.

[0062] Step 209: When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

[0063] Exemplarily, if a target similarity vector that satisfies the similarity condition with the sentence embedding feature can be found in the similarity vector retrieval library, it means that other standardized product units similar to the current standardized product unit already exist in the similarity vector retrieval library. Therefore, the standardized product unit can be determined to be a duplicate product and used as the duplication determination processing result of the current standardized product unit. If the target similarity vector that satisfies the similarity condition with the sentence embedding feature is not found in the similarity vector retrieval library, it means that the current standardized product unit is a non-duplicate product. At this time, the newly extracted sentence embedding feature can be saved to the similarity vector retrieval library for subsequent duplication identification processing of the standardized product unit.

[0064] The above-mentioned product data processing method first obtains text information used to describe the standardized product unit; and constructs the product description sentence of the standardized product unit based on the text information. In this way, the basic data for the repeated identification of the standardized product unit is constructed, and then the sentence embedding features of the product description sentence are extracted through the contrastive learning sentence embedding model. The contrastive learning sentence embedding model is specifically constructed by constructing positive samples and negative samples from the product description sentences in the historical data, and the unsupervised contrastive learning sentence embedding model obtained by unsupervised training is obtained by supervised training through the positive samples and negative samples; that is, the unsupervised contrastive learning sentence embedding model is supplemented by supervised training, so as to improve the accuracy of sentence embedding feature extraction. Then, the target similarity vector that meets the similarity condition with the sentence embedding feature is searched in the similarity vector retrieval library; when the target similarity vector is found, the current standardized product unit is determined to be a duplicate product, and the recognition process of the standardized product unit duplication is completed. In the present application, after constructing the product description sentences of standardized product units, the sentence embedding features of the product description sentences are extracted based on the contrastive learning sentence embedding model. The contrastive learning sentence embedding model obtained by combining unsupervised training with supervised training can make the sentence embedding features of the product description sentences similar and evenly distributed in the high-dimensional space, effectively improving the accuracy of feature extraction, and then searching for similar target similar vectors in the similarity vector retrieval library to ensure the accuracy of duplicate judgment of standardized product units.

[0065] In an exemplary embodiment, step 203 includes: determining the category text information of each information category based on the information category contained in the text information; identifying the uniqueness category in the information category used to characterize the uniqueness of the standardized product unit; and concatenating the category text information of the uniqueness category and the product title in the text information to obtain a product description statement for the standardized product unit.

[0066] Among them, information categories refer to various product-related data contained in text information, such as name, brand, key attributes, other attributes, standard product words, brand, etc., which are all information categories contained in text information, and the category text information of information categories is the specific information corresponding to these information categories, such as name text information, brand text information, etc. Uniqueness categories are information categories that can characterize the uniqueness of standardized product units. Different standardized product units may have different uniqueness categories. For example, brand, standard product word, category, key attribute can be used as the uniqueness category of a standardized product unit.

[0067] Exemplarily, in order to enable the sentence embedding feature to better express the semantic content of the standardized product unit, thereby ensuring the accuracy of the deduplication processing of the standardized product unit, after obtaining the text information of the standardized product unit. First, determine the category text information of each information category in the text information, and then identify in the category text information which are the unique categories used to constitute the uniqueness of the SPU, so as to combine the category text information of the unique category. However, since the attribute expressions of these category text information are often not well normalized, the scheme of the present application can also splice the names of the specific commodities corresponding to the standardized product unit to enrich the semantic information and obtain a product description statement that can represent the complete information of the standardized product unit. In one of the embodiments, when constructing a product description statement, after the SPU splices the product name, the length of the product description statement may be too long, affecting the accuracy of duplicate recognition. At this time, a length limit can be added to alleviate the problem of recognition errors caused by excessive length differences by stipulating the problem of the product description statement process. In this embodiment, by identifying the unique category contained in the text information, and then splicing the text information of the unique category and the product title to obtain the product description sentence to perform repeated identification of standardized product units, the accuracy of the product description sentence can be effectively guaranteed, thereby improving the accuracy of repeated identification of standardized product units.

[0068] In an exemplary embodiment, the method also includes: obtaining product description sentences in historical data; constructing unsupervised samples based on the product description sentences in the historical data; performing unsupervised training on the initial contrastive learning sentence embedding model through the unsupervised samples to obtain an unsupervised contrastive learning sentence embedding model; constructing positive samples and negative samples based on the product description sentences in the historical data; performing supervised training on the unsupervised contrastive learning sentence embedding model through the positive samples and negative samples to obtain a contrastive learning sentence embedding model.

[0069] Among them, unsupervised samples refer to unlabeled product description sentences, which are used for unsupervised training of the initial contrastive learning sentence embedding model. Positive samples refer to sample data that can extract the same sentence embedding features as the input product description sentence after being processed by the model, while negative samples refer to sample data that can extract different sentence embedding features from the input product description sentence after being processed by the model. Both positive and negative samples can be obtained by labeling product description sentences in historical data.

[0070] Exemplarily, before extracting the sentence embedding features of the product description sentence through the contrastive learning sentence embedding model, it is also necessary to complete the model training process of the contrastive learning sentence embedding model. First, it is necessary to complete the unsupervised training of the initial contrastive learning sentence embedding model. At this time, the relevant information of the standardized product unit in the historical data can be directly obtained, and then the unsupervised sample is constructed based on the product description sentence of the standardized product unit. The unsupervised training of the initial contrastive learning sentence embedding model is completed through the unsupervised sample to obtain the unsupervised contrastive learning sentence embedding model. After the unsupervised training is completed, the obtained unsupervised contrastive learning sentence embedding model cannot accurately extract the sentence embedding features. At this time, the present application can obtain the sample data required for the supervised training process by constructing labeled positive samples and negative samples for the product description sentences in the historical data. For each product description sentence, the corresponding positive sample data and negative sample data can be constructed in a targeted manner. After the construction is completed, the unsupervised contrastive learning sentence embedding model is supervised by these labeled positive samples and negative samples to obtain the contrastive learning sentence embedding model. At this time, the unsupervised contrastive learning sentence embedding model can be modified in a targeted manner through supervised training, thereby effectively improving the accuracy of sentence embedding feature extraction. In this embodiment, the contrastive learning sentence embedding model is completed by combining unsupervised training with supervised training, so that the model problem of the unsupervised training process is solved through supervised training, thereby effectively improving the encoding accuracy of the feature embedding encoding process based on the contrastive learning sentence embedding model.

[0071] In an exemplary embodiment, unsupervised training is performed on an initial contrastive learning sentence embedding model through unsupervised samples to obtain the unsupervised contrastive learning sentence embedding model, including: batch processing of the unsupervised samples; encoding and discarding regularization of target unsupervised samples in each batch of unsupervised samples to construct unsupervised positive samples of the target unsupervised samples; based on unsupervised samples other than the target unsupervised samples in each batch of unsupervised samples, unsupervised negative samples of the target unsupervised samples are constructed; unsupervised training is performed on the initial contrastive learning sentence embedding model based on the target unsupervised samples, the unsupervised positive samples and the unsupervised negative samples to obtain the unsupervised contrastive learning sentence embedding model.

[0072] Among them, batch processing is used to divide all the unsupervised samples into batches (patches), and then use patches to train the model for multiple rounds. Dropout regularization processing is also called dropout processing. Dropout regularization processing means that during forward propagation, the activation value of a neuron in the model stops working with a certain probability p, which can make the model more generalizable because it will not rely too much on certain local features.

[0073] Exemplarily, the unsupervised samples can be processed in batches first, and then the initial contrastive learning sentence embedding model can be trained through the samples of each batch. During the unsupervised training process, the model will automatically construct positive sample data and negative sample data. For positive samples, the target unsupervised samples in each batch of unsupervised samples can be encoded and regularized by the discard method to construct unsupervised positive samples of the target unsupervised samples. For negative samples, the unsupervised negative samples of the target unsupervised samples are directly constructed based on the unsupervised samples other than the target unsupervised samples in each batch of unsupervised samples. Then, the target unsupervised samples, unsupervised positive samples and unsupervised negative samples are constructed to perform unsupervised training on the initial contrastive learning sentence embedding model. Through multiple iterative training and model testing and other processes, the unsupervised contrastive learning sentence embedding model can be obtained. In a specific embodiment, the scheme of the present application selects the SimCSE model as the contrastive learning sentence embedding model. At this time, in the case of Figure 4 In the unsupervised training process shown in FIG, the text data of the target unsupervised sample can be encoded (BERT-transformer layer) and subjected to random perturbation regularization by the double dropout method to obtain an unsupervised positive sample. In the same batch, other unsupervised samples other than the target unsupervised sample can be used as unsupervised negative samples of the target unsupervised sample. Figure 4 As shown, three sentences are input as a batch of data at a time, including the sentences "stickers with a price of 100, brand A and size 30", "wallpaper with a price of 200, brand B and size 70", and "plasters with a price of 20, brand C and size 3". For the sentence "stickers with a price of 100, brand A and size 30", encoding processing and random perturbation processing with 2-time dropout regularization can be performed to obtain the unsupervised positive sample corresponding to the sentence, while the sentences "wallpaper with a price of 200, brand B and size 70" and "plasters with a price of 20, brand C and size 3" can be used as unsupervised negative samples of "stickers with a price of 100, brand A and size 30", and then the unsupervised training of the initial contrastive learning sentence embedding model can be realized through the constructed sample data. In this embodiment, the unsupervised training is completed by constructing positive and negative samples of the unsupervised training process, which can ensure the accuracy and efficiency of the unsupervised training.

[0074] In an exemplary embodiment, constructing negative samples based on product description sentences in historical data includes: obtaining a keyword table of standardized product units; based on the keyword table, identifying key attribute sentences in product description sentences in historical data; and performing random masking on the key attribute sentences in the product description sentences to construct negative samples of the product description sentences.

[0075] Among them, the keyword table can mark which product description sentences in the text information involved in the standardized product unit belong to the key attribute sentences. After randomly masking the key attribute sentences in the product description sentences, the obtained sentences are the negative samples of the current product description sentences. These negative samples are specifically called hard negatives. By constructing negative samples in this way, the model can pay more attention to the key attributes.

[0076] Exemplarily, when constructing positive and negative samples in a supervised training process, in order to effectively allow the model to focus on the key information in the product description sentence, positive samples and negative samples can be constructed by identifying the key attribute sentences. For the construction process of positive and negative samples, it is first necessary to identify which sentences in the product description sentences in the historical data are key attribute sentences and which are non-key attribute sentences. At this time, a keyword table can be pre-constructed, and then the identification can be performed by looking up the table. For example, it can be realized that information such as specifications and brands are stored in the keyword table, and then in the table lookup process, if the current text information is found, it can be used as a key attribute sentence. Through the keyword table, all the key attribute sentences in the product description sentences in the historical data can be identified, and at the same time, the non-key attribute sentences therein are also identified. At this time, the key attribute sentences in the product description sentences can be randomly masked to construct negative samples of the product description sentences. Since the uniqueness of the standardized product unit is determined by these key attribute sentences, after these sentences are randomly masked, the sentences obtained lose the uniqueness of the standardized product unit, so they can be used as negative samples of the product description sentences. Similarly, the non-key attribute sentences in the product description can be randomly masked to construct positive samples of the product description. Masking the constructed positive and negative samples can effectively make the contrastive learning sentence embedding model pay more attention to the key attributes in the text, thereby improving the accuracy of sentence embedding feature extraction. In a specific embodiment, for example, Figure 5 The supervised training process shown in the figure can construct positive and negative samples of existing text. For example, for "the price is 100, the brand is A, and the size is 30 stickers", the key attributes can be masked to construct negative samples, such as constructing "the price is 100, the brand is □, and the size is 30 stickers" as negative samples, or constructing "the price is 100, the product is □, and the size is 30 stickers" as positive samples. Figure 6As shown, for the case of splicing product titles and category texts, negative samples and positive samples can also be constructed by masking key attributes. In this embodiment, by identifying key attribute sentences and non-key attribute sentences in product description sentences, and then constructing positive samples and negative samples by random masking, the trained contrastive learning sentence embedding model can pay more attention to the key attributes in the sentence, thereby improving the accuracy of sentence embedding feature extraction based on the contrastive learning sentence embedding model.

[0077] In an exemplary embodiment, the method further includes: when the target similarity vector is not found, saving the sentence embedding feature to a similarity vector retrieval library.

[0078] Exemplarily, when the target similarity vector that meets the similarity condition with the sentence embedding feature is not found in the similarity vector retrieval library, it means that the sentence embedding feature of the similar standardized product unit is not saved in the database, so the current standardized product unit can be determined as a non-repeating standardized product unit. At this time, in order to prevent the subsequent missed recognition of other standardized product units, the sentence embedding feature can be saved to the similarity vector retrieval library. In this embodiment, by saving the newly recognized sentence embedding feature to the similarity vector retrieval library, the effectiveness of the subsequent repeated recognition of standardized product units can be effectively improved to prevent the occurrence of missed detection.

[0079] In an exemplary embodiment, the method also includes: obtaining historical sentence embedding features of standardized product units in historical data; performing product quantization processing on the historical sentence embedding features to construct a vector index of the historical sentence embedding features; and constructing a similar vector retrieval library based on the historical sentence embedding features and the vector index.

[0080] Among them, product quantization is a vector quantization method that can be used for clustering. Product quantization includes clustering and assigning. It can compress feature vectors by splitting multi-dimensional vectors and then clustering them to construct vector indexes. Vector indexes are used to achieve efficient retrieval of similar target vectors in similar vector retrieval libraries.

[0081] Exemplarily, before searching for the target similarity vector that meets the similarity condition with the sentence embedding feature in the similarity vector retrieval library, it is also necessary to build the similarity vector retrieval library corresponding to the standardized product unit in advance. The similarity vector retrieval library is specifically constructed based on the historical sentence embedding features of the standardized product unit in the historical data, and the process of repeated judgment is also a process for determining whether the newly entered standardized product unit is repeated with the standardized product unit already entered in these historical data. For the establishment process of the similarity vector retrieval library, the historical sentence embedding features can be multiplied and quantized to construct a vector index of the historical sentence embedding features, and the vector index can be associated with the historical sentence embedding features to perform a subsequent search process for repeated feature vectors. After the index is established, the historical sentence embedding features can be stored in the similarity vector retrieval library to complete the database construction process. In this embodiment, the vector index is established by the product quantization method to construct the similarity vector retrieval library, which can effectively ensure the accuracy and efficiency of the feature vector retrieval, thereby ensuring the accuracy of the repeated judgment of the standardized product unit.

[0082] In an exemplary embodiment, step 207 includes: obtaining a query residual vector by performing inverted product quantization processing on the sentence embedding feature; and searching for a target similarity vector that meets similarity conditions with the sentence embedding feature based on the query residual vector and the vector index.

[0083] Among them, the inverted product quantization processing is a further accelerated version of product quantization, which can quickly locate the region of interest through clustering and perform retrieval in the determined region of interest, thereby improving the search efficiency of the target similar vector.

[0084] Exemplarily, for the retrieval process of the target similarity vector in the similarity vector retrieval library, the query residual vector can be obtained by performing inverted product quantization processing on the sentence embedding feature. Through the inverted product quantization processing which is more efficient than product quantization, the query residual vector used for retrieval can be extracted from the original sentence embedding feature, and then based on the query residual vector and the vector index, the target similarity vector that meets the similarity condition with the sentence embedding feature is searched. For the query residual, it means that when calculating the distance between the query residual vector and the vector under a cluster in the similarity vector retrieval library, all vectors will be converted into residuals with the cluster center, so that based on the query residual vector and the vector index, the target similarity vector that meets the similarity condition with the sentence embedding feature can be directly searched from the similarity vector retrieval library.

[0085] In an exemplary embodiment, the method also includes: obtaining a product list; extracting the list sentence embedding features of each standardized product unit in the product list; identifying repeated standardized product units in the product list based on the list sentence embedding features; based on the identified repeated standardized product units, updating the popularity information of each standardized product unit in the product list; based on the updated popularity information, identifying the standardized product units to be recommended in the product list.

[0086] Among them, the popularity information of the standardized product unit refers to the data used to characterize the degree of attention paid to the current standardized product unit. The popularity information may specifically include sales volume, page views, collection volume, and query volume, etc.

[0087] Exemplarily, the scheme of the present application can classify the repeated standardized product units in the product list by first extracting the list sentence embedding features of each standardized product unit in the product list. Then, the repeated standardized product units in the product list are identified by the similarity of the list sentence embedding features. When the repeated standardized product units are identified, the heat information of each standardized product unit in the product list is updated, so that the heat information of these standardized product units is gathered together. By aggregating and updating the heat information of the repeated standardized product units, it is possible to effectively prevent the recommendation sorting errors caused by abnormal heat and improve the success rate of product recommendation. In another embodiment, the scheme of the present application can also be used to recommend products. At this time, the historical browsing information of the target object can be obtained first; then the browsing information sentence embedding features of each standardized product unit in the historical browsing information are extracted; the repeated standardized product units in the historical browsing information are identified based on the browsing information sentence embedding features; based on the identified repeated standardized product units, the standardized product units in the historical browsing information are deduplicated; based on the deduplicated historical browsing information, the interest features of the target object are constructed, and the standardized product units to be recommended of the target object are searched based on the interest features. That is, for the browsing information of the target object, the repeated standardized product units in the browsing information are identified, and then the target object is marked based on the historical browsing information after deduplication, the interest characteristics of the target object are constructed, and the standardized product units to be recommended for the target object are found based on the interest characteristics. By deduplicating the standardized product units in the historical browsing information, the number of products can be reduced, thereby avoiding feature sparseness, thereby better characterizing the interests of the target object and achieving more accurate content recommendation processing. In this embodiment, by deduplicating the standardized product units in the product list to optimize the popularity information of the product list, the effect of product recommendation based on the product list can be effectively improved.

[0088] The present application also provides an application scenario, which applies the above-mentioned product data processing method.

[0089] Specifically, the application of the product data processing method in this application scenario is as follows:

[0090] When the user needs to enter the relevant information of the newly included standardized product unit into the database, it is necessary to determine whether the standardized product unit to be included has been included in the database of standardized product units. At this time, the determination can be made using the product data processing method of the present application.

[0091] The repeated identification process of standardized product units can be referred to Figure 7 As shown, first, the training process of the contrastive learning sentence embedding model needs to be completed. Specifically, the product description sentences in the historical data can be obtained first; unsupervised samples can be constructed based on the product description sentences in the historical data; unsupervised training is performed on the initial contrastive learning sentence embedding model through the unsupervised samples to obtain the unsupervised contrastive learning sentence embedding model; positive samples and negative samples are constructed based on the product description sentences in the historical data; supervised training is performed on the unsupervised contrastive learning sentence embedding model through the positive samples and negative samples to obtain the contrastive learning sentence embedding model. For the unsupervised training process, the unsupervised samples are processed in batches; the target unsupervised samples in each batch of unsupervised samples are encoded and regularized by the discard method to construct unsupervised positive samples of the target unsupervised samples; unsupervised negative samples of the target unsupervised samples are constructed based on the unsupervised samples other than the target unsupervised samples in each batch of unsupervised samples; unsupervised training is performed on the initial contrastive learning sentence embedding model based on the target unsupervised samples, unsupervised positive samples and unsupervised negative samples to obtain the unsupervised contrastive learning sentence embedding model. For sample construction in the supervised training process, we can obtain a keyword table of standardized product units; based on the keyword table, identify key attribute sentences in product description sentences in historical data; perform random masking on key attribute sentences in product description sentences to construct negative samples of product description sentences. At the same time, we perform random masking on non-key attribute sentences in product description sentences to construct positive samples of product description sentences.

[0092] Then, the construction process of the similarity vector retrieval library needs to be completed. At this time, the sentence embedding features of the standardized product units in the existing database can be extracted through the trained contrastive learning sentence embedding model, and then the vector index of these sentence embedding features can be constructed through product quantization processing. When a new standardized product unit needs to be entered, the text information used to describe the standardized product unit can be obtained; the product description sentence of the standardized product unit is constructed based on the text information; and the sentence embedding features of the product description sentence are extracted through the contrastive learning sentence embedding model. The extracted sentence embedding features are then compared with the features stored in the similarity vector retrieval library to find out whether there is a target similarity vector that meets the similarity condition with the sentence embedding features. If it exists, it means that the current standardized product unit is a repeated standardized product unit; if it does not exist, it means that the current standardized product unit is not a repeated standardized product unit. At this time, the sentence embedding features of the standardized product unit can be saved in the similarity vector retrieval library.

[0093] In one embodiment, the complete process of the product data processing method of the present application can be referred to Figure 8 As shown, including:

[0094] Step 801, obtain product description sentences in historical data. Step 803, construct unsupervised samples based on product description sentences in historical data. Step 805, perform unsupervised training on the initial contrastive learning sentence embedding model through unsupervised samples to obtain an unsupervised contrastive learning sentence embedding model. Step 807, construct positive samples and negative samples based on product description sentences in historical data. Step 809, perform supervised training on the unsupervised contrastive learning sentence embedding model through positive samples and negative samples to obtain a contrastive learning sentence embedding model. Step 811, obtain text information used to describe standardized product units. Step 813, determine the category text information of each information category based on the information category contained in the text information. Step 815, identify the uniqueness category in the information category used to characterize the uniqueness of the standardized product unit. Step 817, concatenate the category text information of the uniqueness category and the product title in the text information to obtain the product description sentence of the standardized product unit. Step 819, extract the sentence embedding features of the product description sentence through the contrastive learning sentence embedding model. Step 821, search the similarity vector retrieval library for a target similarity vector that meets the similarity condition with the sentence embedding feature. Step 823, when the target similarity vector is found, determine that the standardized product unit is a duplicate product.

[0095] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0096] Based on the same inventive concept, the embodiment of the present application also provides a product data processing device for implementing the product data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more product data processing device embodiments provided below can refer to the limitations of the product data processing method above, and will not be repeated here.

[0097] In an exemplary embodiment, Fig. 9 As shown, a product data processing device is provided, comprising:

[0098] The information acquisition module 902 is used to acquire text information used to describe the standardized product unit.

[0099] The description sentence building module 904 is used to build a product description sentence of the standardized product unit based on the text information.

[0100] The embedding feature extraction module 906 is used to extract the sentence embedding features of the product description sentence through the contrastive learning sentence embedding model. The contrastive learning sentence embedding model constructs positive samples and negative samples from the product description sentences in the historical data, and uses the positive samples and negative samples to perform supervised training on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training.

[0101] The feature query module 908 is used to search for a target similarity vector that meets similarity conditions with the sentence embedding feature in the similarity vector search library.

[0102] The data processing module 910 is used to determine that the standardized product unit is a duplicate product when a target similarity vector is found.

[0103] In one embodiment, the description statement construction module 904 is specifically used to: determine the category text information of each information category based on the information category contained in the text information; identify the uniqueness category in the information category used to characterize the uniqueness of the standardized product unit; and concatenate the category text information of the uniqueness category and the product title in the text information to obtain a product description statement for the standardized product unit.

[0104] In one embodiment, it also includes a model training module, which is used to: obtain product description sentences in historical data; construct unsupervised samples based on the product description sentences in the historical data; perform unsupervised training on the initial contrastive learning sentence embedding model through the unsupervised samples to obtain an unsupervised contrastive learning sentence embedding model; construct positive samples and negative samples based on the product description sentences in the historical data; perform supervised training on the unsupervised contrastive learning sentence embedding model through the positive samples and negative samples to obtain a contrastive learning sentence embedding model.

[0105] In one embodiment, the model training module is specifically used to: process the unsupervised samples in batches; encode and regularize the target unsupervised samples in each batch of unsupervised samples using the discard method to construct unsupervised positive samples of the target unsupervised samples; construct unsupervised negative samples of the target unsupervised samples based on the unsupervised samples other than the target unsupervised samples in each batch of unsupervised samples; perform unsupervised training on the initial contrastive learning sentence embedding model based on the target unsupervised samples, the unsupervised positive samples and the unsupervised negative samples to obtain the unsupervised contrastive learning sentence embedding model.

[0106] In one embodiment, the model training module is specifically used to: obtain a keyword table of standardized product units; based on the keyword table, identify key attribute statements in product description statements in historical data; perform random masking on key attribute statements in product description statements to construct negative samples of product description statements.

[0107] In one embodiment, the model training module is specifically used to: obtain a keyword table of standardized product units; based on the keyword table, identify non-critical attribute statements in product description sentences in historical data; perform random masking on non-critical attribute statements in product description sentences to construct positive samples of product description sentences.

[0108] In one embodiment, a retrieval library updating module is further included, which is used to: when the target similarity vector is not found, save the sentence embedding feature to the similarity vector retrieval library.

[0109] In one embodiment, it also includes a retrieval library construction module, which is used to: obtain historical sentence embedding features of standardized product units in historical data; perform product quantization processing on the historical sentence embedding features to construct a vector index of the historical sentence embedding features; and construct a similar vector retrieval library based on the historical sentence embedding features and the vector index.

[0110] In one embodiment, a retrieval library construction module is also included, which is used to: obtain a query residual vector by performing inverted product quantization processing on the sentence embedding feature; and search for a target similarity vector that meets similarity conditions with the sentence embedding feature based on the query residual vector and the vector index.

[0111] In one embodiment, the method also includes a first product recommendation module, which is used to: obtain a product list; extract the list sentence embedding features of each standardized product unit in the product list; identify repeated standardized product units in the product list based on the list sentence embedding features; based on the identified repeated standardized product units, update the popularity information of each standardized product unit in the product list; based on the updated popularity information, identify the standardized product units to be recommended in the product list.

[0112] In one embodiment, the method also includes a second product recommendation module, which is used to: obtain historical browsing information of the target object; extract browsing information sentence embedding features of each standardized product unit in the historical browsing information; identify repeated standardized product units in the historical browsing information based on the browsing information sentence embedding features; deduplicate the standardized product units in the historical browsing information based on the identified repeated standardized product units; construct interest features of the target object based on the deduplicated historical browsing information, and search for the standardized product units to be recommended for the target object based on the interest features.

[0113] Each module in the above-mentioned product data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0114] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to product data processing. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a product data processing method is implemented.

[0115] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0117] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0118] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A product data processing method, characterized in that: The method comprises: Get text information used to describe standardized product units; constructing a product description statement of the standardized product unit based on the text information; Extracting sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and supervised training is performed on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training through the positive samples and the negative samples; Searching for a target similarity vector that satisfies a similarity condition with the sentence embedding feature in a similarity vector retrieval library; When the target similarity vector is found, the standardized product unit is determined to be a duplicate product.

2. The method according to claim 1, characterized in that The constructing of the product description statement of the standardized product unit based on the text information includes: Determining category text information of each information category based on the information category included in the text information; Identifying a uniqueness category in the information category for characterizing the uniqueness of the standardized product unit; The category text information of the unique category and the product title in the text information are concatenated to obtain a product description statement of the standardized product unit.

3. The method according to claim 1, characterized in that The method further comprises: Get product description sentences in historical data; constructing unsupervised samples based on product description sentences in the historical data; Performing unsupervised training on the initial contrastive learning sentence embedding model through the unsupervised samples to obtain an unsupervised contrastive learning sentence embedding model; Constructing positive samples and negative samples based on product description sentences in the historical data; The unsupervised contrastive learning sentence embedding model is supervisedly trained by using the positive samples and the negative samples to obtain a contrastive learning sentence embedding model.

4. The method according to claim 3, characterized in that The unsupervised training of the initial contrastive learning sentence embedding model by the unsupervised samples to obtain the unsupervised contrastive learning sentence embedding model comprises: Processing the unsupervised samples in batches; Performing encoding processing and discarding regularization processing on the target unsupervised samples in each batch of unsupervised samples to construct unsupervised positive samples of the target unsupervised samples; Based on the unsupervised samples other than the target unsupervised samples in each batch of unsupervised samples, constructing unsupervised negative samples of the target unsupervised samples; Based on the target unsupervised sample, the unsupervised positive sample and the unsupervised negative sample, an initial contrastive learning sentence embedding model is unsupervisedly trained to obtain an unsupervised contrastive learning sentence embedding model.

5. The method according to claim 3, characterized in that: Constructing negative samples based on product description sentences in the historical data includes: Obtaining a keyword table of the standardized product unit; Based on the keyword table, identifying key attribute sentences in the product description sentences in the historical data; Random masking is performed on key attribute statements in the product description statements to construct negative samples of the product description statements.

6. The method according to claim 3, characterized in that Constructing positive samples based on product description sentences in the historical data includes: Obtaining a keyword table of the standardized product unit; Based on the keyword table, identifying non-key attribute sentences in product description sentences in the historical data; Random masking is performed on non-key attribute statements in the product description statements to construct positive samples of the product description statements.

7. The method according to claim 1, characterized in that The method further comprises: When the target similarity vector is not found, the sentence embedding feature is saved to the similarity vector retrieval library.

8. The method according to claim 1, characterized in that The method further comprises: Obtain historical sentence embedding features of standardized product units in historical data; Performing product quantization processing on the historical sentence embedding features to construct a vector index of the historical sentence embedding features; Based on the historical sentence embedding features and the vector index, a similar vector retrieval library is constructed.

9. The method according to claim 8, characterized in that The step of searching the similarity vector retrieval library for a target similarity vector that satisfies a similarity condition with the sentence embedding feature comprises: Obtaining a query residual vector by performing inverted product quantization processing on the sentence embedding feature; Based on the query residual vector and the vector index, a target similarity vector that satisfies a similarity condition with the sentence embedding feature is searched.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Get product list; extracting a manifest sentence embedding feature of each standardized product unit in the product manifest; identifying repeated standardized product units in the product list based on the list sentence embedding features; Based on the identified duplicate standardized product units, updating the popularity information of each standardized product unit in the product list; Based on the updated popularity information, the standardized product units to be recommended in the product list are identified.

11. The method according to claim 1, characterized in that: The method further comprises: Get the target object's historical browsing information; Extracting the browsing information sentence embedding features of each standardized product unit in the historical browsing information; identifying repeated standardized product units in the historical browsing information based on the browsing information sentence embedding feature; Based on the identified duplicate standardized product units, performing deduplication processing on the standardized product units in the historical browsing information; The interest features of the target object are constructed based on the historical browsing information after deduplication processing, and the standardized product units to be recommended for the target object are searched based on the interest features.

12. A product data processing device, characterized in that: The device comprises: An information acquisition module, used for acquiring text information for describing a standardized product unit; A description statement construction module, used to construct a product description statement of the standardized product unit based on the text information; An embedding feature extraction module is used to extract sentence embedding features of the product description sentence through a contrastive learning sentence embedding model, wherein the contrastive learning sentence embedding model constructs positive samples and negative samples from product description sentences in historical data, and uses the positive samples and the negative samples to perform supervised training on the unsupervised contrastive learning sentence embedding model obtained through unsupervised training; A feature query module, used to search a similar vector retrieval library for a target similar vector that satisfies a similarity condition with the sentence embedding feature; The data processing module is used to determine that the standardized product unit is a duplicate product when the target similarity vector is found.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images