A data processing method and apparatus
By using a dual-tower model in dialogue scenarios, combining the dialogue text of the current round and previous rounds with incoming product category information, the problem of existing models failing to fully utilize context and user information is solved, achieving more accurate category prediction and item recommendation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2023-03-08
- Publication Date
- 2026-07-17
AI Technical Summary
Existing models fail to fully utilize dialogue context information and user-entry category information in dialogue scenarios, resulting in low category prediction accuracy and insufficient contextual semantic understanding capabilities.
A dual-tower model structure is adopted, which combines the user's current dialogue text and historical dialogue text, introduces the user's entry category information, and calculates the weight vector through the threshold weighting module to perform cross-category information and multi-level category prediction. The BERT model is used for pre-training to improve semantic understanding ability.
It improves the accuracy of category prediction and contextual understanding in dialogue scenarios, ensures rapid model convergence, and enhances the accuracy of item recommendations.
Smart Images
Figure CN116467589B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology
[0002] Category prediction is widely used. Taking search scenarios as an example, it determines the user's desired category based on the user's search text, and then recalls and displays items in the corresponding category. Currently, the mainstream solution for category prediction is to transform it into a multi-classification problem, where the total number of categories equals the total number of product categories, and then trains it using machine learning models or deep learning models.
[0003] However, compared to search scenarios, applications in dialogue scenarios are more complex. User query dialogue text is mostly spoken language, with non-standard grammar and a wealth of contextual information. Training existing models directly on user queries across thousands of categories would struggle to achieve accurate recognition. Furthermore, the context of dialogue scenarios contains a significant amount of semantic information, which existing models have not fully utilized, nor have they been specifically optimized for category prediction tasks in dialogue scenarios. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a data processing method and apparatus that can at least solve the problem that existing models are not suitable for dialogue scenarios and do not make full use of dialogue context information and user entry category information.
[0005] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising:
[0006] Obtain a training sample set; wherein each training sample includes the user's incoming product category information and dialogue text, as well as the product category information pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item inquired about when the user clicks to enter the dialog box;
[0007] The product category information and the dialogue text are input into the dual-tower model to determine the product category information of each training sample, and the determined product category information is compared with the labeled product category information.
[0008] The parameters of the dual-tower model are adjusted based on the comparison results and iteratively trained to obtain the trained dual-tower model. The trained dual-tower model is used to determine the corresponding category information based on the input dialogue text and the incoming category information, so as to recommend items based on the determined category information.
[0009] Optionally, inputting the product category information and the dialogue text into the dual-tower model to determine the product category information for each training sample includes:
[0010] The dialogue text is input into a pre-trained model for encoding to obtain a first vector; and the product category information is input into a word embedding module for encoding to obtain a second vector.
[0011] The first vector and the second vector are concatenated to obtain a third vector; and the first vector and the second vector are input into the threshold weighting module to obtain the weight vector between the dialogue text and the incoming product category information.
[0012] The third vector and the weight vector are multiplied to obtain the target vector of each training sample. The target vector is then pooled and fused to obtain the cross-category information and category information of each level for each training sample. The cross-category information represents the matching degree between the dialogue text and the incoming category information.
[0013] Optionally, after obtaining the first vector, the method further includes: inputting the first vector into the attention mechanism module for processing to obtain a fourth vector.
[0014] Optionally, the first vector is divided into two parts. The first part includes multiple vectors, each vector representing a single level category vector in the semantic dimension. The second part is the semantic vector of the dialogue text.
[0015] The second vector is a vector of multiple grade categories in the product category dimension.
[0016] Optionally, the dialogue text is obtained by sequentially concatenating the current round of dialogue text in the dialog box and the historical dialogue text of a first preset number of rounds preceding the current round of dialogue text, according to the chronological order of the dialogue time.
[0017] The incoming product category information is obtained by receiving the current item information inquired by the user when entering the dialog box, extracting the category information at each level from the current item information, and concatenating the category information at each level in descending order of category level.
[0018] Optionally, adjusting the parameters of the dual-tower model and iteratively training based on the comparison results includes: adjusting the parameters of other modules in the dual-tower model other than the pre-trained model when the comparison results are inconsistent.
[0019] Optionally, before inputting the dialogue text into the pre-trained model for encoding, the method further includes:
[0020] Construct a pre-training sample set; where each pre-training sample includes item data and an item category information set, the category information set including category parameters at multiple levels;
[0021] The pre-trained sample set is input into the language model for pre-training to obtain the pre-trained model.
[0022] Optionally, inputting the pre-trained sample set into the language model for pre-training includes:
[0023] For a single item data, each word in the single item data is converted into a word vector form, and a preset percentage of word vectors are selected for masking.
[0024] The masked item data is input into a language model to predict the word vector at each masked position. In response to the existence of one or more masked positions where the predicted word vector differs from the masked word vector, the parameters of the language model are adjusted and training is performed iteratively.
[0025] The masked item data, the correspondence between each item's data and the item category information set, are input into the language model for pre-training.
[0026] Optionally, the method further includes:
[0027] The second preset number of categorical variables in the output layer of the language model are displayed in descending order of category level, showing category information at different levels.
[0028] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided, comprising:
[0029] The sample module is used to acquire a training sample set; wherein, each training sample includes the user's incoming product category information and dialogue text, as well as the product category information that is pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item that the user inquires about when clicking to enter the dialog box;
[0030] The prediction module is used to input the incoming product category information and the dialogue text into the dual-tower model to determine the product category information of each training sample and compare the determined product category information with the labeled product category information.
[0031] The iteration module is used to adjust the parameters of the dual-tower model and iteratively train it based on the comparison results to obtain the trained dual-tower model. The trained dual-tower model is used to determine the corresponding category information based on the input dialogue text and the incoming category information, so as to recommend items based on the determined category information.
[0032] Optionally, the prediction module is used for:
[0033] The dialogue text is input into a pre-trained model for encoding to obtain a first vector; and the product category information is input into a word embedding module for encoding to obtain a second vector.
[0034] The first vector and the second vector are concatenated to obtain a third vector; and the first vector and the second vector are input into the threshold weighting module to obtain the weight vector between the dialogue text and the incoming product category information.
[0035] The third vector and the weight vector are multiplied to obtain the target vector of each training sample. The target vector is then pooled and fused to obtain the cross-category information and category information of each level for each training sample. The cross-category information represents the matching degree between the dialogue text and the incoming category information.
[0036] Optionally, the prediction module is further configured to: input the first vector into the attention mechanism module for processing to obtain a fourth vector.
[0037] Optionally, the first vector is divided into two parts. The first part includes multiple vectors, each vector representing a single level category vector in the semantic dimension. The second part is the semantic vector of the dialogue text.
[0038] The second vector is a vector of multiple grade categories in the product category dimension.
[0039] Optionally, the dialogue text is obtained by sequentially concatenating the current round of dialogue text in the dialog box and the historical dialogue text of a first preset number of rounds preceding the current round of dialogue text, according to the chronological order of the dialogue time.
[0040] The incoming product category information is obtained by receiving the current item information inquired by the user when entering the dialog box, extracting the category information at each level from the current item information, and concatenating the category information at each level in descending order of category level.
[0041] Optionally, the iterative module is used to: adjust the parameters of other modules in the dual-tower model other than the pre-trained model when the comparison results are inconsistent.
[0042] Optionally, the device further includes a pre-training module for:
[0043] Construct a pre-training sample set; where each pre-training sample includes item data and an item category information set, the category information set including category parameters at multiple levels;
[0044] The pre-trained sample set is input into the language model for pre-training to obtain the pre-trained model.
[0045] Optionally, the pre-training module is used for:
[0046] For a single item data, each word in the single item data is converted into a word vector form, and a preset percentage of word vectors are selected for masking.
[0047] The masked item data is input into a language model to predict the word vector at each masked position. In response to the existence of one or more masked positions where the predicted word vector differs from the masked word vector, the parameters of the language model are adjusted and training is performed iteratively.
[0048] The masked item data, the correspondence between each item's data and the item category information set, are input into the language model for pre-training.
[0049] Optionally, the pre-training module is further configured to:
[0050] The second preset number of categorical variables in the output layer of the language model are displayed in descending order of category level, showing category information at different levels.
[0051] To achieve the above objectives, according to another aspect of the present invention, a data processing electronic device is provided.
[0052] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the data processing methods described above.
[0053] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described above.
[0054] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: For dialogue scenarios, historical dialogue text is added on the basis of the user's current dialogue text, which solves the problem that the existing model has poor contextual semantic understanding ability due to only considering the current dialogue text, thereby more accurately understanding the category required by the user; the introduction of user entry category information and the construction of multiple auxiliary prediction tasks ensure that the dual-tower model can correctly learn the correlation between user dialogue text and entry category information, without being led astray by either party, and also ensures the rapid convergence of the dual-tower model.
[0055] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0056] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0057] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;
[0058] Figure 2 This is an example of category prediction in existing e-commerce dialogue scenarios;
[0059] Figure 3 This is a schematic diagram of the dual-tower model structure for category prediction;
[0060] Figure 4 A flowchart illustrating an optional data processing method according to an embodiment of the present invention;
[0061] Figure 5 This is a schematic diagram of the pre-training structure of the BERT model;
[0062] Figure 6 This is a schematic diagram of the main modules of a data processing device according to an embodiment of the present invention;
[0063] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0064] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation
[0065] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0066] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. Where applicable, user personal information has been subjected to de-identification and / or anonymization and / or encryption technical processing.
[0067] See Figure 1 The diagram shows the main flowchart of a data processing method provided by an embodiment of the present invention, which includes the following steps:
[0068] S101: Obtain a training sample set; wherein, each training sample includes the user's incoming product category information and dialogue text, as well as the product category information pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item consulted by the user when clicking to enter the dialog box;
[0069] S102: Input the incoming product category information and the dialogue text into the dual-tower model to determine the product category information of each training sample, and compare the determined product category information with the labeled product category information;
[0070] S103: Adjust the parameters of the dual-tower model according to the comparison results and iteratively train to obtain the trained dual-tower model; wherein, the trained dual-tower model is used to determine the corresponding category information based on the input dialogue text and the incoming category information, so as to recommend items based on the determined category information.
[0071] In the above embodiments, for step S101, see [link to documentation]. Figure 2 The image shows an example of category prediction in an existing e-commerce dialogue scenario. The user enters the chat with the category "down jacket," and the user's query text is "recommend white innerwear." The existing model determines that the user's desired category is T-shirts, and then subsequently recalls and displays items in that category. However, this approach has at least the following problems:
[0072] 1) Since existing models only consider the user's current query and do not take into account the context which contains a lot of semantic information, they have poor ability to understand the semantic context. Furthermore, the training method of directly classifying user queries into thousands of categories makes it difficult to achieve accurate recognition.
[0073] 2) The model has not been specifically optimized for category prediction tasks in dialogue scenarios. For example, the category information required by users is usually related to the category information of the products they need, but the existing model does not utilize this information, resulting in low prediction accuracy.
[0074] To address the problems existing in current technologies, this solution also requires training a model and uses the same two features: user entry product category information and dialogue text. However, it still differs from existing technologies in several ways:
[0075] 1) The existing user query dialogue text is the current round's dialogue text, such as... Figure 2The query in this solution includes not only the current dialogue text in the dialog box, but also the dialogue text from the two rounds preceding it (i.e., a first preset number, the actual number is adjustable, with the first two rounds preferred), and these are concatenated in chronological order. For example, if the user enters other dialogue text before "recommend white inner layer," such as "want to buy a down jacket" or "dark color," the concatenated result will be "want to buy a down jacket," "dark color," and "recommend white inner layer."
[0076] 2) Incoming product category information refers to the product category information displayed when a user clicks to enter the dialog box and inquires about the current item. Similarly, using... Figure 2 For example, with the product "XXX Men's Casual Hooded Down Jacket XXX", after a user clicks the product access link to browse the product details page, if they intend to purchase, they will click the "Contact Seller" option to enter a dialog box with the seller. At this time, the dialog box will automatically send the product access link to the seller so that the seller can understand the user's purchase needs.
[0077] However, sometimes users may purchase multiple items from the same merchant, resulting in multiple product inquiries appearing in the dialog box. To address this, this solution only considers the current or latest item information inquired about when the user enters the dialog box. For example, if a user inquired about "pajamas" yesterday and inquired about the same "pajamas" again today, then the "pajamas" inquired about today will be used as the current item information. Or, if a user inquired about a pair of sneakers ten minutes ago and just inquired about a pair of sportswear, and the user did not leave the dialog box during this process, then the "sneakers" inquired about when entering the dialog box will be used as the current item information. However, if the user left the dialog box and then clicked the link to access the sportswear product details page before inquiring about it again, then the "sportswear" will be used as the current item information.
[0078] Therefore, Figure 2 For example, the product category information for the inquired item "XXX Men's Casual Hooded Down Jacket XXX" will be retrieved. Each product includes multiple levels of categories, which are pre-defined by the system or manually, and will not be detailed here. This solution uses a three-level category system as an example, including Level 1, Level 2, and Level 3 categories, which are concatenated in descending order of category level to obtain "Level 1 Category | Level 2 Category | Level 3 Category", such as "Home Appliances | Large Appliances | Air Conditioner".
[0079] Through the above steps, a training sample set is constructed. Each training sample includes the user's query and incoming product category information. By concatenating the previous two rounds of dialogue text on the basis of the user's current round of dialogue text, the contextual information is used to help the model better understand the user's intent. Incoming product category information is referenced to take advantage of the implicit correlation between the user's query and the incoming product category information, thereby improving the accuracy of category prediction.
[0080] In addition to the information mentioned above, the training samples also rely on human analysis of the dialogue text to determine the user's true intent, such as... Figure 2 The "recommended white inner layer" option, as shown, was analyzed by staff to determine the matching degree between the query and the incoming category information across categories, the first-level category being clothing, the second-level category being tops, and the third-level category being T-shirts. These four pieces of information together constitute the tagged category information.
[0081] For step S102, the model in this scheme is a dual-tower structure. The model training process is divided into two branches: one side inputs the user's dialogue text, and the other side inputs the user's product category information.
[0082] 1. Encode the user query dialogue text using a pre-trained model, and output the first vector V. q The dimension is 768*L1, where 768 is the size of the randomly initialized vector, representing a 768-dimensional vector, and L1 represents the vector length.
[0083] 2. Input the user's product category information into the embedding word embedding module for embedding encoding processing to obtain the second vector V. c The dimension is 768*L2, where L2 represents the vector length.
[0084] 3. Concatenate the first vector V q Second vector V c The third vector V is obtained. i The dimension is 768*(L) 1+ L2)
[0085] 4. Transform the first vector V q Second vector V c The threshold-weighted gate module is passed in to calculate the weight vector g between the user dialogue text and the user's product category information in each training sample. The weight vector g has a dimension of 1*(L1+L2). The weight vector g can also be regarded as the concatenation of g1 and g2, where g1 is the first vector V. q The weights are of dimension 1*L1, and g2 is the second vector V. c The weights are 1*L2 in dimension.
[0086] While incoming product category information can be incorporated into the model in other ways, such as directly concatenating it with the user query, this approach is more prone to confusion compared to the dual-tower input method using a gate, as described in this solution. This can cause the model to fail to correctly distinguish the weights between the user query and the incoming product category information, making it susceptible to being misled by either one. Therefore, the preferred approach is to input the user's incoming product category information into the embedding module.
[0087] 5. Transform the third vector V i Multiplying it by the weight vector g, we get V. i ', with dimensions of 768*(L) 1+ L2), which is the target vector of a single training sample.
[0088] 6. Each training sample outputs four labels, meaning four training tasks are designed: cross-category prediction, third-level category prediction, second-level category prediction, and first-level category prediction. The cross-category prediction task can be viewed as a matching task, focusing on the degree of matching between the user query and the incoming category information, with label = 0 / 1. The other three tasks can be viewed as three classification tasks with hierarchical constraints, focusing on the intent classification of the user query, with label = category ID.
[0089] The target vector obtained in step 5 is input into the Multi-Head-Attention Pooling layer for pooling and fusion processing. Then, it is connected to the fully connected layers of the four tasks to obtain the prediction results for the four tasks. Here, the batch size can be set for batch processing, such as processing 20 training samples at once.
[0090] In addition to the operations described above, for the first vector V obtained in step 1 q It can also be input into the Multi-Head-Attention mechanism module for processing to obtain the weighted vector V. q (i.e., the fourth vector), with dimensions 768*L1, see the overall model structure. Figure 3 As shown in the diagram. The attention mechanism module is used to make the model training focus more on queries containing category information across the three rounds of queries.
[0091] Additionally, weight parameters can be set for different tasks. Training with multiple losses together can easily lead to a bias caused by a loss with a large weight, resulting in negative transfer problems—that is, if the metric for one task increases, the metrics for other tasks decrease. Therefore, in addition to the standard loss summation, weight parameters for different tasks are added, allowing adjustment of the weights between different tasks during training. See the following loss calculation formula:
[0092]
[0093]
[0094] Where, ω k,0 As the initial weights, t represents the update rate, and t represents the epoch.
[0095] For step S103, the dual-tower model can predict the category information of each training sample and compare whether the category information is consistent with the information previously manually labeled. According to the above description, the category information of the training sample includes four parts: cross-category information, third-level category information, second-level category information, and first-level category information. That is, compare whether the information of these four parts is consistent with the four parts of labeled information.
[0096] If they match, it means the model parameters are applicable to the training sample, and other training samples are used to train the model. If they do not match, it means the model parameters are not applicable to the training sample. Since the pre-trained model also needs to be iteratively trained before use, there is no need to change the parameters of the pre-trained model. Only the parameters of other modules in the dual-tower model besides the pre-trained model need to be adjusted, such as the aforementioned embedding word embedding module, gate weighting module, and Multi-Head-AttentionPooling.
[0097] The trained dual-tower model can be used to predict a user's intent based on their query and incoming product category information, and then recommend items based on that intent. For example, if the predicted category is "clothes|tops|T-shirts", then the corresponding T-shirts will be recommended to the user.
[0098] The method provided in the above embodiments, taking into account the characteristics of dialogue scenarios, adds the dialogue text from the previous two rounds to the current round of dialogue text, and introduces user entry category information. It designs a weighted dual-tower model for category prediction to adaptively calculate the weight relationship between the dialogue text and the entry category information, ensuring the rapid convergence of the dual-tower model and improving the ability to understand the context, thereby more accurately predicting the categories needed by the user and improving the accuracy of item recommendations.
[0099] See Figure 4 The diagram illustrates an optional data processing method according to an embodiment of the present invention, including the following steps:
[0100] S401: Construct a pre-training sample set; where each pre-training sample includes item data and an item category information set, the category information set including category parameters at multiple levels;
[0101] S402: For a single item data, each word in the single item data is converted into a word vector form, and a preset percentage of word vectors are selected for masking.
[0102] S403: Input the masked item data into the language model to predict the word vector at each mask position. In response to the existence of one or more mask positions where the predicted word vector is inconsistent with the masked word vector, adjust the parameters of the language model and iteratively train.
[0103] S404: Input the masked item data, the correspondence between each item data and the item category information set into the language model for pre-training;
[0104] S405: Obtain the pre-trained model after training.
[0105] In the above implementation, step S401 includes a pre-training stage and a training stage. This implementation describes the training process of the pre-trained model. Similarly, a pre-training sample set needs to be constructed first. The pre-training sample set includes data on multiple items and category information for each item. Item data can be item title data, information recorded on the item details page, or a combination of both. It can also be other information that contains the main core content of the item. Regarding title data and details page, since title data has the advantage of being simpler and more concise than details page, this implementation prefers to use title data.
[0106] For example, the title data of item 100017981386 is "Brand name 1.5 HP new energy efficiency inverter cooling and heating self-cleaning wall-mounted bedroom air conditioner, trade-in". The category information is first-level category = home appliances, second-level category = large appliances, and third-level category = air conditioner. The category information of these three levels usually depends on human setting.
[0107] For steps S402-S405, pre-training typically involves gathering a large number of low-cost training samples, using a pre-training method to learn their commonalities, then transferring these commonalities to a model specific to a particular task. Fine-tuning is then performed using a small amount of labeled data from the relevant domain. In this way, the model only needs to learn the specific parts of the task based on these commonalities. Pre-training, in terms of model parameters, can be understood as having all the model parameters pre-trained for a suitable task, eliminating the need to train all parameters from scratch. However, some parameters may not be suitable for the current task, so slight modifications (fine-tuning) are needed to achieve better results, significantly reducing learning time.
[0108] Therefore, this solution also adopts a model pre-training approach, using the BERT (Bidirectional Encoder Representations from Transformers) model as the training object. The BERT model is a bidirectional Transformer encoder, typically used in two stages: 1. Pre-training stage: training the language model and outputting word vectors; 2. Fine-tuning stage: fine-tuning the language model to make it suitable for downstream tasks such as classification and reading comprehension.
[0109] Training the BERT model involves two tasks, which can be performed simultaneously or sequentially; the order of these tasks is not restricted in this scheme.
[0110] 1) The MLM (Masked Language Model) task is one of the learning tasks in BERT pre-training. It involves randomly selecting a portion of words from the input text, masking them, and then training the model to correctly predict the masked words. 2) The NSP (Next Sentence Prediction) task is another learning task in BERT pre-training. Given two sentences, it determines whether sentence B is the next sentence of sentence A. Positive samples are obtained from the original corpus, and negative samples are randomly negative sampled. Therefore, each word in the item data is converted into word vector form, and then 15% of the words are randomly selected and masked. The masked item data is then used as input text into the BERT model to predict the masked words.
[0111] For example, the title data is "New Energy Efficiency Variable Frequency Cooling and Heating Self-Cleaning". After some of its word vectors are masked, they become "New Energy Efficiency Word Vector Variable Frequency Cooling and Heating Word Vector Y Word Vector". After being input into the BERT model, the masked position is used to predict "Self-Cleaning Word Vector".
[0112] 2) As another pre-training task, the category prediction task is performed. The correspondence between the masked item data, the original item data, and the item category information set is input into the BERT model for training, in order to train the correspondence between the masked item data and the item category information set.
[0113] Through the above pre-training operations, the BERT model not only possesses the semantic understanding ability of a bidirectional language model, but can also predict the correspondence between category information sets, category information, and dialogue context. Therefore, the final prediction result using the BERT model includes two parts: one part consists of vectors of multiple levels of categories, such as first-level category vectors, second-level category vectors, and third-level category vectors; the other part is text semantic vectors, representing the correspondence between dialogue text and category information.
[0114] See details Figure 5 As shown, the first three cls (classification, which can be understood as being used for downstream classification tasks) bit vectors of the BERT model output layer are used to predict the first-level, second-level, and third-level categories, respectively. The latter half of the BERT model outputs cross-category information. For example, based on "Brand Name 1.5 HP New Energy Efficiency Inverter Self-Cleaning Wall-Mounted Bedroom Air Conditioner, Trade-in Program", the predicted cross-category information is "inverter".
[0115] Similarly, for the vector obtained after encoding using the embedding word embedding module, which includes vectors for three levels of categories, since the processing object is a concatenation of categories, there is only one vector here. Accordingly, the final target vector consists of three parts: the first half is three vectors for the three levels of categories in the semantic dimension, the middle part is the text semantic vector, and the second half is one vector for the three levels of categories in the inbound category dimension.
[0116] It should be noted that while the BERT model can be replaced by conventional deep learning models in practice, such as LSTM (Long Short-Term Memory) and TextCNN, these models, compared to BERT, are not pre-trained for e-commerce dialogue scenarios, resulting in poorer semantic understanding and impacting category prediction performance. This solution specifically designs a pre-training task, making it more suitable for category prediction in e-commerce dialogue scenarios.
[0117] The method provided in the above embodiments designs a model pre-training process for dialogue scenarios, performs masking processing on item data and performs unsupervised pre-training, improves the contextual understanding ability of the pre-training module, and uses the correspondence between item data and corresponding categories as input for training, so that the pre-training module can correctly learn the correlation between the two.
[0118] See Figure 6 The diagram shows a schematic representation of the main modules of a data processing apparatus 600 provided in an embodiment of the present invention, including:
[0119] The sample module 601 is used to acquire a training sample set; wherein, each training sample includes the user's incoming product category information and dialogue text, as well as the product category information that is pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item that the user inquires about when clicking to enter the dialog box;
[0120] Prediction module 602 is used to input the incoming product category information and the dialogue text into the dual-tower model to determine the product category information of each training sample and compare the determined product category information with the labeled product category information.
[0121] The iteration module 603 is used to adjust the parameters of the dual-tower model and iteratively train it according to the comparison results to obtain the trained dual-tower model; wherein, the trained dual-tower model is used to determine the corresponding category information according to the input dialogue text and the incoming category information, so as to recommend items according to the determined category information.
[0122] In the apparatus of this invention, the prediction module 602 is used for:
[0123] The dialogue text is input into a pre-trained model for encoding to obtain a first vector; and the product category information is input into a word embedding module for encoding to obtain a second vector.
[0124] The first vector and the second vector are concatenated to obtain a third vector; and the first vector and the second vector are input into the threshold weighting module to obtain the weight vector between the dialogue text and the incoming product category information.
[0125] The third vector and the weight vector are multiplied to obtain the target vector of each training sample. The target vector is then pooled and fused to obtain the cross-category information and category information of each level for each training sample. The cross-category information represents the matching degree between the dialogue text and the incoming category information.
[0126] In the apparatus of the present invention, the prediction module 602 is further configured to: input the first vector into the attention mechanism module for processing to obtain the fourth vector.
[0127] In the device of this invention, the first vector is divided into two parts. The first part includes multiple vectors, each vector representing a single level category vector of semantic dimension. The second part is the semantic vector of the dialogue text.
[0128] The second vector is a vector of multiple grade categories in the product category dimension.
[0129] In the device of this invention, the dialogue text is obtained by sequentially splicing the current round of dialogue text in the dialog box and the historical dialogue text of a first preset number of rounds preceding the current round of dialogue text, according to the chronological order of the dialogue time.
[0130] The incoming product category information is obtained by receiving the current item information inquired by the user when entering the dialog box, extracting the category information at each level from the current item information, and concatenating the category information at each level in descending order of category level.
[0131] In the apparatus of the present invention, the iteration module 603 is used to: adjust the parameters of other modules in the dual-tower model other than the pre-trained model when the comparison results are inconsistent.
[0132] In the apparatus of this invention, the apparatus further includes a pre-training module, used for:
[0133] Construct a pre-training sample set; where each pre-training sample includes item data and an item category information set, the category information set including category parameters at multiple levels;
[0134] The pre-trained sample set is input into the language model for pre-training to obtain the pre-trained model.
[0135] In the apparatus of this invention, the pre-training module is used for:
[0136] For a single item data, each word in the single item data is converted into a word vector form, and a preset percentage of word vectors are selected for masking.
[0137] The masked item data is input into a language model to predict the word vector at each masked position. In response to the existence of one or more masked positions where the predicted word vector differs from the masked word vector, the parameters of the language model are adjusted and training is performed iteratively.
[0138] The masked item data, the correspondence between each item's data and the item category information set, are input into the language model for pre-training.
[0139] In the apparatus of this invention, the pre-training module is further configured to:
[0140] The second preset number of categorical variables in the output layer of the language model are displayed in descending order of category level, showing category information at different levels.
[0141] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.
[0142] Figure 7 An exemplary system architecture 700 to which embodiments of the present invention can be applied is shown, including terminal devices 701, 702, 703, network 704, and server 705 (this is merely an example).
[0143] Terminal devices 701, 702, and 703 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc.
[0144] Network 704 is a medium used to provide a communication link between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0145] Server 705 can be a server that provides various services. It should be noted that the methods provided in the embodiments of the present invention are generally executed by server 705, and correspondingly, the devices are generally set in server 705.
[0146] It should be understood that Figure 7The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0147] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0148] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0149] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.
[0150] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.
[0151] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0153] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a sample module, a prediction module, and an iteration module. The names of these modules do not necessarily limit the module itself; for example, the iteration module can also be described as an "iterative training module."
[0154] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the data processing methods described above.
[0155] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A data processing method, characterized in that, include: Obtain a training sample set; wherein each training sample includes the user's incoming product category information and dialogue text, as well as the product category information pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item inquired about when the user clicks to enter the dialog box; The incoming product category information and the dialogue text are input into a dual-tower model to determine the product category information of each training sample. The determined product category information is compared with the labeled product category information. This includes: inputting the dialogue text into a pre-trained model for encoding to obtain a first vector; inputting the incoming product category information into a word embedding module for encoding to obtain a second vector; concatenating the first vector and the second vector to obtain a third vector; inputting the first vector and the second vector into a threshold weighting module to obtain a weight vector between the dialogue text and the incoming product category information; multiplying the third vector and the weight vector to obtain a target vector for each training sample; and performing pooling fusion on the target vector to obtain cross-category information and product category information at each level for each training sample. The cross-category information represents the matching degree between the dialogue text and the incoming product category information. The parameters of the dual-tower model are adjusted based on the comparison results and iteratively trained to obtain the trained dual-tower model. The trained dual-tower model is used to determine the corresponding category information based on the input dialogue text and the incoming category information, so as to recommend items based on the determined category information.
2. The method according to claim 1, characterized in that, After obtaining the first vector, the method further includes: inputting the first vector into the attention mechanism module for processing to obtain the fourth vector.
3. The method according to claim 1 or 2, characterized in that, The first vector is divided into two parts. The first part includes multiple vectors, each vector representing a single level category vector in the semantic dimension. The second part is the semantic vector of the dialogue text. The second vector is a vector of multiple grade categories in the product category dimension.
4. The method according to claim 1 or 2, characterized in that, The dialogue text is obtained by sequentially piecing together the current dialogue text in the dialog box and the historical dialogue text of the first preset number of rounds preceding the current dialogue text, according to the chronological order of the dialogue time. The incoming product category information is obtained by receiving the current item information inquired by the user when entering the dialog box, extracting the category information at each level from the current item information, and concatenating the category information at each level in descending order of category level.
5. The method according to claim 1, characterized in that, The step of adjusting the parameters of the dual-tower model and iteratively training based on the comparison results includes: adjusting the parameters of other modules in the dual-tower model other than the pre-trained model when the comparison results are inconsistent.
6. The method according to claim 1 or 5, characterized in that, Before inputting the dialogue text into the pre-trained model for encoding, the method further includes: Construct a pre-training sample set; where each pre-training sample includes item data and an item category information set, the category information set including category parameters at multiple levels; The pre-trained sample set is input into the language model for pre-training to obtain the pre-trained model.
7. The method according to claim 6, characterized in that, The step of inputting the pre-trained sample set into the language model for pre-training includes: For a single item data, each word in the single item data is converted into a word vector form, and a preset percentage of word vectors are selected for masking. The masked item data is input into a language model to predict the word vector at each masked position. In response to the existence of one or more masked positions where the predicted word vector differs from the masked word vector, the parameters of the language model are adjusted and training is performed iteratively. The masked item data, the correspondence between each item's data and the item category information set, are input into the language model for pre-training.
8. The method according to claim 6, characterized in that, The method further includes: The second preset number of categorical variables in the output layer of the language model are displayed in descending order of category level, showing category information at different levels.
9. A data processing apparatus, characterized in that, include: The sample module is used to acquire a training sample set; wherein, each training sample includes the user's incoming product category information and dialogue text, as well as the product category information that is pre-labeled on the dialogue text, wherein the incoming product category information is the product category information of the item that the user inquires about when clicking to enter the dialog box; The prediction module is used to input the incoming product category information and the dialogue text into a dual-tower model to determine the product category information of each training sample, and compare the determined product category information with the labeled product category information. This includes: inputting the dialogue text into a pre-trained model for encoding to obtain a first vector; inputting the incoming product category information into a word embedding module for encoding to obtain a second vector; concatenating the first vector and the second vector to obtain a third vector; inputting the first vector and the second vector into a threshold weighting module to obtain a weight vector between the dialogue text and the incoming product category information; multiplying the third vector and the weight vector to obtain a target vector for each training sample; and performing pooling fusion on the target vector to obtain cross-category information and product category information at each level for each training sample. The cross-category information represents the matching degree between the dialogue text and the incoming product category information. The iteration module is used to adjust the parameters of the dual-tower model and iteratively train it based on the comparison results to obtain the trained dual-tower model. The trained dual-tower model is used to determine the corresponding category information based on the input dialogue text and the incoming category information, so as to recommend items based on the determined category information.
10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.
11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.