Data Processing Method, Apparatus, Device, and Medium

Abstract

Embodiments of the present application provide a data processing method, apparatus, device, and medium. The method can be applied in the field of natural language processing to enhance the effectiveness of item embedding representations. The method includes: obtaining a first text representation of each item and a second text representation of the historical item sequence of each object; performing text prediction on a first training sample according to a pre-trained language model to obtain a prediction probability corresponding to a hidden position in the first training sample; performing item comparison on the first training sample and a first positive sample according to the pre-trained language model to obtain a first sample similarity; training the pre-trained language model according to the prediction probability, the hidden position in the first training sample, and the first sample similarity; adjusting the trained pre-trained language model according to a second training sample to obtain an adjusted pre-trained language model; and the adjusted pre-trained language model is used to initialize the embedding representation of the item.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a data processing method, apparatus, device, and medium. Background Art

[0002] As an important task in the recommendation system, sequential recommendation models the historical item interaction sequences of each object (such as the sequence of purchased items) to learn the interest changes of each object, and then can predict the next item to be interacted with by each object. In the current sequential recommendation modeling scenario, the historical item interaction sequences are used to train the sequential recommendation model. After training, the sequential recommendation model can extract the feature representations of object interests, and these feature representations can be combined with the item embedding information corresponding to each item to recommend the next item to be interacted with by the object. It can be seen that the quality of the item embedding information directly affects the recommendation effect of the trained sequential recommendation model. Currently, the item embedding information is usually initialized randomly, or initialized with the vectors obtained by vectorizing the item text representation using a network model. Since there is a huge representational difference between the item text representation and the historical item interaction sequences, the effectiveness of the item embedding information is affected. Summary of the Invention

[0003] Embodiments of this application provide a data processing method, apparatus, device, and medium, which can make full use of the prior knowledge in the pre-trained language model, thereby enhancing the effectiveness of item embedding representations.

[0004] On the one hand, an embodiment of this application provides a data processing method, including:

[0005] Obtain the first text representation corresponding to each item in the item set, and according to the first text representation, obtain the second text representation corresponding to the historical item sequence of each object in the object set;

[0006] Generate a first training sample according to the second text representation, and perform text prediction on the first training sample using the pre-trained language model to obtain the prediction probability corresponding to the hidden position in the first training sample; the first training sample includes positive samples and negative samples;

[0007] According to the pre-trained language model, compare the items in the first training sample and the first positive sample corresponding to the first training sample to obtain the first sample similarity;

[0008] Train the pre-trained language model according to the prediction probability, the hidden position in the first training sample, and the first sample similarity to obtain the trained pre-trained language model;

[0009] Generate a second training sample according to the second text representation, and compare the second training sample with the corresponding second positive sample of the second training sample based on the pre-trained language model after training to obtain a second sample similarity; the second training sample does not include positive samples;

[0010] Adjust the pre-trained language model after training according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of the item.

[0011] One aspect of the embodiments of the present application provides a data processing device, including:

[0012] A text acquisition module, configured to acquire a first text representation corresponding to each item in the item set, and according to the first text representation, acquire a second text representation corresponding to the historical item sequence of each object in the object set;

[0013] A text prediction module, configured to generate a first training sample according to the second text representation, and perform text prediction on the first training sample based on the pre-trained language model to obtain a prediction probability corresponding to the hidden position in the first training sample; the first training sample includes positive samples and negative samples;

[0014] A first item comparison module, configured to compare the first training sample with the corresponding first positive sample of the first training sample based on the pre-trained language model to obtain a first sample similarity;

[0015] A model pre-training module, configured to train the pre-trained language model according to the prediction probability, the hidden position in the first training sample, and the first sample similarity to obtain a pre-trained language model after training;

[0016] A second item comparison module, configured to generate a second training sample according to the second text representation, and compare the second training sample with the corresponding second positive sample of the second training sample based on the pre-trained language model after training to obtain a second sample similarity; the second training sample does not include positive samples;

[0017] A model adjustment module, configured to adjust the pre-trained language model after training according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of the item.

[0018] Among them, the text acquisition module acquires a first text representation corresponding to each item in the item set, and according to the first text representation, acquires a second text representation corresponding to the historical item sequence of each object in the object set, and is used to perform the following steps:

[0019] Obtain the item type, item identifier, and item name corresponding to each item in the item set, and combine the item type, item identifier, and item name corresponding to the same item to obtain the first text representation corresponding to each item;

[0020] Obtain the historical item sequence corresponding to each object in the object set, and sort the items included in the historical item sequence in descending order according to the interaction time of the items included in the historical item sequence to obtain the sorted historical item sequence;

[0021] Concatenate the first text representations corresponding to the items included in the sorted historical item sequence to obtain the second text representation corresponding to the historical item sequence of each object.

[0022] Among them, the text prediction module generates the first training sample according to the second text representation for performing the following steps:

[0023] Determine the hidden position in the second text representation according to the text selection ratio, and determine the text at the hidden position as the candidate text representation;

[0024] Perform a hiding process on the candidate text representation in the second text representation to obtain an initial sample, and add a flag text to the initial sample to obtain the first training sample.

[0025] Among them, the text prediction module performs a hiding process on the candidate text representation in the second text representation to obtain an initial sample for performing the following steps:

[0026] Adopt the first hiding probability and the second hiding probability to perform a hiding process on the candidate text representation in the second text representation to obtain an initial sample;

[0027] Among them, the first hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a mask, and the second hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a random text representation.

[0028] Among them, the text prediction module performs text prediction on the first training sample according to the pre-trained language model to obtain the prediction probability corresponding to the hidden position in the first training sample for performing the following steps:

[0029] Input the first training sample into the pre-trained language model, and perform encoding processing on the first training sample through the pre-trained language model to obtain the first sample embedding representation corresponding to the first training sample;

[0030] Perform text prediction on the hidden position in the first training sample according to the first sample embedding representation to obtain the prediction probability corresponding to the hidden position in the first training sample.

[0031] Among them, the first item comparison module compares the first training sample and the first positive sample corresponding to the first training sample according to the pre-trained language model to obtain the first sample similarity, which is used to perform the following steps:

[0032] Determine the first text representation of the subsequent interaction items corresponding to the first training sample as the first positive sample; the first training sample and the first positive sample belong to the historical interaction items of the same object;

[0033] Input the first positive sample into the pre-trained language model, and encode the first positive sample through the encoder in the pre-trained language model to obtain the second sample embedding representation corresponding to the first positive sample;

[0034] Obtain the first object embedding representation corresponding to the object associated with the first positive sample, and obtain the first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample.

[0035] Among them, the number of the first training samples is M, and M is a positive integer;

[0036] The first item comparison module obtains the first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample, which is used to perform the following steps:

[0037] Obtain the first feature similarity between the first object embedding representation and the second sample embedding representation, perform an exponential operation on the first feature similarity to obtain the first similarity candidate value;

[0038] Obtain the second feature similarity between the first object embedding representation and the first sample embedding representations corresponding to each first training sample, perform an exponential operation on the second feature similarity to obtain the second candidate similarity values associated with each first training sample;

[0039] Accumulate the second candidate similarity values associated with each first training sample to obtain the cumulative similarity value, and determine the first sample similarity according to the ratio between the first similarity candidate value and the cumulative similarity value.

[0040] Among them, the number of hidden positions in the first training sample is N, and N is a positive integer;

[0041] The model pre-training module trains the pre-trained language model according to the prediction probability, the hidden positions in the first training sample, and the first sample similarity to obtain the trained pre-trained language model, which is used to perform the following steps:

[0042] Perform a logarithmic operation on the prediction probabilities corresponding to the N hidden positions to obtain the probability logarithm values of each hidden position, and accumulate the probability logarithm values of each hidden position to obtain the masked language modeling loss;

[0043] Determine the model training loss corresponding to the pre-trained language model according to the masked language modeling loss and the first sample similarity;

[0044] Iteratively train the network parameters of the pre-trained language model according to the model training loss, and stop training until the model training loss meets the training end condition, and obtain the trained pre-trained language model.

[0045] Among them, the second item comparison module generates a second training sample according to the second text representation, and performs item comparison on the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model, and obtains the second sample similarity for performing the following steps:

[0046] Add a flag text to the second text representation to obtain a second training sample, and determine the second positive sample corresponding to the second training sample;

[0047] Obtain the third sample embedding representation corresponding to the second training sample through the pre-trained language model, and obtain the fourth sample embedding representation corresponding to the second positive sample through the pre-trained language model;

[0048] Obtain the second object embedding representation corresponding to the object associated with the second positive sample, and obtain the second sample similarity according to the second object embedding representation, the third sample embedding representation, and the fourth sample embedding representation.

[0049] Among them, the data processing device may further include:

[0050] An item input text acquisition module, configured to add a flag text to the first text representation corresponding to each item in the item set to obtain an item input text corresponding to each item in the item set;

[0051] An item embedding representation acquisition module, configured to input the item input text corresponding to item a in the item set into the adjusted pre-trained language model, and obtain the item embedding representation corresponding to item a through the adjusted pre-trained language model;

[0052] An item embedding table generation module, configured to obtain the item embedding representations corresponding to each item in the item set, and add the item embedding representations corresponding to each item to the item embedding table.

[0053] Among them, the data processing device may further include:

[0054] A training sample generation module, configured to generate a third training sample of the sequence recommendation model according to the item embedding representations in the item embedding table and the historical item sequences corresponding to each object in the object set;

[0055] A recommended model training module, configured to train a sequence recommendation model according to a third training sample to obtain a trained sequence recommendation model;

[0056] Among them, the training of the sequence recommendation model includes a first stage and a second stage. The first stage is used to train the sequence modeling task and pause the training of the item embedding representation. The second stage is used to train the item embedding representation and pause the training of the sequence modeling task. The trained sequence recommendation model is used for item recommendation.

[0057] In one aspect of the embodiments of the present application, a computer device is provided, including a memory and a processor. The memory is connected to the processor. The memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in the above-mentioned aspect of the embodiments of the present application.

[0058] In one aspect of the embodiments of the present application, a computer-readable storage medium is provided. A computer program is stored in the computer-readable storage medium. The computer program is suitable for being loaded and executed by a processor so that a computer device with a processor executes the method provided in the above-mentioned aspect of the embodiments of the present application.

[0059] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device executes the method provided in the above-mentioned aspect.

[0060] In the embodiments of the present application, the first text representation corresponding to each item in the item set is obtained, and the first text representations corresponding to the items included in the historical item sequence of each object are concatenated to obtain the second text representation corresponding to the historical item sequence. According to the second text representation, a first training sample for training a pre-trained language model and a second training sample for adjusting the trained pre-trained language model are constructed. Based on the first training sample, the pre-trained language model is trained by using a dual-task strategy of masked language modeling (text prediction) and item comparison to obtain a trained pre-trained language model. Based on the second training sample, the trained pre-trained language model is adjusted by using the item comparison strategy to obtain an adjusted pre-trained language model. The adjusted pre-trained language model can provide an initial item embedding representation for sequence recommendation, and can make full use of the prior knowledge of the pre-trained language model to extract the sequence features in the historical item sequence, thereby enhancing the effectiveness of the item embedding representation. Description of the Drawings

[0061] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0062] Figure 1 It is a schematic structural diagram of a network architecture provided by an embodiment of the present application;

[0063] Figure 2 It is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0064] Figure 3 It is a schematic diagram of text representation construction provided by an embodiment of the present application;

[0065] Figure 4 It is a schematic diagram of the training of a pre-trained language model provided by an embodiment of the present application;

[0066] Figure 5 It is a schematic flowchart of the initialization of item embedding based on a pre-trained language model provided by an embodiment of the present application;

[0067] Figure 6 It is a schematic diagram of the training of a sequence recommendation task provided by an embodiment of the present application;

[0068] Figure 7 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

[0069] Figure 8 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0071] For ease of understanding, the following first describes the basic technical concepts related to the embodiments of the present application:

[0072] Natural Language Processing (NLP): Natural language processing is an important direction in the fields of computer science and artificial intelligence (AI). It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing is a science that combines linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, that is, the language people use in daily life, so it has a close connection with the research of linguistics. Natural language processing technologies usually include text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0073] Among them, the embodiments of the present application specifically relate to text processing under natural language processing. In the sequence recommendation scenario, a text representation of an item is constructed. By performing text encoding on the text representation of the item, an item embedding representation corresponding to each item can be obtained. This item embedding representation can be used as the initial embedding representation of the item in the sequence recommendation task, so as to enhance the effectiveness of the item embedding representation.

[0074] Please refer to Figure 1 , Figure 1 FIG. is a schematic structural diagram of a network architecture provided by an embodiment of the present application. The network architecture may include a server 10d and a terminal cluster. The terminal cluster may include one or more terminal devices, and the number of terminal devices included in the terminal cluster is not limited here. As Figure 1 shown, the terminal cluster may specifically include terminal devices 10a, 10b, and terminal device 10c, etc.; all terminal devices in the terminal cluster (for example, may include terminal devices 10a, 10b, and terminal device 10c, etc.) can be network-connected to the server 10d, so that each terminal device can perform data interaction with the server 10d through this network connection.

[0075] The terminal devices of the terminal cluster may include electronic devices such as smart phones, tablet computers, laptop computers, palmtop computers, mobile internet devices (MID), wearable devices (such as smart watches, smart bracelets, etc.), intelligent voice interaction devices, smart home appliances (such as smart TVs, etc.), vehicle-mounted devices, and aircraft. The present application does not limit the type of terminal devices. It can be understood that, as Figure 1 shown, each terminal device in the terminal cluster can install a business application. When the business application runs on each terminal device, it can be respectively connected to the above-mentioned Figure 1Data interaction is carried out between the servers 10d shown. Among them, the business applications running in each terminal device can correspond to an independent client or an embedded sub-client integrated in a certain client. This application does not make any limitations in this regard.

[0076] Among them, the business applications can specifically include, but are not limited to: applications with item recommendation functions such as browsers, in-vehicle applications, smart home applications, shopping applications, content interaction applications, etc. Among them, if the terminal devices included in the terminal cluster are in-vehicle devices, then the in-vehicle devices can be intelligent terminals in the intelligent transportation scenario, and the business applications running in the in-vehicle devices can be called in-vehicle applications.

[0077] Among them, the server 10d can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. This application does not make any limitations on the type of the server.

[0078] It can be understood that Figure 1One or more service applications can be installed in each of the terminal devices shown. A sequence recommendation model can be integrated into each service application. The sequence recommendation model here can include, but is not limited to: SASRec (Self-Attentive Sequential Recommendation), BERT4Rec (Sequential Recommendation with Bidirectional Encoder Representations from Transformer), IDA-SR (Towards Universal Sequence Representation Learning for Recommender Systems), etc. The network structure of the sequence recommendation model is not limited in this application. For the sequence recommendation model involved in this application, the input of the sequence recommendation model can refer to the item interaction sequence feature, and the item interaction sequence feature can be formed by splicing item embedding representations corresponding to two or more items. Among them, the item interaction sequence can refer to the item sequence generated by an object on a certain platform. For example, the items purchased by object A on a certain platform in chronological order are: item 1, item 2, item 3, item 4. Then the item interaction sequence of object A can be expressed as [item 1, item 2, item 3, item 4]. The item embedding representation refers to mapping an item into a continuous, low-dimensional vector space so that a computer device can better understand and process these items.

[0079] In the embodiments of this application, a pre-trained language model can be pre-trained in the training sets of multiple fields. Under the condition that resources permit, the pre-trained language model can be further adjusted on the training set of a specific field. Furthermore, the pre-trained language model that has been trained can be used to initialize the item embedding table of the sequence recommendation model to enhance the effectiveness of the item embedding representation, and thus improve the recommendation effect of the sequence recommendation model. Among them, the item embedding table can include item embedding representations corresponding to multiple items, and the item ID (Identity document) of each item can be used as the index information of the item in the item embedding table. For example, the item ID can be used to query the item embedding representations corresponding to the items included in the item interaction sequence of an object from the item embedding table, and form the item interaction sequence feature input into the sequence recommendation model.

[0080] The pre-trained language model involved in the embodiments of this application may refer to a language model trained on large-scale text data, which can be used to learn rich language knowledge and grammar rules. The goal of the pre-trained language model is to learn language representations on unlabeled text data, so that it has good generalization ability and can be applied to various natural language processing tasks. The pre-trained language model usually includes two steps: ① The pre-training stage (Pretraining), in which the model can learn knowledge such as the grammatical structure, semantic information, and context association of the text. ② The fine-tuning stage (Fine-tuning), which is fine-tuned on specific downstream tasks (such as text type, language understanding, etc.) to meet the requirements of specific tasks and improve the performance of the model on this task. Among them, the pre-trained language model may include but is not limited to: BERT (Bidirectional Encoder Representations from Transformers, bidirectional encoding representation based on Transformer), GPT (Generative Pretrained Transformer, generative pre-trained Transformer), RoBERTa (Robustly optimized BERT approach, optimized BERT pre-training method), Longformer (The Long-Document Transformer, Transformer based on long text), etc. This application does not limit the network structure of the pre-trained language model. The pre-trained language model in the embodiments of this application belongs to the pre-trained model, and can also be called a language model, or can be called a large language model. This application does not make a limitation on this.

[0081] Please refer to Figure 2 , Figure 2 is a schematic flow chart of a data processing method provided by the embodiments of this application; it can be understood that this data processing method can be executed by a computer device, and this computer device can be a server (such as Figure 1 the server 10d shown), or it can be a terminal device (such as Figure 1 any terminal device in the terminal cluster shown), and this application does not make a limitation on this. As Figure 2 shown, this data processing method may include the following steps S101 to step S106:

[0082] Step S101, obtain the first text representation corresponding to each item in the item set, and according to the first text representation, obtain the second text representation corresponding to the historical item sequence of each object in the object set.

[0083] In the embodiments of the present application, a business application with an integrated sequence recommendation function can run on a computer device, items in the business application are obtained, and the items in the business application are added to an item set. In addition, registered objects in the business application can be obtained, and the registered objects in the business application are added to an object set. Furthermore, according to the interaction records of each object in the object set in the business platform, an interaction item sequence corresponding to each object in the object set can be obtained. Among them, the above-mentioned interaction records may refer to item purchase records, item sharing records, item collection records, etc. of each object in the business platform, and the present application does not limit this; the items included in the interaction item sequence can be sorted according to the item interaction time (for example, purchase time, sharing time, collection time, etc.), and the items in the interaction item sequence are all historical interaction items of the object. For ease of understanding, the interaction item sequence of each object in the embodiments of the present application can be referred to as a historical item sequence; each object in the object set can correspond to a historical item sequence.

[0084] After obtaining the item set, the object set, and the historical item sequence corresponding to each object in the object set in the business platform, text representation construction can be performed on all items in the item set and the historical item sequence corresponding to each object in the object set. Among them, the process of constructing the text representation of an item may include, but is not limited to: obtaining the item type (Category), item identifier (Brand), and item name (Title) corresponding to each item in the item set, combining the item type, item identifier, and item name corresponding to the same item, and obtaining the text representation of each item. At this time, the text representation can be referred to as the first text representation. Among them, the item type refers to a classification method based on the attributes (such as color, size, shape, etc.), functions, uses, or other characteristics of the item. For example, items can be classified according to characteristics such as material and use to obtain the item type corresponding to each item; the item type may include, but is not limited to: clothing, electronic products, beauty products, medicine, personal care and cleaning, office supplies, beverages, etc. The item identifier may refer to the brand to which the item belongs, and can be used to distinguish the specific image in the market of trademarks, logos, names, designs, etc. of one product from other similar products. The item name may refer to the name (or appellation) used to uniquely identify or distinguish an item, and the item name is usually composed of at least one combination of factors such as the manufacturer, commodity type, brand, model, and specification.

[0085] For the historical item sequence corresponding to each object in the object set, the items included in the historical item sequence can be sorted in descending order according to the interaction time of the items included in the historical item sequence, so as to obtain the sorted historical item sequence; the first text representations corresponding to the items included in the sorted historical item sequence are concatenated to obtain the second text representation corresponding to the historical item sequence of each object. In other words, the first text representations of the items included in the historical item sequence of each object can be concatenated in reverse order to form the second text representation corresponding to the historical item sequence of each object, and one second text representation can be constructed from one historical item sequence.

[0086] Please refer to Figure 3 , Figure 3 which is a schematic diagram of text representation construction provided by an embodiment of this application. As Figure 3 shown, assume that the item interaction record of object A in the service platform is: the interaction time of item 1 is t1; the interaction time of item 2 is t2, and the interaction time of item 3 is t3. If t1 is earlier than t2 and t2 is earlier than t3, then the historical item sequence of object A can be expressed as [item 1, item 2, item 3]. Among them, the first text representation 20a corresponding to item 1 can be expressed as "(type: electronic product) XX mobile phone with 16GB memory and 8-core processor (brand: XX)"; the first text representation 20b corresponding to item 2 can be expressed as "(type: electronic product) XX laptop with i7 (brand: XX)"; the first text representation 20c corresponding to item 3 can be expressed as "(type: electronic product) mouse (brand: XX)". Then the second text representation corresponding to the historical item sequence of this object A can refer to concatenating the first text representation 20c corresponding to item 3, the first text representation 20b corresponding to item 2, and the first text representation 20a corresponding to item 1 in sequence.

[0087] Step S102, generate a first training sample according to the second text representation, and perform text prediction on the first training sample according to the pre-trained language model to obtain the prediction probability corresponding to the hidden position in the first training sample.

[0088] Specifically, the second text representation corresponding to the historical item sequence of each object can be used to generate a first training sample for training the pre-trained language model. Here, the first training sample can refer to the input text representation for inputting into the pre-trained language model; in the pre-training stage of the pre-trained language model, based on the first training sample, a dual-task strategy of masked language modeling task and item-item comparison task (which can be abbreviated as item comparison) can be adopted to train the pre-trained language model.

[0089] The construction process of the first training sample may include but is not limited to: determining the hidden positions in the second text representation according to the text selection ratio, and determining the text at the hidden positions as the candidate text representation; performing a hiding process on the candidate text representation in the second text representation to obtain an initial sample, and adding a flag text (e.g., [CLS] token, which can be added at the first position of the initial sample) to the initial sample to obtain the first training sample. Among them, the hiding process of the candidate text representation may refer to performing a hiding process on the candidate text representation in the second text representation using a first hiding probability and a second hiding probability to obtain an initial sample; the first hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a mask (e.g., [MASK] token), and the second hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a random text representation. For example, the text selection ratio in the embodiments of this application can be set to 15%, the first hiding probability can be set to 80%, the second hiding probability can be set to 10%, etc. The embodiments of this application can be set according to the specific requirements of the actual application scenario, and the specific settings of the above text selection ratio, first hiding probability, and second selection probability are not limited.

[0090] Among them, the above text selection ratio may refer to the hidden text selection ratio preset for the masked language modeling task of the pre-trained language model. For example, if the length of the text corresponding to the input text (such as the second text representation) of the pre-trained language model is C (C is an integer greater than 1), and the preset text selection ratio is p (p is any value greater than 0 and less than 1), then it can be determined that the total length of the text that needs to be hidden in the masked language modeling task is C×p. The hidden position may refer to the position where the text representation selected for the hiding process in the second text representation is located; the candidate text representation may refer to all the text representations selected for the hiding process in the second text representation; the initial sample may refer to the second text representation after the hiding process of the candidate text representation is completed; the first training sample refers to the second text representation with the [CLS] token added at the first position and after the hiding process of the candidate text representation is completed.

[0091] In a feasible embodiment, the text length can be represented by the number of tokens (lemmas or word instances) included in the text representation (e.g., the second text representation). In natural language processing, a token can be understood as the smallest unit with independent meaning in the text, usually referring to parts with independent semantic meanings such as words, numbers, phrases, etc. For example, when the text length to be hidden in the masked language modeling task is C×p, it means that C×p tokens in the second text processing need to be hidden; thus, the positions of the tokens to be hidden can be determined in the second text representation, that is, the hidden positions in the second text representation are determined; the hidden positions at this time can refer to the positions where individual tokens are located, that is, each token in the second text representation can correspond to a hidden position. The above C×p tokens to be hidden can be randomly selected from the second text representation, and the selected C×p tokens can be called the candidate text representation.

[0092] For example, it is illustrated with a text selection ratio of 15%, a first hiding probability of 80%, and a second hiding probability of 10%. The masked language modeling task in the pre-trained language model can randomly select 15% of the tokens in the second text representation and hide these selected tokens in the following ways: ① Replace the selected tokens with masks ([MASK] tokens) with the first hiding probability (80%); ② Replace the selected tokens with random tokens with the second hiding rate (10%); ③ Do not change the selected tokens with the third hiding probability (10%). The pre-trained language model is used to predict the original values of the selected tokens to obtain the prediction probability corresponding to each selected token in the masked language modeling task.

[0093] In a feasible embodiment, the text length can be represented by the number of items included in the text representation (e.g., the second text representation), or it can be understood as the number of the first text representations included in the second text representation; it should be understood that the first text representation corresponding to each item can include one or more tokens. For example, when the text length to be hidden in the masked language modeling task is C×p, it means that C×p first text representations in the second text processing need to be hidden; thus, the positions of the first text representations to be hidden can be determined in the second text representation, that is, the hidden positions in the second text representation are determined; the hidden positions at this time can refer to the positions where the entire first text representation is located, that is, the entire first text representation in the second text representation corresponds to a hidden position. The above C×p first text representations to be hidden can be randomly selected from the second text representation, and the selected C×p first text representations can be called the candidate text representation.

[0094] For example, an illustration is given with a text selection ratio of 15%, a first hiding probability of 80%, and a second hiding probability of 10%. In the masked language modeling task of the pre-trained language model, 15% of the first text representations in the second text representation can be randomly selected, and these selected first text representations are hidden as follows: ① Replace the selected first text representations with a mask ([MASK] token) at the first hiding probability (80%); ② Replace the selected first text representations with random first text representations at the second hiding rate (10%); ③ Do not change the selected first text representations at the third hiding probability (10%). The pre-trained language model is used to predict the original values of the selected first text representations to obtain the prediction probabilities corresponding to each selected first text representation in the masked language modeling task.

[0095] In summary, in the embodiments of the present application, in the masked language modeling task of the pre-trained language model, individual tokens in the second text representation can be hidden, or the entire first text representation in the second text representation can be hidden. The present application does not make any limitations in this regard.

[0096] It should be noted that before the text representation corresponding to an item or a sequence of historical items is input into the pre-trained language model, a special flag text (e.g., [CLS] token) can be added before this text representation. The output of this flag text at the last layer of the pre-trained language model can be used as the embedded representation of the item or the sequence of historical items for subsequent model training and item embedding initialization. Adding the flag text is beneficial for the model to learn the bidirectional semantic knowledge in the sequence of historical items. For example, assume that the second text representation corresponding to a certain sequence of historical items is X1 = {T1, T2... T n}, then before the second text representation corresponding to this sequence of historical items is input into the pre-trained language model, the flag text needs to be added, that is, X2 = {[CLS], T1, T2... T n}; where X1 represents the second text representation corresponding to a certain sequence of historical items, and T1, T2... T n represent the first, second,..., nth first text representations in the second text representation. The text representation after adding the flag text ([CLS] token) to the second text representation X1 is denoted as X2. Among them, in X1 = {T1, T2... T n} and X2 = {[CLS], T1, T2... T n}, the interaction time of the item corresponding to T1 is later than the interaction time of the item corresponding to T2, the interaction time of the item corresponding to T2 is later than the interaction time of the item corresponding to T3, and so on, T nThe corresponding item has the earliest interaction time in the entire sequence; in other words, X1 and X2 are sorted in reverse order of interaction time from late to early.

[0097] In a feasible embodiment, a pre-trained language model can be used to perform text prediction on the first training sample to obtain the prediction probability corresponding to the hidden position in the first training sample. Optionally, if the pre-trained language model is any language model with an encoder architecture, then the first training sample can be input into the encoder in the pre-trained language model, and the first training sample can be encoded by the encoder to obtain the first sample embedding representation corresponding to the first training sample; this first sample embedding representation can be used to predict the prediction probabilities of each hidden position in the first training sample. Here, the prediction probability can refer to the probability of predicting the selected candidate text representation (token or the first text representation) according to the text representation other than the hidden position in the first training sample in the masked language modeling task of the pre-trained language model. Among them, the encoder can be used to learn the semantic association between the first text representations other than the hidden position in the first training sample to obtain the first sample embedding representation corresponding to the first training sample. This encoder can be used to perform bidirectional encoding on the first training sample, which can improve the effectiveness of the first sample embedding representation.

[0098] Step S103: According to the pre-trained language model, compare the items of the first training sample and the first positive sample corresponding to the first training sample to obtain the first sample similarity.

[0099] Specifically, in the pre-training stage of the pre-trained language model, for the item-item comparison task, the first text representation corresponding to the next real interaction item of the same object in sequence recommendation can be used as the positive sample, and the first text representations corresponding to other items in the same batch can be used as the negative samples; among them, the first training sample can include positive samples and negative samples. The first text representation of the subsequent interaction item corresponding to the first training sample can be determined as the first positive sample; the first training sample and the first positive sample belong to the historical interaction items of the same object, and the subsequent interaction item can refer to the next real interaction item of the first training sample in sequence recommendation; input the first positive sample into the pre-trained language model, and encode the first positive sample through the encoder in the pre-trained language model to obtain the second sample embedding representation corresponding to the first positive sample; obtain the first object embedding representation corresponding to the object associated with the first positive sample, and obtain the first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample.

[0100] Among them, it is assumed that the number of the first training samples is M, and M is a positive integer. Then, according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample, obtaining the first sample similarity may include, but is not limited to: obtaining the first feature similarity between the first object embedding representation and the second sample embedding representation, performing an exponential operation on the first feature similarity to obtain the first similarity candidate value; obtaining the second feature similarity between the first object embedding representation and the first sample embedding representations corresponding to each of the first training samples, performing an exponential operation on the second feature similarity to obtain the second candidate similarity values associated with each of the first training samples; accumulating the second candidate similarity values associated with each of the first training samples to obtain an accumulated similarity value, and determining the first sample similarity according to the ratio between the first similarity candidate value and the accumulated similarity value; the first sample similarity may be used as the loss of the item-item comparison task in the pre-trained language model. Among them, the calculation method of the first sample similarity may be as shown in the following formula (1):

[0101]

[0102] Among them, L1 represents the first sample similarity between the i-th first training sample and the first positive sample, where i is a positive integer less than or equal to M, log represents the logarithmic function with base 10, and h s represents the first object embedding representation calculated by the pre-trained language model. Here, the first object embedding representation may refer to the embedding representation corresponding to the object associated with the first positive sample; represents the second sample embedding representation of the first positive sample (the first positive sample corresponding to the i-th first training sample) calculated by the pre-trained language model; h i represents the first sample embedding representation of the i-th first training sample calculated by the pre-trained language model. B may represent the batch size, and τ is a hyperparameter; represents the cosine similarity (the first feature similarity) between the first object embedding representation and the second sample embedding representation; sim(h s , h i ) represents the cosine similarity (the second feature similarity) between the first object embedding representation and the first sample embedding representation.

[0103] Step S104, training the pre-trained language model according to the prediction probability, the hidden positions in the first training samples, and the first sample similarity to obtain the trained pre-trained language model.

[0104] Specifically, assume that there are N hidden positions in the first training sample, where N is a positive integer, that is, the number of text representations selected in the second training sample is N. Take the logarithm of the prediction probabilities corresponding to the N hidden positions to obtain the logarithmic values of the probabilities of each hidden position, and accumulate the logarithmic values of the probabilities of each hidden position to obtain the masked language modeling loss. The calculation method of this masked language modeling loss can be shown as the following formula (2):

[0105]

[0106] Among them, L2 in formula (2) represents the masked language modeling loss, X represents the first training sample input into the pre-trained language model, and m(X) represents the set of all text representations that have been hidden in the first training sample, or can be understood as the set of text representations selected in the second text representation; represents any text representation that has been hidden in the first training sample; X \m(x) represents the text representation that has not been hidden in the first training sample, that is, the remaining text representation in the first training sample except m(X); represents the probability that the pre-trained language model predicts the hidden text representation (the selected token or the selected first text representation, etc.) in the first training sample, that is, the prediction probability corresponding to the hidden position.

[0107] Furthermore, the model training loss corresponding to the pre-trained language model can be determined according to the masked language modeling loss and the first sample similarity, and the calculation method of this model training loss can be shown as the following formula (3):

[0108] L PT = L1 + γL2 (3)

[0109] Among them, L in formula (3) PT can represent the model loss of the pre-trained language model in the pre-training stage. γ, as a hyperparameter, can be used to control the weight of the masked language modeling task.

[0110] In the pre-training stage of the pre-trained language model, the network parameters of the pre-trained language model can be iteratively trained according to the model training loss until the model training loss meets the training end condition, and then the training is stopped to obtain the trained pre-trained language model. Among them, the training end condition here can be that the number of training times of the pre-trained language model reaches the maximum number of iterations, or it can be that the model training loss reaches the convergence condition. This application does not limit the setting of the training end condition. When the pre-trained language model reaches the training end condition, the pre-training of the pre-trained language model can be stopped, and the network parameters at the time of training stop are saved, and the network parameters saved here are used as the trained pre-trained language model.

[0111] In step S105, a second training sample is generated according to the second text representation. Based on the pre-trained pre-trained language model, the second training sample and the second positive sample corresponding to the second training sample are compared for items to obtain a second sample similarity.

[0112] In a feasible implementation manner, under the condition that resources permit, it is possible to select to further adjust the pre-trained pre-trained language model in the target domain to further improve the performance of the pre-trained language model. In the fine-tuning stage of the pre-trained language model, the item-item comparison task can still be used to adjust the trained pre-trained language model. The training samples in the fine-tuning stage of the pre-trained language model can be called second training samples. The second training samples can also be generated based on the second text representation corresponding to the historical item sequence. The second training samples can be without hidden processing, and the second training samples are all negative samples, that is, all the second training samples are negative samples.

[0113] Among them, the first training sample and the second training sample can be selected according to attributes. The first training sample can refer to the text representation with generalization attributes, and the second training sample can refer to the text representation with personalized attributes. In this way, both the text representation with generalization attributes and the text representation with personalized attributes can be obtained, enhancing the embedding representation of the pre-trained language model in the personalized field while ensuring the generalization performance of the pre-trained language model.

[0114] The construction process of the second training sample can include but is not limited to: adding a flag text (for example, [CLS] token) to the second text representation to obtain the second training sample, and determining the second positive sample corresponding to the second training sample; obtaining the third sample embedding representation corresponding to the second training sample through the pre-trained language model, and obtaining the fourth sample embedding representation corresponding to the second positive sample through the pre-trained language model; obtaining the second object embedding representation corresponding to the object associated with the second positive sample, and obtaining the second sample similarity according to the second object embedding representation, the third sample embedding representation, and the fourth sample embedding representation. Among them, the second training samples used in the fine-tuning stage of the pre-trained language model can be all negative samples. After determining a second positive sample in this stage, the second training samples for adjusting the pre-trained language model can be determined in the item set. For example, the second training sample can be the first text representation corresponding to other items in the item set except the second positive sample. The calculation method of the second sample similarity can refer to the calculation method of the first sample similarity described above and will not be elaborated here; the calculation method of the second sample similarity can be as shown in the following formula (4):

[0115]

[0116] Among them, L FTRepresents the second sample similarity between the second training sample and the second positive sample, that is, the item comparison task loss of the pre-trained language model during the fine-tuning stage. log represents the logarithmic function with base 10, and h in formula (4). s Represents the second object embedding representation calculated by the pre-trained language model after training. Here, the second object embedding representation can refer to the embedding representation corresponding to the object associated with the second positive sample; Represents the fourth sample embedding representation of the second positive sample calculated by the pre-trained language model after training; I i Represents the third sample embedding representation of the second training sample calculated by the pre-trained language model after training. I can represent the batch size, and τ is a hyperparameter; Represents the cosine similarity between the second object embedding representation and the fourth sample embedding representation; sim(h s ,I i ) represents the cosine similarity between the second object embedding representation and the third sample embedding representation.

[0117] Step S106, adjust the pre-trained language model after training according to the second sample similarity to obtain an adjusted pre-trained language model.

[0118] Specifically, in the fine-tuning stage of the pre-trained language model, the network parameters of the pre-trained language model after training can be iteratively trained according to the second sample similarity until the second sample similarity meets the training end condition, at which point the training stops to obtain an adjusted pre-trained language model. Among them, the training end condition here can be that the number of training times of the pre-trained language model after training reaches the maximum number of iterations, or it can be that the second sample similarity reaches the convergence condition. This application does not limit the setting of the training end condition. When the pre-trained language model after training reaches the training end condition, the fine-tuning of the pre-trained language model after training can be stopped, and the network parameters at the time of training stop are saved. The network parameters saved here are used as the adjusted pre-trained language model, and this adjusted pre-trained language model can be used to initialize the embedding representation of the item.

[0119] Please refer to Figure 4 , Figure 4 is a training schematic diagram of a pre-trained language model provided by an embodiment of this application. As Figure 4As shown, the training process of the pre-trained language model may include two stages, denoted as the pre-training stage and the fine-tuning stage respectively. In the pre-training stage of the pre-trained language model, strategies of masked language modeling tasks and item-item comparison tasks can be adopted to pre-train the pre-trained language model, and the trained pre-trained language model can be obtained. In the fine-tuning stage of the pre-trained language model, the strategy of item-item comparison tasks can be adopted to adjust the trained pre-trained language model, and the adjusted pre-trained language model can be obtained.

[0120] In the embodiments of the present application, the first text representations corresponding to each item in the item set are obtained, and the first text representations corresponding to the items included in the historical item sequence of each object are concatenated to obtain the second text representation corresponding to the historical item sequence; the first training sample for training the pre-trained language model and the second training sample for adjusting the trained pre-trained language model are constructed according to the second text representation. Based on the first training sample, the pre-trained language model is trained by using the dual-task strategy of masked language modeling (text prediction) and item comparison to obtain the trained pre-trained language model. Based on the second training sample, the trained pre-trained language model is adjusted by using the item comparison strategy to obtain the adjusted pre-trained language model. The adjusted pre-trained language model can provide an initial item embedding representation for sequence recommendation, and can make full use of the prior knowledge of the pre-trained language model to extract the sequence features in the historical item sequence, thereby enhancing the effectiveness of the item embedding representation.

[0121] In the embodiments of the present application, the adjusted pre-trained language model can be used to initialize the item embedding of a general sequence recommendation model. Please refer to Figure 5 , Figure 5 which is a schematic flowchart of item embedding initialization based on a pre-trained language model provided by the embodiments of the present application; it can be understood that Figure 5 the corresponding embodiment is the application of the adjusted pre-trained language model. As Figure 5 shown, the data processing method may include the following steps S201 to S205:

[0122] Step S201, adding a flag text to the first text representation corresponding to each item in the item set to obtain the item input text corresponding to each item in the item set.

[0123] In the embodiments of the present application, a flag text (for example, [CLS] token) can be added to the first text representation corresponding to each item in the item set, and the item input text corresponding to each item in the item set can be obtained.

[0124] Step S202: Input the item input text corresponding to item a in the item set into the adjusted pre-trained language model, and obtain the item embedding representation corresponding to item a through the adjusted pre-trained language model.

[0125] Specifically, for each item in the item set, an item input text can be obtained. This item input text can be input into the adjusted pre-trained language model, and the output corresponding to the flag text (e.g., [CLS] token) output by the adjusted pre-trained language model can be used as the item embedding representation corresponding to this item. In the same way, the item embedding representations corresponding to each item in the item set can be obtained.

[0126] Step S203: Obtain the item embedding representations corresponding to the various items in the item set, and add the item embedding representations corresponding to the various items to the item embedding table.

[0127] Specifically, every time the item embedding representation corresponding to an item in the item set is calculated, the item embedding representation of this item can be added to the item embedding table to initialize the item embedding table; this item embedding table can be used as the initial sequence features in sequence recommendation modeling.

[0128] Step S204: Generate the third training sample of the sequence recommendation model according to the item embedding representations in the item embedding table and the historical item sequences corresponding to the various objects in the object set.

[0129] Step S205: Train the sequence recommendation model according to the third training sample to obtain the trained sequence recommendation model.

[0130] Among them, the training of the sequence recommendation model includes a first stage and a second stage. The first stage is used to train the sequence modeling task and suspend the training of the item embedding representations; the second stage is used to train the item embedding representations and suspend the training of the sequence modeling task; the trained sequence recommendation model is used for item recommendation.

[0131] After completing the initialization process of the item embedding table, a two-stage training strategy can be adopted to train the sequence recommendation model. In the first stage, the item embedding representations in the sequence recommendation model can be frozen, and only the sequence modeling part in the model is trained. In the second stage, the sequence modeling part in the model can be frozen, and only the item embedding representations in the model are trained. During the training process of the sequence recommendation model, other training settings are the same as the original general settings of this sequence recommendation model.

[0132] It can be understood that using the adjusted pre-trained language model to initialize the item embedding table in the sequential recommendation model and training the sequential recommendation model with a two-stage training strategy based on the initialized item embedding table is only an example provided by this application. This application can also adopt other strategies to train the sequential recommendation model and conduct two-stage model training on the sequential recommendation model. For example, after completing the initialization of the item embedding table, the full parameters of the model can be directly trained on the dataset in the target domain. Or, the item representations encoded by the adjusted pre-trained language model can be used as additional features for each item and frozen, and added to the original item embedding representations during the training and inference processes, and the full parameters in the sequential modeling model can be trained on the dataset in the target domain, etc. This application does not limit the training strategy for the sequential recommendation model.

[0133] Please refer to Figure 6 , Figure 6 which is a training schematic diagram of a sequential recommendation task provided by an embodiment of this application. As Figure 6 shown, the first text representation can be constructed for each item in the item set, and the second text representation corresponding to the historical item sequence of each object in the object set can be constructed based on the first text representation of each item. The first training sample and the second training sample are constructed based on the second text representation, and the pre-trained language model is pre-trained using the first training sample to obtain the trained pre-trained language model; the trained pre-trained language model is adjusted based on the second training sample to obtain the adjusted pre-trained language model. Through the adjusted pre-trained language model, the item embedding table can be initialized. Based on the initialized item embedding table, the initialized sequence features of the sequential recommendation model can be constructed. Based on the initialized sequence features, the sequential recommendation model is trained in two stages to obtain the trained sequential recommendation model, and the trained sequential recommendation model can be iteratively trained according to the object representation output by the model; the trained sequential recommendation model can be used for item recommendation.

[0134] Please refer to Table 1 and Table 2 below, which show the test results obtained by a sequential recommendation model that initializes an item embedding table using an adjusted pre-trained language model, and some other sequential recommendation models when tested on different datasets (i.e., making item recommendations using text representations in different datasets). Among them, other sequential recommendation models involved in the embodiments of this application may include, but are not limited to: SASRec, UniSRec (Towards Universal Sequence Representation Learning for Recommender Systems, a general sequence representation learning model enhanced by a pre-trained language model), RECFORMER (a general sequence representation learning model enhanced by a pre-trained language model), etc.

[0135] Table 1

[0136]

[0137]

[0138] Table 2

[0139]

[0140] Among them, this solution refers to the sequential recommendation model provided in this application that initializes an item embedding table using an adjusted pre-trained language model. HR (Hit Radio, hit rate) is used to evaluate the accuracy of the model's predicted recommended items. The higher the HR@1 and HR@5 metrics, the higher the accuracy of the model's recommended items. NDCG (Normalized Discounted Cumulative Gain) is used to evaluate the rationality of the recommendation order when the model makes item recommendations. The higher the NDCG@5 metric, the more preferentially the model can recommend items that match the object's own interests to the object.

[0141] As can be seen from the results in Table 1 and Table 2 above, the performance of this solution is almost better than that of UniSRec and RECFORMER in each dataset. At the same time, after initializing the item embedding table using the pre-trained language model proposed in this application, the performance of SASRec and BERT4Rec in the sequential recommendation task has been greatly improved. The above experimental results fully illustrate the effectiveness of this solution.

[0142] In the embodiments of the present application, the pre-trained language model after adjustment is used to initialize the item embedding representation, so that the text information and sequence features can be better aligned. While making full use of the prior knowledge of the pre-trained language model, the historical item interaction sequence features are accurately extracted, thereby enhancing the effectiveness of the item embedding representation, and further improving the recommendation effect of the sequence recommendation model.

[0143] It can be understood that in the specific implementation of the present application, it may involve interaction information such as the item purchase records of users in the shopping platform. When the above embodiments of the present application are applied to specific products or technologies, the permission or consent of relevant institutions or departments, or the users themselves needs to be obtained, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards in the relevant regions.

[0144] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a data processing device provided by the embodiments of the present application. As Figure 7 shown, the data processing device 1 includes: a text acquisition module 101, a text prediction module 102, a first item comparison module 103, a model pre-training module 104, a second item comparison module 105, and a model adjustment module 106;

[0145] The text acquisition module 101 is configured to acquire the first text representation corresponding to each item in the item set, and according to the first text representation, acquire the second text representation corresponding to the historical item sequence of each object in the object set;

[0146] The text prediction module 102 is configured to generate a first training sample according to the second text representation, perform text prediction on the first training sample according to the pre-trained language model, and obtain the prediction probability corresponding to the hidden position in the first training sample; the first training sample includes positive samples and negative samples;

[0147] The first item comparison module 103 is configured to perform item comparison on the first training sample and the first positive sample corresponding to the first training sample according to the pre-trained language model, and obtain the first sample similarity;

[0148] The model pre-training module 104 is configured to train the pre-trained language model according to the prediction probability, the hidden position in the first training sample, and the first sample similarity, and obtain the trained pre-trained language model;

[0149] The second item comparison module 105 is configured to generate a second training sample according to the second text representation, perform item comparison on the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model, and obtain the second sample similarity; the second training sample does not include positive samples;

[0150] The model adjustment module 106 is configured to adjust the pre-trained language model after training according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of the item.

[0151] In one or more embodiments, the text acquisition module 101 acquires the first text representation corresponding to each item in the item set, and according to the first text representation, acquires the second text representation corresponding to the historical item sequence of each object in the object set, for performing the following steps:

[0152] Acquire the item type, item identifier, and item name corresponding to each item in the item set, and combine the item type, item identifier, and item name corresponding to the same item to obtain the first text representation corresponding to each item;

[0153] Acquire the historical item sequence corresponding to each object in the object set, and sort the items included in the historical item sequence in descending order according to the interaction time of the items included in the historical item sequence to obtain a sorted historical item sequence;

[0154] Concatenate the first text representations corresponding to the items included in the sorted historical item sequence to obtain the second text representation corresponding to the historical item sequence of each object.

[0155] In one or more embodiments, the text prediction module 102 generates a first training sample according to the second text representation, for performing the following steps:

[0156] Determine the hidden position in the second text representation according to the text selection ratio, and determine the text at the hidden position as the candidate text representation;

[0157] Perform a hiding process on the candidate text representation in the second text representation to obtain an initial sample, and add a flag text to the initial sample to obtain a first training sample.

[0158] In one or more embodiments, the text prediction module 102 performs a hiding process on the candidate text representation in the second text representation to obtain an initial sample, for performing the following steps:

[0159] Perform a hiding process on the candidate text representation in the second text representation by using a first hiding probability and a second hiding probability to obtain an initial sample;

[0160] Wherein, the first hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a mask, and the second hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a random text representation.

[0161] In one or more embodiments, the pre-trained language model includes an encoder and a decoder;

[0162] The text prediction module 102 performs text prediction on the first training sample according to the pre-trained language model to obtain the prediction probability corresponding to the hidden position in the first training sample, and is used to perform the following steps:

[0163] Input the first training sample into the pre-trained language model, and perform encoding processing on the first training sample through the pre-trained language model to obtain the first sample embedding representation corresponding to the first training sample;

[0164] Perform text prediction on the hidden position in the first training sample according to the first sample embedding representation to obtain the prediction probability corresponding to the hidden position in the first training sample.

[0165] In one or more embodiments, the first item comparison module 103 performs item comparison on the first training sample and the first positive sample corresponding to the first training sample according to the pre-trained language model to obtain the first sample similarity, and is used to perform the following steps:

[0166] Determine the first text representation of the subsequent interaction item corresponding to the first training sample as the first positive sample; the first training sample and the first positive sample belong to the historical interaction items of the same object;

[0167] Input the first positive sample into the pre-trained language model, and perform encoding processing on the first positive sample through the encoder in the pre-trained language model to obtain the second sample embedding representation corresponding to the first positive sample;

[0168] Obtain the first object embedding representation corresponding to the object associated with the first positive sample, and obtain the first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample.

[0169] In one or more embodiments, the number of the first training samples is M, and M is a positive integer;

[0170] The first item comparison module 103 obtains the first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample, and is used to perform the following steps:

[0171] Obtain the first feature similarity between the first object embedding representation and the second sample embedding representation, perform an exponential operation on the first feature similarity to obtain the first similarity candidate value;

[0172] Obtain the second feature similarity between the first object embedding representation and the first sample embedding representations corresponding to each first training sample, perform an exponential operation on the second feature similarity to obtain the second candidate similarity values associated with each first training sample;

[0173] Accumulate the second candidate similarity values associated with each first training sample to obtain an accumulated similarity value, and determine the first sample similarity according to the ratio between the first similarity candidate value and the accumulated similarity value.

[0174] In one or more embodiments, the number of hidden positions in the first training sample is N, where N is a positive integer;

[0175] The model pre-training module 104 trains the pre-trained language model according to the prediction probability, the hidden positions in the first training sample, and the first sample similarity, and obtains the trained pre-trained language model for performing the following steps:

[0176] Perform a logarithmic operation on the prediction probabilities corresponding to the N hidden positions to obtain the probability logarithm values of each hidden position, and accumulate the probability logarithm values of each hidden position to obtain the masked language modeling loss;

[0177] Determine the model training loss corresponding to the pre-trained language model according to the masked language modeling loss and the first sample similarity;

[0178] Iteratively train the network parameters of the pre-trained language model according to the model training loss, and stop training until the model training loss meets the training end condition, and obtain the trained pre-trained language model.

[0179] In one or more embodiments, the second item comparison module 105 generates a second training sample according to the second text representation, and performs item comparison on the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model to obtain the second sample similarity for performing the following steps:

[0180] Add flag text to the second text representation to obtain a second training sample, and determine the second positive sample corresponding to the second training sample;

[0181] Obtain the third sample embedding representation corresponding to the second training sample through the pre-trained language model, and obtain the fourth sample embedding representation corresponding to the second positive sample through the pre-trained language model;

[0182] Obtain the second object embedding representation corresponding to the object associated with the second positive sample, and obtain the second sample similarity according to the second object embedding representation, the third sample embedding representation, and the fourth sample embedding representation.

[0183] In one or more embodiments, the data processing device 1 may further include: an item input text acquisition module 107, an item embedding representation acquisition module 108, and an item embedding table generation module 109;

[0184] An item input text acquisition module 107, configured to add flag text to the first text representation corresponding to each item in the item set to obtain the item input text corresponding to each item in the item set;

[0185] An item embedding representation acquisition module 108, configured to input the item input text corresponding to item a in the item set into the adjusted pre-trained language model, and obtain the item embedding representation corresponding to item a through the adjusted pre-trained language model;

[0186] An item embedding table generation module 109, configured to obtain the item embedding representations corresponding to each item in the item set, and add the item embedding representations corresponding to each item to the item embedding table.

[0187] In one or more embodiments, the data processing device 1 may further include: a training sample generation module 110, a recommendation model training module 111;

[0188] The training sample generation module 110 is configured to generate a third training sample for the sequence recommendation model according to the item embedding representations in the item embedding table and the historical item sequences corresponding to each object in the object set;

[0189] The recommendation model training module 111 is configured to train the sequence recommendation model according to the third training sample to obtain a trained sequence recommendation model;

[0190] Among them, the training of the sequence recommendation model includes a first stage and a second stage. The first stage is used to train the sequence modeling task and suspend the training of the item embedding representations; the second stage is used to train the item embedding representations and suspend the training of the sequence modeling task; the trained sequence recommendation model is used for item recommendation.

[0191] According to an embodiment of the present application, the steps involved in the foregoing Figure 2 The data processing method shown can be performed by Figure 7 Each module in the data processing device 1 shown. For example, Figure 2 The step S101 shown can be performed by Figure 7 The text acquisition module 101 shown, Figure 2 The step S102 shown can be performed by Figure 7 The text prediction module 102 shown, Figure 2 The step S103 shown can be performed by Figure 7 The first item comparison module 103 shown, Figure 2 The step S104 shown can be performed by Figure 7 The model pre-training module 104 shown, Figure 2 The step S105 shown can be performed by Figure 7executed by the second item comparison module 105 shown Figure 2 The step S106 shown can be performed by Figure 7 the model adjustment module 106 shown, etc.

[0192] According to an embodiment of the present application Figure 7 Each module in the data processing device 1 shown can be separately or all combined into one or several modules to form, or a certain one (or some) of the modules can be further split into at least two smaller functional units, and the same operations can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In actual applications, the function of one module can also be realized by at least two units, or the functions of at least two modules are realized by one module. In other embodiments of the present application, the data processing device 1 can also include other modules or units. In actual applications, these functions can also be assisted by other modules and can be realized by the cooperation of at least two modules.

[0193] In the embodiments of the present application, the first text representation corresponding to each item in the item set is obtained, and the first text representations corresponding to the items included in the historical item sequence of each object are concatenated to obtain the second text representation corresponding to the historical item sequence; based on the second text representation, the first training sample for training the pre-trained language model and the second training sample for adjusting the trained pre-trained language model are constructed. Based on the first training sample, the pre-trained language model is trained by using the dual-task strategy of masked language modeling (text prediction) and item comparison to obtain the trained pre-trained language model. Based on the second training sample, the trained pre-trained language model is adjusted by using the item comparison strategy to obtain the adjusted pre-trained language model. The adjusted pre-trained language model can provide an initial item embedding representation for sequence recommendation, can make full use of the prior knowledge of the pre-trained language model to extract the sequence features in the historical item sequence, and thus enhance the effectiveness of the item embedding representation; by initializing the item embedding table of the sequence recommendation model with the adjusted pre-trained language model, the recommendation effect of the sequence recommendation model can be improved.

[0194] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a computer device provided by an embodiment of the present application. As Figure 8 shown, the computer device 1000 can be a terminal device. For example, the terminal device 10a in the corresponding embodiment above, or it can also be a server. For example, the above Figure 1 corresponding embodiment, and it can also be a server. For example, the above Figure 1The server 10d in the corresponding embodiment will not be limited here. For the sake of understanding, this application takes a computer device as an example of a terminal device. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may further include: a user interface 1003 and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 1005 may optionally be at least one storage device located far from the aforementioned processor 1001. As Figure 8 shown, the memory 1005, as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0195] Among them, the network interface 1004 in the computer device 1000 may further provide a network communication function, and the optional user interface 1003 may further include a display screen and a keyboard. In Figure 8 the computer device 1000 shown, the network interface 1004 can provide a network communication function; while the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application program stored in the memory 1005 to achieve:

[0196] Obtain the first text representation corresponding to each item in the item set, and according to the first text representation, obtain the second text representation corresponding to the historical item sequence of each object in the object set;

[0197] Generate a first training sample according to the second text representation, perform text prediction on the first training sample according to the pre-trained language model, and obtain the prediction probability corresponding to the hidden position in the first training sample; the first training sample includes positive samples and negative samples;

[0198] According to the pre-trained language model, compare the items in the first training sample and the first positive sample corresponding to the first training sample to obtain the first sample similarity;

[0199] Train the pre-trained language model according to the prediction probability, the hidden position in the first training sample, and the first sample similarity to obtain the trained pre-trained language model;

[0200] Generate a second training sample according to the second text representation. Based on the pre-trained language model after training, compare the second training sample with the second positive sample corresponding to the second training sample to obtain the second sample similarity; the second training sample does not include positive samples.

[0201] Adjust the pre-trained language model after training according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of the item.

[0202] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the description of the data processing method in any one of the foregoing Figure 2 、 Figure 5 embodiments, and can also execute the description of the data processing device 1 in the corresponding embodiments of the foregoing Figure 7 which will not be elaborated here. In addition, the description of the beneficial effects of adopting the same method will not be elaborated either.

[0203] In addition, it should be pointed out here that: the embodiments of the present application also provide a computer-readable storage medium, and the computer-readable storage medium stores the computer program executed by the foregoing data processing device 1, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the description of the data processing method in any one of the foregoing Figure 2 、 Figure 5 embodiments, so it will not be elaborated here. In addition, the description of the beneficial effects of adopting the same method will not be elaborated either. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc. For the technical details not disclosed in the embodiments of the computer-readable storage medium involved in the present application, please refer to the description of the method embodiments of the present application. As an example, the program instructions can be deployed to be executed on one computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. The multiple computer devices distributed at multiple locations and interconnected through a communication network can form a blockchain system.

[0204] In addition, it should be noted that: the embodiments of the present application also provide a computer program product or a computer program. The computer program product or the computer program may include computer instructions, and the computer instructions can be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor can execute the computer instructions, so that the computer device executes the foregoing Figure 2 、 Figure 5The description of the data processing method in any of the embodiments will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated either. For the technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0205] The terms "first", "second", etc. in the specification, claims and drawings of the embodiments of this application are used to distinguish different media contents, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally further includes steps or modules not listed, or optionally further includes other step units inherent to these processes, methods, devices, products or equipment.

[0206] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0207] The method and related device provided by the embodiments of this application are described with reference to the method flowcharts and / or structure diagrams provided by the embodiments of this application. Specifically, each process and / or block of the method flowchart and / or structure diagram, and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or structure diagrams Figure 1 one block or multiple blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions in Figure 1 one process or multiple processes and / or structure diagrams Figure 1The functions specified in one or more boxes. These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps in the process Figure 1 Steps in one or more processes and / or structural diagrams that illustrate the functions specified in one or more boxes.

[0208] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of that module or unit.

[0209] The above-disclosed are only the preferred embodiments of the present application. Of course, the scope of the rights of the present application cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A data processing method, characterized in that, Including: Obtain the first text representation corresponding to each item in the item set, and based on the first text representation, obtain the second text representation corresponding to the historical item sequence of each object in the object set; Generate a first training sample according to the second text representation, and perform text prediction on the hidden positions in the first training sample according to the pre-trained language model to obtain the prediction probability corresponding to the hidden positions; The first training sample includes positive samples and negative samples, and the number of hidden positions is N, where N is a positive integer; According to the pre-trained language model, perform item comparison between the first training sample and the first positive sample corresponding to the first training sample to obtain a first sample similarity; the first positive sample refers to the first text representation of the subsequent interaction items corresponding to the first training sample; Perform a logarithmic operation on the prediction probabilities corresponding to the N hidden positions to obtain the probability logarithm values of each hidden position, and accumulate the probability logarithm values of each hidden position to obtain a masked language modeling loss; Sum the product of the hyperparameter and the masked language modeling loss and the first sample similarity to obtain a model training loss, and train the pre-trained language model according to the model training loss to obtain a trained pre-trained language model; the hyperparameter is used to control the weight of the masked language modeling loss; Generate a second training sample according to the second text representation, and perform item comparison between the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model to obtain a second sample similarity; the second training sample does not include positive samples; Adjust the trained pre-trained language model according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of the item.

2. The method according to claim 1, characterized in that, The obtaining the first text representation corresponding to each item in the item set, and based on the first text representation, obtaining the second text representation corresponding to the historical item sequence of each object in the object set includes: Obtain the item type, item identifier, and item name corresponding to each item in the item set, and combine the item type, item identifier, and item name corresponding to the same item to obtain the first text representation corresponding to each item; Obtain the historical item sequence corresponding to each object in the object set, and sort the items included in the historical item sequence in descending order according to the interaction time of the items included in the historical item sequence to obtain a sorted historical item sequence; Concatenate the first text representations corresponding to the items included in the sorted historical item sequence to obtain the second text representation corresponding to the historical item sequence of each object.

3. The method according to claim 1, characterized in that, The generating a first training sample according to the second text representation includes: Determine the hidden positions in the second text representation according to the text selection ratio, and determine the text at the hidden positions as the candidate text representation; Perform a hiding process on the candidate text representation in the second text representation to obtain an initial sample, and add a flag text to the initial sample to obtain a first training sample.

4. The method according to claim 3, characterized in that The performing a hiding process on the candidate text representation in the second text representation to obtain an initial sample includes: Using a first hiding probability and a second hiding probability, perform a hiding process on the candidate text representation in the second text representation to obtain an initial sample; Wherein, the first hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a mask, and the second hiding probability refers to the probability of replacing the candidate text representation in the second text representation with a random text representation.

5. The method according to claim 1, characterized in that, The obtaining the prediction probability corresponding to the hidden position in the first training sample according to the pre-trained language model includes: Input the first training sample into the pre-trained language model, and perform an encoding process on the first training sample through the pre-trained language model to obtain a first sample embedding representation corresponding to the first training sample; According to the first sample embedding representation, perform text prediction on the hidden position in the first training sample to obtain the prediction probability corresponding to the hidden position in the first training sample.

6. The method according to claim 5, wherein The obtaining a first sample similarity by comparing items between the first training sample and the first positive sample corresponding to the first training sample according to the pre-trained language model includes: Determine the first text representation of the subsequent interaction item corresponding to the first training sample as the first positive sample; the first training sample and the first positive sample belong to the historical interaction items of the same object; Input the first positive sample into the pre-trained language model, and perform an encoding process on the first positive sample through the encoder in the pre-trained language model to obtain a second sample embedding representation corresponding to the first positive sample; Obtain a first object embedding representation corresponding to the object associated with the first positive sample, and determine a first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample.

7. The method according to claim 6, wherein The number of the first training samples is M, and M is a positive integer; The obtaining a first sample similarity according to the first object embedding representation, the second sample embedding representation, and the first sample embedding representation corresponding to the first training sample includes: Obtain a first feature similarity between the first object embedding representation and the second sample embedding representation, perform an exponential operation on the first feature similarity to obtain a first similar candidate value; Obtain a second feature similarity between the first object embedding representation and the first sample embedding representations corresponding to each first training sample, perform an exponential operation on the second feature similarity to obtain second candidate similar values associated with each first training sample; Accumulate the second candidate similar values associated with each first training sample to obtain an accumulated similarity value, and determine a first sample similarity according to the ratio between the first similar candidate value and the accumulated similarity value.

8. The method according to claim 1, wherein Generating a second training sample according to the second text representation, and performing item comparison on the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model to obtain a second sample similarity, including: Adding a flag text to the second text representation to obtain a second training sample, and determining a second positive sample corresponding to the second training sample; Obtaining a third sample embedding representation corresponding to the second training sample through the pre-trained language model, and obtaining a fourth sample embedding representation corresponding to the second positive sample through the pre-trained language model; Obtaining a second object embedding representation corresponding to the object associated with the second positive sample, and obtaining a second sample similarity according to the second object embedding representation, the third sample embedding representation, and the fourth sample embedding representation.

9. The method according to claim 1, characterized in that, The method further includes: Adding a flag text to the first text representation corresponding to each item in the item set to obtain an item input text corresponding to each item in the item set; Inputting the item input text corresponding to item a in the item set into the adjusted pre-trained language model, and obtaining an item embedding representation corresponding to item a through the adjusted pre-trained language model; Obtaining item embedding representations corresponding to each item in the item set, and adding the item embedding representations corresponding to each item to an item embedding table.

10. The method according to claim 9, characterized in that, The method further includes: Generating a third training sample of a sequence recommendation model according to the item embedding representations in the item embedding table and the historical item sequences corresponding to each object in the object set; Training the sequence recommendation model according to the third training sample to obtain a trained sequence recommendation model; Wherein, the training of the sequence recommendation model includes a first stage and a second stage. The first stage is used to train the sequence modeling task and pause the training of the item embedding representation; the second stage is used to train the item embedding representation and pause the training of the sequence modeling task; the trained sequence recommendation model is used for item recommendation.

11. A data processing device, characterized in that, Including: A text acquisition module, configured to acquire a first text representation corresponding to each item in an item set, and acquire a second text representation corresponding to a historical item sequence of each object in an object set according to the first text representation; A text prediction module, configured to generate a first training sample according to the second text representation, and perform text prediction on a hidden position in the first training sample according to a pre-trained language model to obtain a prediction probability corresponding to the hidden position; the first training sample includes a positive sample and a negative sample, and the number of hidden positions is N, and N is a positive integer; A first item comparison module, configured to perform item comparison on the first training sample and a first positive sample corresponding to the first training sample according to the pre-trained language model to obtain a first sample similarity; the first positive sample refers to a first text representation of a subsequent interaction item corresponding to the first training sample. A model pre-training module, configured to perform logarithmic operations on the prediction probabilities corresponding to N hidden positions to obtain the logarithmic values of the probabilities of each hidden position, and accumulate the logarithmic values of the probabilities of each hidden position to obtain a masked language modeling loss; The model pre-training module is further configured to perform a summation operation on the product of the hyperparameter and the masked language modeling loss, and the first sample similarity to obtain a model training loss, and train the pre-trained language model according to the model training loss to obtain a trained pre-trained language model; the hyperparameter is used to control the weight of the masked language modeling loss; A second item comparison module, configured to generate a second training sample according to the second text representation, and perform item comparison on the second training sample and the second positive sample corresponding to the second training sample according to the trained pre-trained language model to obtain a second sample similarity; the second training sample does not include a positive sample; A model adjustment module, configured to adjust the trained pre-trained language model according to the second sample similarity to obtain an adjusted pre-trained language model; the adjusted pre-trained language model is used to initialize the embedded representation of an item.

12. A computer device, characterized in that, Comprising a memory and a processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and the computer program is adapted to be loaded and executed by a processor so that a computer device having the processor executes the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, Comprising computer programs / instructions, and when the computer programs / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images