Method and apparatus for enhancing effect of recommendation system based on pre-trained language model

Abstract

The application discloses a recommendation system effect enhancement method and device based on a pre-trained language model, and relates to the technical field of artificial intelligence.The method comprises the following steps: collecting user behavior sequence data of multiple regions; extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions; fitting the user behavior vectors of the multiple regions to obtain a training data set; training an Item2vec model based on the training data set to obtain a trained Item2vec model; and predicting user behavior by using the trained Item2vec model.The application can optimize the recommendation preferences of a small part of users, and fuse and train the user behaviors of other regions, so that the training data set is increased, and the recommendation preferences of users in different regions can be comprehensively considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a recommendation system effect enhancement method and device based on a pre-trained language model. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior art is prior art nor, that anything in this section is "prior art" with respect to the application.

[0003] There are many types of recommendation systems used in the government field. In general, for a specific province or county, a local user behavior data is used to train and optimize the model. To some extent, this can obtain a recommendation model that is suitable for the habits of most users in the region, but at the same time, it ignores the recommendation preferences of a small part of users and special groups. For example, in A province, there is a small minority group in the province. The behavior data of local users is obtained, and the model is trained from the data. The recommendation result obtained by the model will be relatively unfriendly to the minority group, because the minority users in this area are relatively small, and the recommendation result obtained by the model training is not in line with the habits of the minority. This is a problem encountered by the current recommendation system. There is a barrier between the recommendation systems in the same field, and the amount of model training data of the recommendation system is also a big difficulty of the recommendation system. SUMMARY

[0004] The embodiment of the present application provides a recommendation system effect enhancement method based on a pre-trained language model, which is used to solve the problem that the prediction result is not friendly due to the use of data in one area for prediction. The method comprises:

[0005] Collecting user behavior sequence data of multiple regions;

[0006] Extracting semantic feature information of the user behavior sequence data of the multiple regions by a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions;

[0007] Fitting the user behavior vectors of the multiple regions to obtain a training data set;

[0008] Training an Item2vec model based on the training data set to obtain a trained Item2vec model, and using the trained Item2vec model to predict user behavior.

[0009] In one embodiment, collecting user behavior sequence data of multiple regions comprises:

[0010] Obtaining user behavior log files of multiple regions;

[0011] Filtering the user behavior log files of the plurality of regions to obtain user behavior sequence data of the plurality of regions.

[0012] In one embodiment, collecting the user behavior sequence data of the plurality of regions comprises:

[0013] Collecting user behavior data of the plurality of regions;

[0014] Cleaning the user behavior data of the plurality of regions to obtain required user behavior sequence data of the plurality of regions.

[0015] In one embodiment, extracting semantic feature information from the user behavior sequence data of the plurality of regions by a pre-trained semantic model respectively to obtain user behavior vectors of the plurality of regions comprises:

[0016] Extracting semantic feature information from the user behavior sequence data of the plurality of regions by a pre-trained semantic model BERT respectively to obtain user behavior vectors of the plurality of regions.

[0017] In one embodiment, extracting semantic feature information from the user behavior sequence data of the plurality of regions by a pre-trained semantic model LSTM or GPT respectively to obtain user behavior vectors of the plurality of regions.

[0018] In one embodiment, based on the user behavior vector of a region, mapping the user behavior vectors of the remaining regions to the user behavior vector of the region to obtain a mapping relationship;

[0019] Obtaining a training data set based on the mapping relationship.

[0020] In one embodiment, calculating the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region, and obtaining the mapping relationship based on the similarity.

[0021] In one embodiment, comparing the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region with a preset similarity;

[0022] When the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region meets the preset similarity, the mapping relationship is obtained.

[0023] In one embodiment, fitting the user behavior vectors of the plurality of regions to obtain a training data set, comprising:

[0024] Calculating the similarity of the user behavior vectors of the plurality of regions, and determining the user behavior vectors whose similarity meets a preset similarity;

[0025] Taking the user behavior vectors whose similarity meets the preset similarity as the training data set.

[0026] The embodiment of the application also provides a pre-trained language model-based recommendation system effect enhancement device to solve the problem of unfriendly prediction results caused by using data of one region for prediction.

[0027] A data collection module is configured to collect user behavior sequence data of multiple regions.

[0028] A semantic feature information extraction module is configured to extract semantic feature information of the user behavior sequence data of the multiple regions respectively through a pre-trained semantic model to obtain user behavior vectors of the multiple regions.

[0029] A fitting module is configured to fit the user behavior vectors of the multiple regions to obtain a training data set.

[0030] A training module is configured to train an Item2vec model based on the training data set to obtain a trained Item2vec model, and use the trained Item2vec model to perform user behavior prediction.

[0031] In one embodiment, the data collection module is specifically configured to:

[0032] Obtain user behavior log files of multiple regions.

[0033] Filter the user behavior log files of the multiple regions to obtain user behavior sequence data of the multiple regions.

[0034] In one embodiment, the data collection module is specifically configured to:

[0035] Collect user behavior data of multiple regions.

[0036] Clean the user behavior data of the multiple regions to obtain required user behavior sequence data of the multiple regions.

[0037] In one embodiment, the semantic feature information extraction module is specifically configured to:

[0038] Extract semantic feature information of the user behavior sequence data of the multiple regions respectively through a pre-trained semantic model BERT to obtain user behavior vectors of the multiple regions.

[0039] In one embodiment, the semantic feature information extraction module is specifically configured to:

[0040] Extract semantic feature information of the user behavior sequence data of the multiple regions respectively through a pre-trained semantic model LSTM or GPT to obtain user behavior vectors of the multiple regions.

[0041] In one embodiment, the fitting module is specifically configured to:

[0042] The user behavior vector of the remaining areas is mapped into the user behavior vector of the area, and a mapping relationship is obtained based on the user behavior vector of the area.

[0043] Based on the mapping relationship, a training data set is obtained.

[0044] In one embodiment, the fitting module is specifically configured to:

[0045] The similarity of the user behavior vector of the remaining areas and the user behavior vector of the area is calculated, and the mapping relationship is obtained based on the similarity.

[0046] In one embodiment, the fitting module is specifically configured to:

[0047] The similarity of the user behavior vector of the remaining areas and the user behavior vector of the area is compared with a preset similarity.

[0048] When the similarity of the user behavior vector of the remaining areas and the user behavior vector of the area meets the preset similarity, the mapping relationship is obtained.

[0049] In one embodiment, the fitting module is specifically configured to:

[0050] The similarity of the user behavior vectors of the plurality of areas is calculated, and the user behavior vector whose similarity meets the preset similarity is determined.

[0051] The user behavior vector whose similarity meets the preset similarity is taken as the training data set.

[0052] The embodiment of the application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned recommendation system effect enhancement method based on the pre-trained language model when executing the computer program.

[0053] The embodiment of the application also provides a computer readable storage medium, which stores a computer program, and the program is executed by the processor to implement the steps of the above-mentioned recommendation system effect enhancement method based on the pre-trained language model.

[0054] Compared with the prior art using data of one region for prediction, the embodiment of the present application collects user behavior sequence data of multiple regions, extracts semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model to obtain user behavior vectors of the multiple regions, fits the user behavior vectors of the multiple regions to obtain a training data set, trains an Item2vec model based on the training data set to obtain a trained Item2vec model, and uses the trained Item2vec model for user behavior prediction. The present application can optimize the recommendation preferences of a small part of users and fuse and train user behaviors of other regions, thereby increasing the training data set and comprehensively recommending the preferences of users in different regions. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort. In the drawings:

[0056] Figure 1 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (one);

[0057] Figure 2 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (two);

[0058] Figure 3 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (three);

[0059] Figure 4 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (four);

[0060] Figure 5 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (five);

[0061] Figure 6 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (six);

[0062] Figure 7 Flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the present application (seven);

[0063] Figure 8 A user behavior vector prediction flowchart in an embodiment of the present application;

[0064] Figure 9 A structural diagram of a recommendation model Item2vec in an embodiment of the present application;

[0065] Figure 10 A vector distance diagram of matters in different regions in an embodiment of the present application;

[0066] Figure 11 A structural block diagram of a recommendation system effect enhancement device based on a pre-training language model in an embodiment of the present application;

[0067] Figure 12 A schematic block diagram of a system configuration of the electronic device 600 of the embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, further detailed descriptions will be given to the embodiments of the present application in combination with the drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but not as a limitation to the present application.

[0069] Term explanation

[0070] Embedding: Embedding refers to representing an object, which can be a word, a commodity, or a movie, etc., with a low-dimensional dense vector. The embedding vector after embedding contains certain characteristics of the object, and the similarity (cosine) of embedding vectors of different objects can reflect the similarity between the objects.

[0071] Item2vec: Item2vec is an extension of word2vec, which changes the text sequence input of word2vec to item sequence input, and changes the embedding object from word to item. For example, assuming that we have many user behavior sequences that purchase [“beer”, “diapers”, xx] and other commodities, the training process of item2vec is to perform a vector coding for each commodity, and the trained model will make the similarity of beer and diapers higher than that of beer and others.

[0072] Natural language understanding pre-training model: Pre-training, in a broad sense, refers to training a model on a large-scale data first, obtaining a pre-training model irrelevant to a specific task from the large-scale data through self-supervised learning, and then fine-tuning in specific downstream task data. It reflects the semantic representation of a word in a specific context. The second step is fine-tuning, which corrects the network for a specific task.

[0073] For the problems existing in the prior art, the barriers of the recommendation system in the same field can be broken, the data and models of different recommendation systems in the same field are shared, and multiple recommendation systems are fused to optimize into a recommendation system with stronger applicability. Based on this, the application provides a recommendation system effect enhancement method based on a pre-trained language model. User behavior data in different provinces and regions are fused, feature extraction is performed through a pre-trained language model, the model is fused and trained, and a recommendation model with stronger generalization ability is obtained, and the recommendation preferences of more people are considered.

[0074] Figure 1 A flowchart of the recommendation system effect enhancement method based on the pre-trained language model in the embodiment of the application is shown in FIG. Figure 1 As shown in the figure, it includes:

[0075] Step 101: Collect user behavior sequence data in multiple regions.

[0076] Step 102: Extract semantic feature information of the user behavior sequence data in the multiple regions through a pre-trained semantic model respectively, and obtain user behavior vectors in the multiple regions.

[0077] Step 103: Fit the user behavior vectors in the multiple regions, and obtain a training data set.

[0078] Step 104: Train an Item2vec model based on the training data set, obtain a trained Item2vec model, and use the trained Item2vec model to predict user behavior.

[0079] In the embodiment of the application, in actual application, user behavior can exist in a user behavior log file, therefore, as shown in the figure, step 101 of collecting user behavior sequence data in multiple regions includes: Figure 2

[0080] Step 1011: Obtain user behavior log files in multiple regions.

[0081] Step 1012: Screen the user behavior log files in the multiple regions, and obtain user behavior sequence data in the multiple regions.

[0082] In the embodiment of the application, in actual application, user behavior can exist in a database, therefore, as shown in the figure, step 101 of collecting user behavior sequence data in multiple regions includes: Figure 3

[0083] Step 1013: Collect user behavior data in multiple regions.

[0084] ​​Step 1014: cleaning the user behavior data of the plurality of regions to obtain required user behavior sequence data of the plurality of regions.

[0085] Step 1011, step 1012 and step 1013, and step 1014 have no sequential relationship.

[0086] In the embodiment of the application, step 102 extracts semantic feature information from the user behavior sequence data of the plurality of regions respectively through a pre-trained semantic model to obtain user behavior vectors of the plurality of regions, including:

[0087] The semantic feature information is extracted from the user behavior sequence data of the plurality of regions respectively through a pre-trained semantic model BERT to obtain user behavior vectors of the plurality of regions.

[0088] Or, the semantic feature information is extracted from the user behavior sequence data of the plurality of regions respectively through a pre-trained semantic model LSTM or GPT to obtain user behavior vectors of the plurality of regions.

[0089] Specifically, the pre-trained semantic model proposed in the application can use BERT, LSTM, GPT, etc.

[0090] Bidirectional Encoder Representations from Transformers, abbreviated as BERT, the essence of BERT is to learn a good feature representation for a word by running a self-supervised learning method on a large amount of corpus, and the so-called self-supervised learning is a supervised learning run on data without human annotation. Corresponding to inputting a sentence, a semantic related vector corresponding to the sentence can be output.

[0091] LSTM (long-short term memory, long-short term memory model) is a special RNN model, which is proposed to solve the problem of gradient dispersion of RNN model.

[0092] GPT (Generative Pre-Training) is a generative pre-training, and GPT adopts a two-stage process, the first stage is pre-training by using a language model (unsupervised form), and the second stage solves a downstream task by using a Fine-tuning mode (supervised mode).

[0093] In the embodiment of the application, as shown in Figure 4 Step 103 fits the user behavior vectors of the plurality of regions to obtain a training data set, including:

[0094] Step 1031: mapping the user behavior vectors of the remaining regions into the user behavior vector of one region based on the user behavior vector of the region, to obtain a mapping relationship;

[0095] Step 1032: obtaining a training data set based on the mapping relationship.

[0096] In the embodiment of the present application, as shown in the figure, Figure 5 Step 1031 maps the user behavior vectors of the remaining regions into the user behavior vector of one region based on the user behavior vector of the region, to obtain a mapping relationship, including:

[0097] Step 1031 maps the user behavior vectors of the remaining regions into the user behavior vector of one region based on the user behavior vector of the region, to obtain a mapping relationship, including:

[0098] In the embodiment of the present application, as shown in the figure, Figure 6 Step 1031 maps the user behavior vectors of the remaining regions into the user behavior vector of one region based on the user behavior vector of the region, to obtain a mapping relationship, including:

[0099] Step 10311: comparing the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region with a preset similarity;

[0100] Step 10312: obtaining the mapping relationship when the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region meets the preset similarity.

[0101] In the embodiment of the present application, as shown in the figure, Figure 7 Step 103 maps the user behavior vectors of multiple regions to obtain a training data set, including:

[0102] Step 1033: calculating the similarity of the user behavior vectors of multiple regions, and determining the user behavior vectors whose similarity meets a preset similarity;

[0103] Step 1034: taking the user behavior vectors whose similarity meets the preset similarity as the training data set.

[0104] Step 1031, step 1032 and step 1033, step 1034 have no order relationship.

[0105] In the embodiment of the present application, the training model of the present application is still an item2Vec-based vector similarity recommendation model. Generally, a recommendation system collects user behavior data in an area to train the model, embeds the user behavior data in the area to obtain a behavior sequence after user embedding, and uses an item2Vec model to model. According to the principle analysis, the items interacted by the user in the recent period are relatively close in the vector space, so similar items of the items interacted by the user in the recent period can be searched in the vector space to make recommendations.

[0106] In the above scheme, collecting user behavior data in an area to train an item2Vec model can indeed obtain the recommendation preferences of most users. In this area, there are a part of minority users, and due to the problem of data quantity, the user behavior data of local minority users will be less, and using the data of the area to train the model will ignore the recommendation preferences of the minority users, resulting in poor experience of the minority users, so that a part of the recommendation preferences of the users will be lost. Therefore, the present application collects user behavior sequence data in multiple areas, extracts semantic feature information through a pre-trained semantic model, and then uses item2Vec to fuse multiple recommendation models, so as to obtain a recommendation model considering more users.

[0107] The following will start from actual examples to describe the technical implementation scheme in detail.

[0108] Starting from accumulated user behavior data in multiple provinces, the user behavior data needs to be cleaned first, and the required user data is selected from the log file. The user behavior in a session (from entering the APP to exiting the APP) is regarded as a sequence, so the extracted serialized behavior data is as follows:

[0109] userid: ['card handling', 'public accumulation fund payment information query','renewal of ID card',...]

[0110] Apparently, a problem is that the model will exist when training data is insufficient or the characteristic of the data is too strong, so that the behavior preferences of a small part of people are ignored. For example, in A province, there are only a small number of minority groups, which will result in that there is basically no minority matter in the collected user behavior data. Using such data to train the model will result in a very low recommendation probability of the matters related to the minority in the matters.

[0111] For example, Figure 8As shown, now the user behavior data of B province is to be used in A province, obviously there are some differences between different provinces, and the matters of B province need to be fitted into the matters of A province, so that the user data of B province can be used to train the model. In B province, all the matter names of the province (i.e. the matter set, including public accumulation fund query, temporary identity card application, enterprise basic information query) are input into the pre-trained semantic model to encode the matter, and the model will output a vector corresponding to the matter (B province matter vector), which will contain the semantic information related to the matter. At the same time, the same operation is performed on the matters of A province (i.e. the matter set, including query of public accumulation fund information, replacement of temporary identity card, enterprise information query), and the encoding of the matters of A province can be obtained, and the model will output a vector corresponding to the matters of A province (A province matter vector), which will contain the semantic information related to the matter. In this way, the matters of B province are mapped to the matters of A province, and only the similarity of the matter vectors needs to be calculated to obtain the mapping relationship. For example: in B province, a matter of handling social security card can be obtained in A province. In order to further improve the accuracy, in the later stage, manual intervention will be used for calibration and verification.

[0112] In actual processing, it is assumed that a matter handling recommendation system is to be made for A province, at this time there is a problem that the minority ethnic groups in the users of A province are few, resulting in that only a small part of the behavior sequence about the minority ethnic matter is accumulated in the user behavior data, for example: ['pay public accumulation fund','minority ethnic marriage registration','minority ethnic school information query',...], and the corresponding training data is lacking. The processing to be done is to find a province B with more minority ethnic groups among many provinces, and map the user behavior data of the province to A province. For some matters of B province, there may be a large difference from the matters of A province, and the similarity is low, which cannot be accurately mapped. For such a poor mapping, if the matter is related to the minority ethnic matter, the sequence in the sequence will be removed, such as: ['pay public accumulation fund','minority ethnic marriage registration','minority ethnic school information query',...], and'minority ethnic marriage registration' in the sequence cannot be well mapped, so a sequence of ['pay public accumulation fund','minority ethnic school information query',...] is taken, and the proportion of this part of data is appropriately reduced in the final statistics. Because the main target is to supplement the user sequence of the minority ethnic group, even if a part of the data is discarded, more behavior data related to the minority ethnic group can be obtained, and the influence on the final result is small.

[0113] The purpose of the present application is to obtain a larger data set for training. Obviously, with the same conditions, a larger data set can obtain more data, and the model can learn more information, so that better results can be obtained when recommending. The training set can be supplemented with a purpose. In the above example, the goal to be achieved is to supplement the missing behavior sequence of ethnic minorities. In this case, the provinces or regions where ethnic minorities are concentrated can be targeted to find more behavior sequences related to ethnic minorities in these areas. Therefore, when supplementing data, more attention is paid to behavior sequences related to ethnic minorities, which are processed and mapped to supplement part of the model's training data set to improve the model's recommendation effect.

[0114] Of course, in some cases, the region itself lacks data sets, so more data will be sought to map more behavior data to this region. At this time, there may also be some matters that cannot be accurately mapped. For this part of the matter, one is to manually check and confirm whether the matter can be mapped, and the other is to directly exclude this part of the matter from the sequence. Regardless of how to handle it, the purpose of supplementing the data set can indeed be achieved, and the model is provided with more user behavior information to learn, and a better recommendation system is obtained through this method.

[0115] When processing the mapping of matters, a pre-trained semantic model used in natural language processing is used. Through comparative analysis, the Bidirectional Encoder Representations from Transformers (BERT) based on the transformer is finally adopted. The matter vectors in the two regions are input into the model, so that the vector expression of the matter can be obtained. Finally, the similarity of the vectors is calculated to obtain the corresponding matter mapping.

[0116] After the above processing, the user data of B province can be used to supplement the training data set to train the model.

[0117] The model used during training is a variation of the commonly used word2vec model, Item2vec.

[0118] Word2vec is a group of related models used to generate word vectors. These models are shallow and double-layer neural networks that can quickly and effectively express a word as a vector based on a given corpus through an optimized training model. That is, after training, the word2vec model can be used to map each word to a vector, which can be used to represent the relationship between words.

[0119] Item2vec model is to migrate the algorithm idea of Word2vec Skipgram with Negative Sampling (SGNS) to item-based CF, to take the co-occurrence of items as the context relationship in natural language, and to construct a neural network to learn the vector representation of items in the hidden space. Item2vec model is used to generate item vectors, and finally only the similarity between the current browsed item and other items needs to be calculated to find similar items.

[0120] Based on the transformed Item2vec model, the data of different regions are fused together for training, and the structure diagram of the recommendation model is as shown in Figure 9

[0121] In the Item2vec model, the input of the model is a complete sentence. In contrast to the item recommendation, a complete sequence of a user is taken as the input (i.e. from the user logs in the APP to logs out of the APP), and such an input is a complete sequence, [apply for a social security card, query public accumulation fund, pay electricity bill, …].

[0122] First, the serialized behavior data of different regions are input into the Item2vec model. The input of the Item2vec model is the one-hot encoding of the context items, and then the respective is multiplied by the shared matrix W (V x N). V is the vector space of the items, and if there are 100 items, V is 100, and N can be set by the user. The vectors obtained by multiplication ([1, 0, 0, 0, …], [0, 1, 0, 0, …], [0, 0, 0, 1, …]) are added and averaged as the vector of the hidden layer (N-dimensional vector), and then multiplied by the output weight matrix W (N x V) to obtain the output layer vector (V-dimensional vector). The final obtained vector 1 x V is processed by the activation function to obtain the V-dim probability distribution, and the item corresponding to the maximum probability is the predicted item. According to the normal algorithm, the loss function is taken, and the gradient descent algorithm is used to update the weight matrix. Finally, the input layer matrix W (V x N) is the encoding vector of the corresponding item.

[0123] From the above model, the vector encoding of all items can be obtained, and finally the similarity of the items can be calculated to obtain the recommended result in the recall layer. When the model makes recommendations, the behavior sequence of the user in the recent period can be first vectorized, and then similar vectors are searched in the entire recommended range as the recall layer of the recommendation, and the recommended range is all the items in the local area.

[0124] ​Item2vec model is trained using CBOW. CBOW (Continuous Bag of Words Model) is essentially a neural network using three layers (including input layer, one hidden layer and output layer; of course, multiple layers can also be used). The coefficients of each layer can be trained, but in actual application, they are fixed as simple 1; the input and output are word vectors that need to be trained, and the CBOW algorithm inputs context and outputs target words;

[0125] Traditionally, CBOW uses softmax function in the output layer to represent the hit probability of the input word predicting the output word (probability value in the entire word space); and the cross-entropy of this probability and the probability of the real output word (one-hot vector) is the loss function;

[0126] The CBOW algorithm uses word vectors in the context window as input, sums (or takes the average) these vectors, and then obtains the relevance distribution with the output word space, and then uses the softmax function to obtain the hit probability in the entire output word space. The cross-entropy of the one-hot encoding of the target word is the loss value, and through the gradient of the loss with respect to the input and output word vectors, the gradient descent method can be used to obtain an iterative adjustment of the input and output word vectors.

[0127] Through comparative analysis, using user behavior data in a region to train the Item2vec model, the vector representation of each matter is obtained, and the similar vectors in the vector space are searched. In the result, it can be found that the similarity of matters in different regions is different. For example Figure 10 As shown in the figure, in A province, using only the user behavior data of the province, the vector distance of the two matters of identity card replacement and minority marriage registration will be farther apart; by fusing the user behavior data of other provinces and different recommendation systems in this field, the result shows that the vector distance of identity card replacement and minority marriage registration will be closer. By fusing the user behavior data of different regions for training, a recommendation model that takes into account the preferences of more users and groups is obtained. Therefore, in this model, not only the recommendation preferences of ordinary users are taken into account, but also the experience of a small part of the group (minority groups, etc.) is better.

[0128] Using the same field of recommendation system, the user behavior data of different regions is fused and trained to fully mine the correlation between matters, and then a recommendation model that meets the preferences of more users is trained.

[0129] The model can complement the data set to some extent, provide more data information for the missing region, and train and fit the model, and the main purpose of the application is to fit more user groups to consider more users and improve the recommendation effect. In the recommendation system in the same field, the data barrier is broken, and the data of multiple regions is used to train and fit the model to build the recommendation system.

[0130] The embodiment of the application also provides a recommendation system effect enhancement device based on a pre-trained language model, as described in the following embodiment. Since the principle of solving the problem of the device is similar to that of the recommendation system effect enhancement method based on the pre-trained language model, the implementation of the device can be referred to the implementation of the recommendation system effect enhancement method based on the pre-trained language model, and the repeated parts will not be described again.

[0131] Figure 11 The structure block diagram of the recommendation system effect enhancement device based on the pre-trained language model in the embodiment of the application is shown as Figure 11 The recommendation system effect enhancement device based on the pre-trained language model comprises:

[0132] The data acquisition module 02 is configured to acquire user behavior sequence data of multiple regions.

[0133] The semantic feature information extraction module 04 is configured to extract semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively, and obtain user behavior vectors of the multiple regions.

[0134] The fitting module 06 is configured to fit the user behavior vectors of the multiple regions to obtain a training data set.

[0135] The training module 08 is configured to train an Item2vec model based on the training data set to obtain a trained Item2vec model, and use the trained Item2vec model to predict user behavior.

[0136] In the embodiment of the application, the data acquisition module 02 is specifically configured to:

[0137] The data acquisition module is specifically configured to:

[0138] Acquire user behavior log files of multiple regions.

[0139] Filter the user behavior log files of the multiple regions to obtain user behavior sequence data of the multiple regions.

[0140] In the embodiment of the application, the data acquisition module 02 is specifically configured to:

[0141] Acquire user behavior data of multiple regions.

[0142] The user behavior data of the plurality of regions is cleaned to obtain required user behavior sequence data of the plurality of regions.

[0143] In the embodiment of the present application, the semantic feature information extraction module 04 is specifically configured to:

[0144] The user behavior sequence data of the plurality of regions are respectively extracted by the pre-trained semantic model BERT to obtain user behavior vectors of the plurality of regions.

[0145] In the embodiment of the present application, the semantic feature information extraction module 04 is specifically configured to:

[0146] The user behavior sequence data of the plurality of regions are respectively extracted by the pre-trained semantic model LSTM or GPT to obtain user behavior vectors of the plurality of regions.

[0147] In the embodiment of the present application, the fitting module 06 is specifically configured to:

[0148] Based on the user behavior vector of one region, the user behavior vectors of the remaining regions are mapped into the user behavior vector of the region to obtain a mapping relationship.

[0149] Based on the mapping relationship, a training data set is obtained.

[0150] In the embodiment of the present application, the fitting module 06 is specifically configured to:

[0151] The similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region is calculated, and the mapping relationship is obtained based on the similarity.

[0152] In the embodiment of the present application, the fitting module 06 is specifically configured to:

[0153] The similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region is compared with a preset similarity.

[0154] When the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region meets the preset similarity, the mapping relationship is obtained.

[0155] In the embodiment of the present application, the fitting module 06 is specifically configured to:

[0156] The similarity of the user behavior vectors of the plurality of regions is calculated, and the user behavior vector whose similarity meets a preset similarity is determined.

[0157] The user behavior vector whose similarity meets the preset similarity is taken as a training data set.

[0158] The embodiment of the present application also provides a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned recommendation system effect enhancement method based on a pre-trained language model when executing the computer program.

[0159] The embodiment of the present application also provides a computer device, which can be a desktop computer, a tablet computer and a mobile terminal, and the embodiment is not limited to this. In the embodiment, the computer device can refer to Figure 11 The device is incorporated herein, and the repeated parts will not be described.

[0160] As Figure 12 shown, the computer device 600 can include a central processor 100 and a memory 140; the memory 140 is coupled to the central processor 100.

[0161] In an embodiment, the recommendation system effect enhancement function based on the pre-trained language model can be integrated into the central processor 100. The central processor 100 can be configured to control as follows:

[0162] Collect user behavior sequence data of multiple regions;

[0163] Extract semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions;

[0164] Fit the user behavior vectors of the multiple regions to obtain a training data set;

[0165] Train an Item2vec model based on the training data set to obtain a trained Item2vec model, and use the trained Item2vec model to predict user behavior.

[0166] Wherein, collecting user behavior sequence data of multiple regions includes:

[0167] Collect user behavior data of multiple regions;

[0168] Clean the user behavior data of the multiple regions to obtain the required user behavior sequence data of the multiple regions.

[0169] Wherein, extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions includes:

[0170] Extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model BERT respectively to obtain user behavior vectors of the multiple regions.

[0171] wherein the user behavior vectors of the plurality of regions are fitted to obtain a training data set, including:

[0172] The similarity of the user behavior vectors of the plurality of regions is calculated, and the user behavior vector whose similarity satisfies a preset similarity is determined;

[0173] The user behavior vector whose similarity satisfies the preset similarity is taken as the training data set.

[0174] In another embodiment, the recommendation system effect enhancement device based on the pre-trained language model can be configured separately from the central processor 100, for example, the recommendation system effect enhancement device based on the pre-trained language model can be configured as a chip connected with the central processor 100, and the recommendation system effect enhancement function based on the pre-trained language model is realized through the control of the central processor.

[0175] As shown in Figure 12 The computer device 600 can also include a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the computer device 600 does not necessarily include all the components; in addition, the computer device 600 can also include components not mentioned above, which can refer to prior art.

[0176] The central processor 100, also known as a controller or operation control, can include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of each component of the computer device 600.

[0177] The memory 140, for example, can be one or more of a cache, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. Programs for executing relevant information can be stored. And the central processor 100 can execute the program stored in the memory 140 to realize information storage or processing, etc.

[0178] The input unit 120 provides input to the central processor 100. The input unit 120 is, for example, a key or touch input device. The power supply 170 is used to provide power to the computer device 600. The display 160 is used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0179] The memory 140 can be a solid state memory such as a read only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased and provided with more data, examples of which are sometimes referred to as EPROM, etc. The memory 140 can also be some other type of device. As shown in Figure 12 The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 can include an application / function storage 142 for storing application programs and function programs or for executing the flow of the operation of the computer device 600 by the central processor 100.

[0180] The memory 140 can also include a data storage 143 for storing data such as contacts, digital data, pictures, sounds and / or any other data used by the computer device. The driver storage 144 of the memory 140 can include various drivers of the computer device for communication functions and / or for performing other functions of the computer device such as a messaging application, an address book application, etc.

[0181] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processor 100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0182] Based on different communication technologies, a plurality of communication modules 110 such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. can be provided in the same computer device. As shown in Figure 12 The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby implementing a conventional telecommunication function. The audio processor 130 can include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 130 is also coupled to the central processor 100, thereby enabling recording on the local by the microphone 132 and enabling playing of the sound stored on the local by the speaker 131.

[0183] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the following steps of the method for enhancing the effect of a recommendation system based on a pre-trained language model:

[0184] Collecting user behavior sequence data of a plurality of regions;

[0185] The user behavior sequence data of the plurality of regions are respectively extracted by a pre-trained semantic model to obtain user behavior vectors of the plurality of regions;

[0186] The user behavior vectors of the plurality of regions are fitted to obtain a training data set;

[0187] The Item2vec model is trained based on the training data set to obtain a trained Item2vec model, and the trained Item2vec model is used for user behavior prediction.

[0188] The user behavior sequence data of the plurality of regions are collected, including:

[0189] The user behavior data of the plurality of regions are collected;

[0190] The user behavior data of the plurality of regions are cleaned to obtain required user behavior sequence data of the plurality of regions.

[0191] The user behavior sequence data of the plurality of regions are respectively extracted by a pre-trained semantic model to obtain user behavior vectors of the plurality of regions, including:

[0192] The user behavior sequence data of the plurality of regions are respectively extracted by a pre-trained semantic model embedding to obtain user behavior vectors of the plurality of regions.

[0193] The user behavior vectors of the plurality of regions are fitted to obtain a training data set, including:

[0194] The similarity of the user behavior vectors of the plurality of regions is calculated, and user behavior vectors with a similarity satisfying a preset similarity are determined;

[0195] The user behavior vectors with the similarity satisfying the preset similarity are used as the training data set.

[0196] In summary, in the process of using the traditional recommendation system, the user data of the local region is often used to establish a model. If most users in the region will go to "medical information query" after "identity card replacement", using the user data of this region to train the model will lose part of the user's preferences. At the same time, this small part of the users can be ethnic groups, and they will go to "ethnic marriage registration" after "identity card replacement", so the recommendation preferences of these users will be ignored. In the method of the present application, the user behavior data of different regional provinces is fully extracted, and the habits of different regions are fully integrated, the data is re-input into the model, and the item2Vec is used for training, and a more widely applicable recommendation model will be obtained.

[0197] The object to be achieved by the present application is to build a more user-applicable matter recommendation system. User behavior data of enough regions are selected, feature extraction is performed using a pre-trained semantic model starting from basic matter names, and finally an item2Vec model is used to fuse different recommendation models, training and optimization are performed, and a more general recommendation model is obtained. The recommendation system can optimize the recommendation preferences of a small part of users, and fuse the user behaviors of other regions for training, which not only increases the training data set, but also integrates the recommendation preferences of users in different regions.

[0198] Those skilled in the art will appreciate that embodiments of the application can be supplied as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Also, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) having computer-usable program code embodied thereon.

[0199] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce an apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0200] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1steps of the functions specified in the one or more blocks.

[0202] The above-described embodiments of the present application are merely intended to further explain the purpose, technical solutions and advantages of the present application, and are not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for enhancing the effect of a recommendation system based on a pre-trained language model, characterized in that, The method comprises the following steps: collecting user behavior sequence data of multiple regions; extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions; fitting the user behavior vectors of the multiple regions to obtain a training data set; training an Item2vec model based on the training data set to obtain a trained Item2vec model, and using the trained Item2vec model to predict user behavior; wherein fitting the user behavior vectors of the multiple regions to obtain the training data set comprises: mapping the user behavior vectors of the remaining regions into the user behavior vector of a region based on the user behavior vector of the region to obtain a mapping relationship; and obtaining the training data set based on the mapping relationship; wherein mapping the user behavior vectors of the remaining regions into the user behavior vector of a region to obtain a mapping relationship comprises: calculating the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region, and obtaining the mapping relationship based on the similarity; obtaining the mapping relationship based on the similarity comprises: comparing the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region with a preset similarity; and obtaining the mapping relationship when the similarity of the user behavior vectors of the remaining regions and the user behavior vector of the region meets the preset similarity; fitting the user behavior vectors of the multiple regions to obtain the training data set comprises: calculating the similarity of the user behavior vectors of the multiple regions, determining the user behavior vectors whose similarity meets a preset similarity, and taking the user behavior vectors whose similarity meets the preset similarity as the training data set. 2.The pre-trained language model based recommendation system effect enhancement method of claim 1, wherein, Collecting user behavior sequence data of multiple regions comprises: obtaining user behavior log files of the multiple regions; filtering the user behavior log files of the multiple regions to obtain user behavior sequence data of the multiple regions. 3.The pre-trained language model based recommendation system effect enhancement method of claim 1, wherein, Collecting user behavior sequence data of multiple regions comprises: collecting user behavior data of the multiple regions; cleaning the user behavior data of the multiple regions to obtain the required user behavior sequence data of the multiple regions. 4.The pre-trained language model based recommendation system effect enhancement method of claim 1, wherein, Extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions comprises: extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model BERT respectively to obtain user behavior vectors of the multiple regions. 5.The pre-trained language model based recommendation system effect enhancement method of claim 1, wherein, Extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions comprises: extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model LSTM or GPT respectively to obtain user behavior vectors of the multiple regions. 6.A device for enhancing the effect of a recommendation system based on a pre-trained language model, characterized by, The method comprises the following steps: a data collection module for collecting user behavior sequence data of multiple regions; a semantic feature information extraction module for extracting semantic feature information of the user behavior sequence data of the multiple regions through a pre-trained semantic model respectively to obtain user behavior vectors of the multiple regions; The fitting module is configured to fit user behavior vectors of multiple regions to obtain a training data set; The training module is configured to train an Item2vec model based on the training data set to obtain a trained Item2vec model, and use the trained Item2vec model to perform user behavior prediction; The fitting module is specifically configured to map user behavior vectors of other regions to a user behavior vector of a region based on the user behavior vector of the region to obtain a mapping relationship; The training data set is obtained based on the mapping relationship; The fitting module is specifically configured to calculate the similarity of the user behavior vectors of the other regions and the user behavior vector of the region, and obtain the mapping relationship based on the similarity; The fitting module is specifically configured to compare the similarity of the user behavior vectors of the other regions and the user behavior vector of the region with a preset similarity; when the similarity of the user behavior vectors of the other regions and the user behavior vector of the region meets the preset similarity, the mapping relationship is obtained. The fitting module is specifically configured to calculate the similarity of the user behavior vectors of multiple regions, determine the user behavior vectors whose similarity meets a preset similarity, and use the user behavior vectors whose similarity meets the preset similarity as the training data set. 7.The pre-trained language model based recommendation system effect enhancement apparatus of claim 6, wherein, The data collection module is specifically configured to: Collect user behavior log files of multiple regions; Filter the user behavior log files of the multiple regions to obtain user behavior sequence data of the multiple regions. 8.The pre-trained language model based recommendation system effect enhancement apparatus of claim 6, wherein, The data collection module is specifically configured to: Collect user behavior data of multiple regions; Clean the user behavior data of the multiple regions to obtain required user behavior sequence data of the multiple regions. 9.The pre-trained language model based recommendation system effect enhancement apparatus of claim 6, wherein, The semantic feature information extraction module is specifically configured to: Extract semantic feature information from the user behavior sequence data of the multiple regions respectively through a pre-trained semantic model BERT to obtain user behavior vectors of the multiple regions. 10.The pre-trained language model based recommendation system effect enhancement apparatus of claim 6, wherein, The semantic feature information extraction module is specifically configured to: Extract semantic feature information from the user behavior sequence data of the multiple regions respectively through a pre-trained semantic model LSTM or GPT to obtain user behavior vectors of the multiple regions.

11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the recommendation system effect enhancement method based on the pre-trained language model according to any one of claims 1 to 5.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps of the recommendation system effect enhancement method based on the pre-trained language model according to any one of claims 1 to 5.

Images