A method, device and equipment for training and transferring pre-trained language models

Abstract

The present invention discloses a method, device and equipment for training and transferring demands of a pre-trained language model, wherein the training method comprises: obtaining multiple sets of training samples; calculating the text vectors of two pieces of demand information in the current training sample through a preset pre-trained language model; predicting the category and category matching of the demand information based on the text vector to obtain the predicted category and the predicted category matching result; correcting the model parameters of the preset pre-trained language model according to a first error and a second error, wherein the first error is the error between the predicted category and the category label, and the second error is the error between the predicted category matching result and the category matching label; when the pre-trained language model meets the preset conditions, outputting the pre-trained language model. The technical solution provided by the present invention improves the accuracy of the vector expression of the demand information.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis, and in particular to a method, device and equipment for training and transferring demands of a pre-trained language model. Background Art

[0002] Usually, enterprises, institutions and government agencies have a public service platform that is specially used to handle hotline matters, which is composed of telephone, mailbox, mobile phone text messages, mobile phone APP, Weibo, WeChat and other means. It integrates the consultation hotlines set up by various departments at all levels to provide business inquiries, consultations, complaints, help, public services, opinion and suggestion collection, public opinion surveys and other consultation hotlines for the public. When receiving the user's appeal information, an important link is to analyze the matters in the appeal information, so as to distribute the appeal information to the corresponding processing department for the processing of the matters. With the development of natural language processing technology, more and more local governments use natural language processing algorithms instead of manual distribution. Among them, an important technical core is how to learn the vector expression of each matter appeal information through natural language algorithms based on historical data, that is, to convert the text, words or sentences of the appeal information into digital vector form, and then perform downstream matter analysis and appeal distribution tasks based on the word vector, word vector or sentence vector in the appeal information. The existing technology usually uses pre-trained language models such as word2vector and Bert to directly extract the vector expression of the appeal information, but after practical verification, the actual meaning of the words and sentences represented by the vector is not accurate enough, which leads to poor results in the downstream appeal distribution tasks. Therefore, how to improve the accuracy of demand allocation by improving the accuracy of vector expression of demand information is an urgent problem to be solved. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method, device and equipment for training and transferring a pre-trained language model, thereby improving the accuracy of the vector expression of the demand information.

[0004] According to a first aspect, the present invention provides a training method for a pre-trained language model, the method comprising: obtaining multiple groups of training samples, the training samples comprising two semantically similar appeal information, and category labels and category matching labels corresponding to the two appeal information, the category matching labels being used to characterize whether the two appeal information belong to the same category; calculating text vectors of the two appeal information in the current training sample using a preset pre-trained language model; predicting the category and category matching of the appeal information based on the text vector to obtain a predicted category and a predicted category matching result; correcting model parameters of the preset pre-trained language model according to a first error and a second error, the first error being an error between the predicted category and the category label, and the second error being an error between the predicted category matching result and the category matching label; and outputting the pre-trained language model when the pre-trained language model meets preset conditions.

[0005] Optionally, obtaining multiple groups of training samples includes: traversing each demand information in the historical demand information library, and combining the demand information in the historical demand information library whose similarity is above a preset threshold based on the BM25 algorithm; obtaining the category label and category matching label corresponding to each group of demand information, and then obtaining multiple groups of training samples.

[0006] Optionally, the predicting of the category and category matching of the appeal information based on the text vector to obtain the predicted category and predicted category matching results includes: inputting two text vectors into a first fully connected layer respectively, calculating the predicted categories of the two text vectors respectively, and correcting the parameters of the first fully connected layer by the first error; inputting the two text vectors into a second fully connected layer at the same time, calculating the predicted category matching results of the two text vectors, and correcting the parameters of the second fully connected layer by the second error.

[0007] According to the second aspect, the present invention provides a demand transfer method, which includes: receiving target demand information of a target user, converting the target demand information into a target text vector through a pre-trained language model trained by any optional implementation method of the first aspect; extracting candidate demand information with semantics similar to the target demand information from a historical demand information library using a first similarity algorithm; calculating the candidate text vector corresponding to each candidate demand information through the pre-trained language model; calculating the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and finding the candidate text vector with the highest similarity to the target text vector; and transferring the target demand information to the processing department corresponding to the candidate text vector with the highest similarity.

[0008] Optionally, the first similarity algorithm is a BM25 algorithm, and the extraction of candidate demand information that is semantically similar to the target demand information from the historical demand information library using the first similarity algorithm includes: traversing each demand information in the historical demand information library, and using the BM25 algorithm to select candidate demand information in the historical demand information library whose similarity with the target demand information is above a preset threshold.

[0009] Optionally, the method further includes: calculating text vectors of each demand information in the historical demand information library using a pre-trained language model trained by any optional implementation method of the first aspect; establishing an association relationship between the calculated text vector and the corresponding demand information, and storing the text vector in the historical demand information library.

[0010] Optionally, the second similarity algorithm is a second fully connected layer.

[0011] According to a third aspect, the present invention provides a demand switching device, which includes: a demand receiving module, used to receive target demand information of a target user, and convert the target demand information into a target text vector through a pre-trained language model trained by any optional implementation method of the first aspect; a candidate demand module, used to extract candidate demand information with semantics similar to the target demand information from a historical demand information library using a first similarity algorithm; a vector conversion module, used to calculate the candidate text vector corresponding to each candidate demand information through the pre-trained language model; a demand matching module, used to calculate the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and find the candidate text vector with the highest similarity to the target text vector; a switching module, used to switch the target demand information to the processing department corresponding to the candidate text vector with the highest similarity.

[0012] According to the fourth aspect, an embodiment of the present invention provides a demand switching device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method described in the first aspect, the second aspect, or any optional implementation manner of the first and second aspects by executing the computer instructions.

[0013] According to the fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect, the second aspect, or any optional implementation of the first and second aspects.

[0014] The technical solution provided by this application has the following advantages:

[0015] The technical solution provided by the present application first constructs a sample pair consisting of two semantically similar appeal information, then divides the two appeal information into two pre-trained language models with exactly the same input model parameters, then calculates the text vectors of the two appeal information, and then optimizes the model through two errors: a fully connected layer is used to predict the processing department of each appeal information based on the text vector of the appeal information, thereby calculating the error between the predicted processing department and the actual processing department, and another fully connected layer is used to comprehensively calculate the matching degree of the text vectors of the two appeal information, thereby predicting whether the two appeal information belong to the same processing department, and calculating the error between the predicted matching result and the actual matching result. Based on the above two-way optimization goals, a large number of samples are used to optimize the model parameters of the pre-trained language model, which can greatly improve the accuracy of the text vector calculated by the pre-trained language model compared to traditional single model training techniques.

[0016] In the actual detection stage, the first similarity algorithm is used to find multiple candidate demand information similar to the target demand information from a large amount of demand information in the historical demand information database by text matching to reduce redundant data. Then, the text vector of the candidate demand information is extracted through the trained pre-trained language model, and the vector similarity is calculated between the text vector and the text vector of the target demand information to find the candidate text vector with the highest similarity. Then, the target demand information is assigned to the same processing department as the candidate text vector to complete the accurate allocation of the processing department.

[0017] In addition, in the actual detection stage, offline calculations were also performed. The text vectors of each demand information in the historical demand information database were calculated in advance through the trained pre-trained language model, and the text vectors were stored. In the subsequent detection stage, when the text vector of the target demand information was calculated for similarity with the text vector of the candidate demand information in the historical demand information database, it was only necessary to extract the corresponding text vector from the historical demand information database, without calculating the text vector of the candidate demand information. This improved the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0019] Figure 1 A schematic diagram showing the steps of a training method for a pre-trained language model in one embodiment of the present invention is shown;

[0020] Figure 2 A schematic diagram showing a flow chart of a training method for a pre-trained language model in one embodiment of the present invention;

[0021] Figure 3A schematic diagram showing the steps of a method for transferring a demand in one embodiment of the present invention is shown;

[0022] Figure 4 A schematic flow chart of a method for transferring a request in one embodiment of the present invention is shown;

[0023] Figure 5 A schematic structural diagram of a demand switching device in one embodiment of the present invention is shown;

[0024] Figure 6 A schematic structural diagram of an electronic device in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 and Figure 2 In one embodiment, a method for training a pre-trained language model comprises the following steps:

[0027] Step S101: obtaining multiple sets of training samples, wherein the training samples include two pieces of demand information with similar semantics, and category labels and category matching labels corresponding to the two pieces of demand information, wherein the category matching labels are used to indicate whether the two pieces of demand information belong to the same category.

[0028] Step S102: Calculate the text vectors of the two pieces of appeal information in the current training sample using a preset pre-trained language model.

[0029] Step S103: predicting the category and category matching of the appeal information based on the text vector to obtain the predicted category and predicted category matching results.

[0030] Step S104: Correcting the model parameters of the preset pre-trained language model according to the first error and the second error, wherein the first error is the error between the predicted category and the category label, and the second error is the error between the predicted category matching result and the category matching label.

[0031] Step S105: When the pre-trained language model meets the preset conditions, the pre-trained language model is output.

[0032] Specifically, in this embodiment, a dual-channel, multi-task training method is proposed to solve the problem that the single pre-trained language model in the prior art is trained on training samples, and its training effect is not good. If the model parameters of the pre-trained language model can simultaneously match the prediction effect of multiple targets, the accuracy of the model parameters of the pre-trained language model can be greatly improved. In this embodiment, a pre-trained language model is preferably preset, and the models that can be used include but are not limited to: word2vector, Bert, ALBert, MacBert. In this embodiment, MacBert is used to calculate the text vector of the appeal information (the specific operation is to use MacBert to calculate the word vector of the appeal information, and then merge the word vector into a sentence vector through a pooling layer). Then, training samples for implementing the dual-channel training method are constructed, and semantically similar appeal information is combined in pairs to form multiple sample pairs. The calculation process of each sample pair is a round of training. During training, the two appeal information of a sample pair are input into the MacBert model separately, two text vectors are calculated, and then a fully connected layer is added to MacBert (each node of the fully connected layer is connected to all nodes of the previous layer, which is used to integrate the features extracted previously and plays the role of a "classifier" in the entire neural network) to perform two training operations of "predicting categories" and "predicting whether categories match". In this embodiment, the category label of the appeal information is the processing department corresponding to the appeal information. On the one hand, the processing departments of the two text vectors are predicted, and the error (i.e., the first error) between the predicted processing department and the actual processing department corresponding to each of them is calculated. Then, the model parameters of the fully connected layer and MacBert are corrected at the same time with the goal of minimizing the error. On the other hand, it is predicted whether the processing departments of the two texts are the same processing department. The second error is calculated by the predicted matching result obtained by prediction and the actual matching of the two appeal information. The model parameters of the fully connected layer and MacBert are corrected at the same time through the second error. The loss function for optimizing the above objectives is as follows:

[0033]

[0034] Among them, λ 1 +2×λ 2 =1,λ 1 is the weight of the loss function for the binary classification task, λ 2 is the weight of the loss function of the multi-classification task, which can be adjusted according to the actual situation. Loss is the cross entropy loss function, and LOSS is the overall loss function. true and pred They are the actual matching results and the predicted matching results, label1 true and label1 predThey are the actual processing department and the prediction processing department of the appeal information, label2 true and label2 pred They are respectively the actual processing department and the predictive processing department of another demand information.

[0035] Afterwards, when the pre-trained language model meets the preset conditions, the training is terminated. In this embodiment, the preset conditions include but are not limited to: 1. All the multiple sets of training samples are input into the preset pre-trained language model for a round of training; 2. The training is terminated after the preset number of trainings; 3. Multiple sets of training samples are input into the preset pre-trained language model for cyclic training until the model parameters tend to a stable value that almost does not change. After the training is completed with a large number of samples, the model parameters of the Macbert model are recorded, and the Macbert model is applied to the text vector of the calculated appeal information, which can greatly improve the accuracy of the text vector.

[0036] Specifically, in one embodiment, the above step S101 specifically includes the following steps:

[0037] Step 1: Traverse each demand information in the historical demand information database, and use the BM25 algorithm to combine the demand information in the historical demand information database with similarity above the preset threshold.

[0038] Step 2: Obtain the category label and category matching label corresponding to each set of demand information, and then obtain multiple sets of training samples.

[0039] Specifically, in this embodiment, the BM25 algorithm is used to construct data. For each record in the historical data, N similar historical records are found and combined with the record in pairs to form N demand content pairs. At the same time, labels can be constructed based on the historical records: if the categories of the two demand contents are the same, the label is 1, otherwise the label is 0. The final constructed data set is in the form of [(s1, label1, s2, label2, y), ...], where s1 and s2 are demand contents, label1 and label2 are the category labels corresponding to s1 and s2 respectively, and y indicates whether the category labels of s1 and s2 are the same. The BM25 algorithm is an algorithm used to evaluate the relevance between documents. It is proposed based on the probabilistic retrieval model. First, two pairs of texts are segmented, and then the similarity of the two texts is scored based on each segmentation. Compared with the TF-IDF text matching algorithm, the algorithm is more flexible and powerful in application, and has higher practicality, so that the construction of sample pairs is more accurate, avoiding the situation where the content semantics in the sample pairs are greatly different, so as to improve the accuracy of subsequent training.

[0040] Specifically, in one embodiment, the above step S103 specifically includes the following steps:

[0041] Step 3: Input the two text vectors into the first fully connected layer respectively, calculate the predicted categories of the two text vectors respectively, and correct the parameters of the first fully connected layer by the first error.

[0042] Step 4: Input the two text vectors into the second fully connected layer at the same time, calculate the predicted matching results of the two text vectors, and correct the parameters of the second fully connected layer through the second error.

[0043] Specifically, in this embodiment, the fully connected layer of the pre-trained language model that implements dual-channel optimization is divided into two types, one for predicting the category of the text vector, and the other for predicting the matching result of whether two text vectors belong to the same category. For example: if the two text vectors are a 1×10 matrix, the input dimension of the first fully connected layer is set to 10, and the input dimension of the second fully connected layer is set to 20. The category label in the training sample label and the output of the first fully connected layer can calculate the first error, so that the parameters of the first fully connected layer and the pre-trained language model are corrected simultaneously according to the error between the predicted category and the true category label. On the other hand, the category matching label in the training sample label and the output of the second fully connected layer can calculate the second error. The second fully connected layer essentially calculates the similarity of the two text vectors, and then determines whether the two text vectors belong to the same category based on the size relationship between the similarity value and the preset threshold. For example, if the similarity value output by the second fully connected layer is greater than the preset threshold 0.5, it is considered that the two text vectors belong to the same category. Based on the above second error, the parameters of the second fully connected layer and the pre-trained language model are corrected simultaneously. After the training is completed, the parameters of the pre-trained language model, the first fully connected layer, and the second fully connected layer are all accurate parameters. In actual detection, if the pre-trained language model is combined with the fully connected layer, it is possible to calculate the text vector of the target demand information, and also to match the demand information belonging to the same department as the target demand information, thereby improving the detection efficiency and accuracy.

[0044] See also Figure 3 In one embodiment, a method for transferring a request includes the following steps:

[0045] Step S201: Receive target demand information of a target user, and convert the target demand information into a target text vector through the pre-trained language model trained in steps S101 to S105.

[0046] Step S202: extracting candidate demand information that is semantically similar to the target demand information from the historical demand information database using a first similarity algorithm.

[0047] Step S203: Calculate the candidate text vector corresponding to each candidate appeal information through the pre-trained language model.

[0048] Step S204: Calculate the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and find the candidate text vector with the highest similarity to the target text vector.

[0049] Step S205: Transfer the target appeal information to the processing department corresponding to the candidate text vector with the highest similarity.

[0050] Specifically, in this embodiment, the pre-trained language model obtained through the training in the above training method can accurately calculate the target text vector of the target user's target demand information, and calculate the text vectors of other demand information with similar semantics to the target demand information. After that, the target text vector and other text vectors are subjected to pairwise similarity calculations to obtain the text vector with the highest similarity to the target text vector (hereinafter referred to as the "matching text vector"). After that, the processing department of the demand information corresponding to the matching text vector is obtained, and the target demand information can be accurately allocated to the processing department, so that the external staff of the processing department can accurately process the target demand information. The pre-trained model obtained through the above training method can accurately calculate the text vectors of each demand information, thereby greatly improving the accuracy of similarity calculation, and further improving the accuracy of target demand information allocation.

[0051] Specifically, in this embodiment, the implementation method of the above step S202 is: traverse each demand information in the historical demand information library, and use the BM25 algorithm to select candidate demand information in the historical demand information library whose similarity with the target demand information is above a preset threshold. Specifically, a preset number of demand information with semantically similar target demand information is extracted from the historical demand information library in advance. When the similarity between a certain demand information calculated by the BM25 algorithm and the target demand information is above the preset threshold, it is extracted as candidate demand information. For example, the preset threshold is 80, and any demand information in the historical demand information library is used as candidate demand information when the score calculated by the BM25 algorithm is above 80 points. In this way, in the subsequent precise matching, repeated calculation of redundant data is avoided, and the calculation efficiency is improved. The explanation of the BM25 algorithm can be found in the description in the above training method embodiment, which will not be repeated here.

[0052] Specifically, Figure 4 As shown, in one embodiment, a request transfer method further includes the following steps:

[0053] Step 5: Calculate the text vector of each demand information in the historical demand information database through the pre-trained language model trained in steps S101 to S105;

[0054] Step 6: Establish an association between the calculated text vector and the corresponding demand information, and store it in the historical demand information database.

[0055] Specifically, in this embodiment, before detecting the target demand information, the pre-trained language model is used to calculate the text vectors corresponding to all the demand information in the historical demand information library, and then all the text vectors are stored in the historical demand information library to complete the offline processing. Afterwards, when the BM25 algorithm is used to find candidate demand information similar to the target demand information from the historical demand information library, the text vector corresponding to the candidate demand information is directly searched from the historical demand information library again, saving a large number of calculation steps and improving the efficiency of demand information distribution.

[0056] Specifically, in one embodiment, the second similarity algorithm in the above step S204 is a second fully connected layer. Specifically, after obtaining the target text vector and each candidate text vector, the similarity thereof can be calculated by methods including but not limited to vector inner product and Euclidean distance. However, in this embodiment, since the second fully connected layer is trained at the same time during the training of MacBert, the layer can analyze the results of whether the two text vectors match or not. For example, label 1 during training represents a match, and label 0 represents a mismatch. After training, the actual output result of the second fully connected layer is to map the similarity of the two text vectors to the interval [0,1] through an activation function. Assuming that the output value of the second fully connected layer is 0.6, the similarity set between the target text vector and the current candidate text vector is considered to be 0.6. Then, by comparing the output value with the preset threshold, the matching result of whether the two text vectors match or not is obtained. The second fully connected layer trained with a large amount of data has a higher accuracy in calculating the similarity than the traditional vector inner product method, so the second fully connected layer is used to calculate the similarity of the text vector, which further improves the accuracy of the distribution of the appeal information to the corresponding processing department.

[0057] Through the above steps, the technical solution provided by the present application first constructs a sample pair consisting of two semantically similar appeal information, then divides the two appeal information into two pre-trained language models with exactly the same input model parameters, and then calculates the text vectors of the two appeal information. Then, the model is optimized through two errors: a fully connected layer is used to predict the processing department of each appeal information based on the text vector of the appeal information, thereby calculating the error between the predicted processing department and the actual processing department, and another fully connected layer is used to comprehensively calculate the matching degree of the text vectors of the two appeal information, thereby predicting whether the two appeal information belong to the same processing department, and calculating the error between the predicted matching result and the actual matching result. Based on the above two-way optimization goals, a large number of samples are used to optimize the model parameters of the pre-trained language model, which can greatly improve the accuracy of the text vector calculated by the pre-trained language model compared to traditional single model training techniques.

[0058] In the actual detection stage, the first similarity algorithm is used to find multiple candidate demand information similar to the target demand information from a large amount of demand information in the historical demand information database by text matching to reduce redundant data. Then, the text vector of the candidate demand information is extracted through the trained pre-trained language model, and the vector similarity is calculated between the text vector and the text vector of the target demand information to find the candidate text vector with the highest similarity. Then, the target demand information is assigned to the same processing department as the candidate text vector to complete the accurate allocation of the processing department.

[0059] In addition, in the actual detection stage, offline calculations were also performed. The text vectors of each demand information in the historical demand information database were calculated in advance through the trained pre-trained language model, and the text vectors were stored. In the subsequent detection stage, when the text vector of the target demand information was calculated for similarity with the text vector of the candidate demand information in the historical demand information database, it was only necessary to extract the corresponding text vector from the historical demand information database, without calculating the text vector of the candidate demand information. This improved the detection efficiency.

[0060] like Figure 5 As shown, this embodiment also provides a demand switching device, which includes:

[0061] The appeal receiving module 201 is used to receive the target appeal information of the target user, and convert the target appeal information into a target text vector through the pre-trained language model trained in steps S101 to S105. For details, please refer to the relevant description of step S201 in the above method embodiment, which will not be repeated here.

[0062] The candidate demand module 202 is used to extract candidate demand information with semantic similarity to the target demand information from the historical demand information database using a first similarity algorithm. For details, please refer to the relevant description of step S202 in the above method embodiment, which will not be repeated here.

[0063] The vector conversion module 203 is used to calculate the candidate text vector corresponding to each candidate appeal information through the pre-trained language model. For details, please refer to the relevant description of step S203 in the above method embodiment, which will not be repeated here.

[0064] The demand matching module 204 is used to calculate the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and find the candidate text vector with the highest similarity to the target text vector. For details, please refer to the relevant description of step S204 in the above method embodiment, which will not be repeated here.

[0065] The transfer module 205 is used to transfer the target appeal information to the processing department corresponding to the candidate text vector with the highest similarity. For details, please refer to the relevant description of step S205 in the above method embodiment, which will not be repeated here.

[0066] The demand transfer device provided in the embodiment of the present invention is used to execute the demand transfer method provided in the above embodiment. Its implementation method and principle are the same. For details, please refer to the relevant description of the above method embodiment, which will not be repeated here.

[0067] Through the collaborative cooperation of the above-mentioned components, the technical solution provided by this application first constructs a sample pair consisting of semantically similar demand information, and then divides the two demand information into two pre-trained language models with exactly the same input model parameters, and then calculates the text vectors of the two demand information. After that, the model is optimized through two errors: a fully connected layer is used to predict the processing department of each demand information based on the text vector of the demand information, thereby calculating the error between the predicted processing department and the actual processing department, and another fully connected layer is used to comprehensively calculate the matching degree of the text vectors of the two demand information, thereby predicting whether the two demand information belong to the same processing department, and calculating the error between the predicted matching result and the actual matching result. Based on the above two-way optimization goals, a large number of samples are used to optimize the model parameters of the pre-trained language model, which can greatly improve the accuracy of the text vector calculated by the pre-trained language model compared to traditional single model training techniques.

[0068] In the actual detection stage, the first similarity algorithm is used to find multiple candidate demand information similar to the target demand information from a large amount of demand information in the historical demand information database by text matching to reduce redundant data. Then, the text vector of the candidate demand information is extracted through the trained pre-trained language model, and the vector similarity is calculated between the text vector and the text vector of the target demand information to find the candidate text vector with the highest similarity. Then, the target demand information is assigned to the same processing department as the candidate text vector to complete the accurate allocation of the processing department.

[0069] In addition, in the actual detection stage, offline calculations were also performed. The text vectors of each demand information in the historical demand information database were calculated in advance through the trained pre-trained language model, and the text vectors were stored. In the subsequent detection stage, when the text vector of the target demand information was calculated for similarity with the text vector of the candidate demand information in the historical demand information database, it was only necessary to extract the corresponding text vector from the historical demand information database, without calculating the text vector of the candidate demand information. This improved the detection efficiency.

[0070] Figure 6A demand switching device according to an embodiment of the present invention is shown. The device includes a processor 901 and a memory 902, which can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0071] The processor 901 may be a central processing unit (CPU). The processor 901 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0072] The memory 902 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above method embodiments. The processor 901 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 902, that is, implementing the methods in the above method embodiments.

[0073] The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 901, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 902 may optionally include a memory remotely arranged relative to the processor 901, and these remote memories may be connected to the processor 901 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] One or more modules are stored in the memory 902 , and when executed by the processor 901 , the method in the above method embodiment is executed.

[0075] The specific details of the above-mentioned demand switching device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned method embodiment, and will not be repeated here.

[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the implemented program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0077] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for training a pre-trained language model, characterized in that: The method comprises: Acquire multiple sets of training samples, wherein the training samples include two pieces of appeal information with similar semantics, and category labels and category matching labels corresponding to the two pieces of appeal information, wherein the category matching labels are used to indicate whether the two pieces of appeal information belong to the same category; Calculate the text vectors of the two appeal messages in the current training sample through the preset pre-trained language model; Based on the text vector, the category and category matching of the appeal information are predicted to obtain the predicted category and the predicted category matching result; the predicting the category and category matching of the appeal information based on the text vector to obtain the predicted category and the predicted category matching result includes: inputting two text vectors into a first fully connected layer respectively, calculating the predicted category of each of the two text vectors, and correcting the parameters of the first fully connected layer by a first error; inputting the two text vectors into a second fully connected layer at the same time, calculating the predicted category matching results of the two text vectors, and correcting the parameters of the second fully connected layer by a second error; Correcting a model parameter of the preset pre-trained language model according to a first error and a second error, wherein the first error is an error between the predicted category and the category label, and the second error is an error between the predicted category matching result and the category matching label; When the pre-trained language model meets a preset condition, the pre-trained language model is output.

2. The method according to claim 1, characterized in that: The obtaining of multiple sets of training samples comprises: Traversing each piece of demand information in the historical demand information database, and combining the demand information in the historical demand information database with a similarity above a preset threshold in pairs based on the BM25 algorithm; The category label and category matching label corresponding to each set of demand information are obtained, and then multiple sets of training samples are obtained.

3. A request transfer method, characterized in that: The method comprises: Receiving target demand information of a target user, and converting the target demand information into a target text vector using a pre-trained language model trained by the method described in any one of claims 1 to 2; Extracting candidate demand information that is semantically similar to the target demand information from a historical demand information database using a first similarity algorithm; Calculate the candidate text vector corresponding to each candidate appeal information by using the pre-trained language model; Calculating the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and finding the candidate text vector having the highest similarity to the target text vector; The target appeal information is transferred to the processing department corresponding to the candidate text vector with the highest similarity.

4. The method according to claim 3, characterized in that The first similarity algorithm is the BM25 algorithm. The step of extracting candidate demand information that is semantically similar to the target demand information from the historical demand information database using the first similarity algorithm includes: Each demand information in the historical demand information database is traversed, and the candidate demand information in the historical demand information database whose similarity with the target demand information is above a preset threshold is selected using the BM25 algorithm.

5. The method according to claim 3, characterized in that: The method further comprises: Calculate the text vector of each demand information in the historical demand information database using the pre-trained language model trained by the method described in any one of claims 1 to 2; The calculated text vector is associated with the corresponding demand information and stored in the historical demand information database.

6. The method according to claim 3, characterized in that The second similarity algorithm is a second fully connected layer.

7. A demand switching device, characterized in that: The device comprises: A demand receiving module, used for receiving target demand information of a target user, and converting the target demand information into a target text vector by a pre-trained language model trained by the method according to any one of claims 1 to 2; A candidate demand module, used to extract candidate demand information with semantic similarity to the target demand information from the historical demand information database using a first similarity algorithm; A vector conversion module, used to calculate the candidate text vector corresponding to each candidate appeal information through the pre-trained language model; A demand matching module, used to calculate the similarity between the target text vector and each candidate text vector using a second similarity algorithm, and find the candidate text vector with the highest similarity to the target text vector; The transfer module is used to transfer the target appeal information to the processing department corresponding to the candidate text vector with the highest similarity.

8. A demand switching device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 3 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 3 to 6.

Images