Intelligent hotline text transferring method based on generative model and hierarchical training

Abstract

An intelligent hotline text transferring method based on a generative model and hierarchical training performs text classification on special text data such as 12345 livelihood service hotlines and the like, and mainly comprises three parts of text preprocessing, model training and model reasoning. Data augmentation is performed through same label data pairing based on the generative model, and samples of a small number of labels are diversified, so that the problem of data imbalance is effectively relieved. A hierarchical training model is proposed, and when a model corresponding to a low-hierarchy department is trained, a data set contains department information of a previous hierarchy, so that the model more focuses on the department information of the current hierarchy and the relevance with the department of the previous hierarchy, and the illusion problem caused by an overlong department circulation path is improved. And a model is independently trained for each level, so that intelligent transfer of the hotline text is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing, further to the field of natural language data processing, specifically to a text classification method, and in particular to a method, system, electronic device and readable storage medium for intelligent hotline text transfer based on a generative model and hierarchical training. Background Art

[0002] In the past decade, many smart city projects have emerged across the country to solve urban problems and improve the effectiveness and efficiency of urban governance. These projects provide citizens with a more convenient livelihood service system, allowing them to easily report livelihood issues such as urban services and transportation, and maintain continuous contact with management departments. Taking the most important 12345 hotline service as an example, this service helps to improve public service capabilities in market supervision, social work, ecological and environmental protection, and also helps to implement administrative streamlining and optimize services in related fields.

[0003] The platform provides 24-hour manual service, allowing citizens to submit non-emergency service issues in the city anytime and anywhere via telephone or online platforms. Relevant departments will accurately and quickly handle and feedback the results within the deadline to meet the needs of citizens.

[0004] The 12345 livelihood service hotline itself is special, and its main feature is "handling complaints immediately". Its processing process can be summarized as follows: the platform's operators receive complaints from the public. If it is a consulting question and the answer can be found in the knowledge base, they will directly reply online; if it cannot be solved, a work order text will be generated and stored in the system platform, and the seat staff will dispatch the work order to other departments. This method still has the following problems:

[0005] (1) There is a huge volume of work orders and a shortage of human resources. According to a report on the handling of public appeals in a certain city in the fourth quarter of 2021, a total of 603,100 cases were accepted, with an average of more than 5,000 work orders accepted every day. The operators were under great pressure and the management department needed to invest more manpower.

[0006] (2) The three-fixed plan is not clear enough, resulting in a long work order circulation cycle. In the "handle complaints immediately" business, the manual event transfer cycle is long and inefficient. In addition, the quality of event transfer depends largely on the operator's experience and understanding of the business department, which may lead to errors in event flow.

[0007] (3) Complex business leads to an increase in the number of levels and time required for work order circulation. For some work orders involving villages and communities, due to the wide scope involved, the number of levels may be too long and multiple agents may be required to handle them, which will also increase the difficulty of work order circulation. This situation will affect the efficiency of work order processing.

[0008] At present, hotline business is becoming increasingly heavy, and a large amount of hotline text information needs to be processed in a timely manner. Manual classification and distribution alone cannot meet the processing needs. With the development of computer information processing technology, more and more industries and departments are using automatic information processing to replace the original manual information processing method. Good text information processing results can help achieve refined and accurate management of social functional management departments, which is conducive to reducing the burden of manual processing, improving management efficiency, and enhancing public satisfaction. However, some existing text information processing methods cannot meet the special processing needs of these complex hotline cases.

[0009] Therefore, it has become a technical problem to be solved urgently to conduct targeted research on special text data such as the 12345 people's livelihood service hotline and explore new text classification methods to process this information, so as to accurately classify and process such complex hot issues in a timely and effective manner and realize accurate and refined allocation and management to social functional management departments.

[0010] Public Content

[0011] The technical problem to be solved by the present disclosure is to explore a practical new text classification method for special text data such as the 12345 people's livelihood service hotline, process the text data information, and realize the intelligent transfer of hotline texts.

[0012] In order to solve the technical problems, after comparative studies, the present invention proposes a method for intelligent transfer of the 12345 hotline based on generative models and hierarchical training.

[0013] Generative models are a type of machine learning model whose goal is to learn the distribution of data and be able to generate new samples similar to the training data. This is different from discriminative models, whose goal is to learn and discriminate the decision boundaries between different classes.

[0014] Most of the existing methods for solving the hotline intelligent dispatching task are based on discriminant models. However, using discriminant models to implement 12345 hotline intelligent dispatching will face the following difficulties:

[0015] (1) Data imbalance;

[0016] (2) Ignoring the hierarchical relationships between departments;

[0017] (3) Error transmission problem.

[0018] The core idea of ​​generative models is to learn the data generation process so that the model can generate new samples that are similar to the training data. Generative models are widely used in the context of probability statistics, and they try to model the potential distribution of observed data. However, there are no application examples of generative models in the scenario of solving the hotline intelligent transfer task.

[0019] Large language models are also a type of generative model. Large language models (LLMs) are also called large language models. They are artificial intelligence models designed to understand and generate human language. Typically, LLMs refer to Transformer language models that contain hundreds of billions (or more) parameters, such as GPT-3, PaLM, LLaMA, and GLM. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are characterized by their large size and billions of parameters, which help them learn complex patterns in language data. These models are often based on deep learning architectures such as transformers, which helps them achieve impressive performance on a variety of NLP tasks.

[0020] After a long period of development, LLM has evolved to its current state, a general and capable learner. In this process, many important technologies have been proposed to greatly enhance the capabilities of LLM, including expansion, training, capability guidance, alignment fine-tuning, and tool operation.

[0021] (1) Scaling: Transformer language models have a significant scaling effect. Larger models and more training computations usually lead to improved model capabilities. As two representative models, GPT-3 and PaLM have increased their model sizes to 175 billion and 540 billion, respectively. In addition, since the computing budget is usually limited, scaling rules can be used to allocate computing resources more efficiently.

[0022] (2) Training: Due to the huge model size, successfully training a powerful LLM is very challenging. Distributed training algorithms are required to learn the parameters of the LLM network, where various parallel strategies are usually used in conjunction. To support distributed training, several optimization frameworks have been released to facilitate the implementation and deployment of parallel algorithms, such as DeepSpeed ​​and Megatron-LM. In addition, optimization techniques are also important for training stability and model performance, such as restarts to overcome training loss surges and mixed precision training.

[0023] (3) Capability Guidance: After pre-training on a large-scale corpus, LLM has the potential to be a general task solver. However, these capabilities may not be explicitly demonstrated when LLM performs some specific tasks. As a technical means, designing appropriate task instructions or specific ICL strategies can stimulate these capabilities. In addition, we can also use task descriptions expressed in natural language to fine-tune LLM instructions to improve the generalization ability of LLM on unseen tasks.

[0024] (4) Fine-tuning: Since LLMs are trained to capture the data characteristics of the pre-training corpus (including both high-quality and low-quality data), they may generate toxic, biased, or even harmful content for humans. Therefore, it is necessary to align LLMs with human values, such as usefulness, honesty, and harmlessness. InstructGPT designs an effective fine-tuning method to enable LLMs to operate according to the desired instructions, which utilizes reinforcement learning technology based on human feedback, which incorporates humans into the training loop and adopts a carefully designed annotation strategy.

[0025] (5) Tool Operation: LLMs are essentially trained for text generation based on a massive corpus of plain text, and therefore perform poorly on tasks that are not suitable for expression in text form (e.g., numerical computation). In addition, their capabilities are also limited by the pre-training data, such as the inability to obtain the latest information. To address these issues, a technique has recently been proposed that uses external tools to make up for the shortcomings of LLMs. For example, LLMs can use calculators to perform accurate calculations and search engines to retrieve unknown information.

[0026] Therefore, the present disclosure proposes a method for intelligent hotline text transfer based on generative models and hierarchical training, which uses the large language model GLM developed by Tsinghua University to enhance the data, and then uses the enhanced data and department information of relevant regions to fine-tune the language model. In the training stage, the transfer departments are classified into multiple departments at different levels, and a model is trained separately for each level.

[0027] In order to achieve the above object, according to one aspect of the present disclosure, a hotline text intelligent transfer method based on a generative model and hierarchical training is provided, comprising the following steps:

[0028] The first step is text preprocessing, which includes:

[0029] Step 1.1, obtain a hotline text data set; the data of the hotline text data set comes from the hotline text data recorded by the hotline phone or web page; each hotline text data contains two parts: "text" and "label";

[0030] Step 1.2: Data cleaning: Cleaning the hotline text dataset includes processing redundant characters and removing duplicates from the dataset.

[0031] Step 1.3: Label stratification; labels record the flow path of the department, and a hierarchical structure tree is constructed for all labels;

[0032] Step 1.4, constructing a JSON dataset: First, the dataset is split into several datasets with the same “text” but different “labels” according to the layers of the hierarchical structure tree, and then the datasets are transcoded respectively;

[0033] The second step is model training, which includes:

[0034] Step 2.1: Model pre-training: The GLM pre-trained language model framework independently developed by Tsinghua University is used as the basic model; the model is trained and improved by optimizing an autoregressive blank filling objective;

[0035] Step 2.2, data augmentation: use the pre-trained model to balance the data set;

[0036] Step 2.3: fine-tune the model layer by layer: pass all data sets into the model for fine-tuning;

[0037] The third step is model reasoning, which includes:

[0038] The validation sets are input into the pre-trained model. The model architecture is similar to the Transformer converter and uses the self-attention mechanism. The inference output process is as shown in the formula As shown, Q represents the matrix composed of query vectors, K represents the matrix composed of key vectors, and K T represents the transposed matrix of K, V represents the matrix composed of value vectors, softmax represents the normalized exponential function, d k is the dimension of the K matrix; the self-attention calculation maps a set of key-value pairs consisting of the query vector query, the key vector and the value vector to the output, and the output is calculated as the weighted sum of the value vectors values, where the weight assigned to each value vector value is calculated by the similarity function between the query vector query and the corresponding key vector key; then the feature vector λ processed by the multi-head self-attention is sent to the feedforward layer, which consists of two linear transformations, and the activation function is as follows:

[0039]

[0040] Where λ represents the eigenvector λ and erf represents the error function.

[0041] Preferably,

[0042] The “text” is the complaint or feedback information received by the hotline or webpage, and the “label” is the flow path of the hotline text data recorded by the functional department.

[0043] Preferably,

[0044] The hotline text data set is cleaned using a python program; the redundant characters include spaces, tabs, and line breaks;

[0045] The processing of redundant characters includes extracting and processing the "text" and "label" of each data separately, first determining whether the "label" extracted from each sample is an empty character, if it is an empty character, recording the "text" and repeating the operation for the next sample until the extracted "label" is a normal character, merging the recorded "text" to form a new sample, and finally deleting the tabs, spaces, and line breaks that are not the last in the "text";

[0046] The deduplication processing of the data set includes: 1. If there are two completely identical data, only the sample that appears for the first time is retained; 2. If the content of the "label" of the sample is the same as its "text" content, or the "label" is other garbled characters, these samples are extracted separately for manual annotation or deletion.

[0047] Preferably,

[0048] A python program is used to construct a hierarchical structure tree for all labels. The departments that appear in the bundle are regarded as labels at the same level, and the constructed hierarchical structure tree is traversed in order. If the current node has only one child node, the child node of the node is merged with it.

[0049] Preferably,

[0050] The hierarchical structure tree is a four-layer structure with only one root node. The data set is divided into three data sets with the same "text" but different "labels" only according to the 2nd, 3rd and 4th layers of the hierarchical structure tree, which are named as data set 1, data set 2 and data set 3 respectively, and then the three data sets are transcoded respectively.

[0051] Preferably,

[0052] Use Python program for transcoding. Each sample in the JSON dataset has two keys, "text" and "label", which correspond to the "text" and "label" in the original hotline text dataset.

[0053] Preferably,

[0054] Given an input text x = [x1,...,x n], x is the hotline text data vector, x1 is the first item of the vector, x n is the nth item of the vector, n is the number of items of the input text, from which multiple text segments {s1,...,s m}, where each fragment s i (i=1,…,m), corresponding to a series of consecutive words in x [s i,1 ,...,s i,l ], where i is an integer between 1 and m, m is the number of text segments, s i,1 Indicates the first word, s i,l represents the lth word; each segment is replaced by a single [MASK] symbol, forming a corrupted text x corrupt ; The model predicts missing words from the damaged text in an autoregressive manner. When predicting missing words in a segment, the model accesses the damaged text and the previously predicted segments. In order to fully capture the interdependencies between different segments, the order of the segments is randomly shuffled. Corresponding to the number of previous segments, let Z m is an index sequence of length m, let s z<i Represents the fragment corresponding to the index sequence whose index length is less than i, where The first item, is the i-1th item; the pre-training objective function is expressed as:

[0055]

[0056] in, It means to find the maximum value of expectation, θ represents the variable that needs to be traversed to obtain the optimal value when taking the maximum value, where z~Z m Indicates that Z m Index sequence as expected expected variable z,p θ Indicates the generated fragment s i The probability of Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and its length is less than the number j;

[0057] Generate the words in each blank in order from left to right, that is, generate fragment s i The probability can be decomposed into:

[0058]

[0059] in, Indicates the existence of x corruptWhen generating fragments i The probability of Indicates the existence of x corrupt When generating fragments i The words in i,j The product of the probabilities of i Representative fragments i Length, Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and should be less than the number j, s i,<j Represents fragments i The number of consecutive words in is also limited, and its number is less than the number j.

[0060] Preferably, the input text x is divided into two parts, respectively denoted as Part A and Part B, where Part A is the damaged text x corrupt , Part B is the covered segment; the words in Part A can see each other, but cannot see any words in Part B; the words in Part B can see the preceding words in Part A and Part B, but cannot see the following words in Part B; each segment is filled with special symbols start mark [START] and end mark , which are used for input and output respectively.

[0061] Preferably, the method further comprises the following steps:

[0062] (1) The input text x = [x1, x2, x3, x4, x5, x6], and the input text is randomly and continuously [MASK] masked. In the figure, the [MASK] mask masks the third input word [x3] and the fifth and sixth input words [x5, x6];

[0063] (2) Replace the third input word [x3] and the fifth and sixth input words [x5, x6] with the [M] flag. In order to capture the intrinsic connection between spans, the order of spans is randomly swapped.

[0064] (3) Part A is input into the model as Part B, and the model generates Part B by autoregression. The [S] sign is inserted at the head of each fragment during input, and the [E] sign is inserted at the tail during output. The self-attention mask used by the model during generation is different from the traditional Transformer converter. Part of it is masked, and the words in Part A can obtain all their own information, but cannot obtain the information of Part B. The words in Part B can see the information of Part A and the words before this word in the sequence. In addition, during input, Part A will also insert two position codes, which are recorded as Position1 and Position2 respectively. Position1 and Position2 are the two-dimensional codes of the input. Position1 records the relative position of the fragment in Part B in Part A, and Position2 records the relative position of each fragment in Part B.

[0065] Preferably, the balancing process includes: first constructing a dedicated data set required for data augmentation, extracting all texts corresponding to each label in data set 3, i.e., the last layer of data sets, pairing them in pairs to form a new data set, recorded as a text data set, wherein the two key values ​​of the text data set are recorded as "text1" and "text2", which correspond to two successfully paired texts respectively, and only one new data is generated for each pair of texts; wherein, when some labels are too scarce or cannot be paired with the corresponding texts, the parent label of the corresponding text is searched in data set 2, i.e., the upper layer of data sets, and paired with its parent label; after each label is paired, a new data set is generated.

[0066] Preferably, the fine-tuning method adopts p-tuning v2 prompt fine-tuning, and the p-tuning v2 prompt fine-tuning includes:

[0067] For the input text [x1,…,x n ], add the start mark [CLS] at the beginning of the sentence as the intermediate input text Next, freeze the model parameters and Add the prompt embedding hint vector to get the original input text X of the model:

[0068]

[0069] prompt embedding=[h0,...,h i ]

[0070] X=[h0,...,h i ][CLS][x1,…,x n ]

[0071] Among them, [CLS] indicates the start mark, Represents the input text [x1,…,x n ] plus the start mark [CLS] to form the middle input text, [h0,...,h i ] represents multiple word vectors in the prompt embedding prompt vector, and X represents the original input text of the model;

[0072] During fine-tuning, the model only adjusts and optimizes the parameters in the prompt embedding prompt vector. Each output Y passes through a linear layer and optimizer to adjust the parameters in the prompt embedding prompt vector.

[0073] Y=GLM(X)

[0074] prompt embedding=optimization(a·Y+b)

[0075] Where Y represents the vector output by the model's original input text X after the pre-trained language model GLM, optimization(a·Y+b) represents the linear optimizer in p-tuning v2 prompt fine-tuning, and a and b represent the optimization parameters of the linear optimizer.

[0076] According to another aspect of the present disclosure, a hotline text intelligent dispatching system based on a generative model and hierarchical training is provided, comprising:

[0077] Text preprocessing unit, specifically including:

[0078] Acquire a hotline text data set unit; the data of the hotline text data set comes from the hotline text data recorded by the hotline phone or web page; each hotline text data includes two parts: "text" and "label";

[0079] Data cleaning unit: Cleaning the hotline text data set includes processing redundant characters and deduplication of the data set;

[0080] Label hierarchical unit; labels record the flow path of the department and construct a hierarchical structure tree for all labels;

[0081] Constructing a JSON dataset unit: First, the dataset is split into several datasets with the same “text” but different “labels” according to the layers of the hierarchical structure tree, and then the datasets are transcoded respectively;

[0082] Model training unit, including:

[0083] Model pre-training unit: Based on the GLM pre-training language model framework independently developed by Tsinghua University; train and improve the model by optimizing an autoregressive blank filling objective;

[0084] Data augmentation unit: Use the pre-trained model to balance the data set;

[0085] Model hierarchical fine-tuning unit: all data sets are passed into the model for fine-tuning;

[0086] Model reasoning unit, specifically including:

[0087] The validation sets are input into the pre-trained model. The model architecture is similar to the Transformer converter and uses the self-attention mechanism. The inference output process is as shown in the formula As shown, Q represents the matrix composed of query vectors, K represents the matrix composed of key vectors, and K T represents the transposed matrix of K, V represents the matrix composed of value vectors, softmax represents the normalized exponential function, d k is the dimension of the K matrix; the self-attention calculation maps a set of key-value pairs consisting of the query vector query, the key vector and the value vector to the output, and the output is calculated as the weighted sum of the value vectors values, where the weight assigned to each value vector value is calculated by the similarity function between the query vector query and the corresponding key vector key; then the feature vector λ processed by the multi-head self-attention is sent to the feedforward layer, which consists of two linear transformations, and the activation function is as follows:

[0088]

[0089] Where λ represents the eigenvector λ and erf represents the error function.

[0090] According to another aspect of the present disclosure, the present disclosure provides an electronic device, including:

[0091] a memory storing execution instructions; and

[0092] A processor executes the execution instructions stored in the memory, so that the processor executes the above method.

[0093] According to another aspect of the present disclosure, the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement the above method when executed by a processor.

[0094] Beneficial effects of the present disclosure:

[0095] Compared with the prior art, the present invention has the following outstanding substantive features and significant improvements:

[0096] (1) In the past, most of the methods used to solve the task of intelligent hotline transfer were based on discriminative models. Therefore, solving the problem of intelligent hotline transfer based on generative models is a new attempt in the context of community governance issues.

[0097] (2) To address the data imbalance problem in the dataset, this method performs data augmentation by pairing data with the same label based on a generative model, diversifying samples with a small number of labels, thereby effectively alleviating the problem of data imbalance.

[0098] (3) Since the emergence of Transformer, the difficulty of text classification has been greatly reduced, but the traditional Transformer-based discriminative model can only focus on the individual information of each sample, and cannot focus on the connection between each sample, such as the similarity of context. Therefore, this method uses a generative model to model the data distribution, which can better understand the sample content in the data set and the context information of each sample. Therefore, when classifying, it can take into account the global information in the text, and can better identify the key features and patterns in the text, thereby improving the accuracy of classification.

[0099] (4) In traditional text classification methods based on generative models, "hallucination problems" often occur, that is, the departments generated by the model do not exist, or the generated departments have no correlation with the upper-level departments. Therefore, this method proposes a hierarchical training model, and when training the model corresponding to the lower-level department, the data set will include the department information of the previous level, so that the model can focus more on the department information of the current level and the correlation with the department of the previous level, improving the hallucination problem caused by the long department circulation path. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings illustrate exemplary embodiments of the present disclosure and are used together with the description thereof to explain the principles of the present disclosure, wherein these drawings are included to provide a further understanding of the present disclosure, and the drawings are included in and constitute a part of this specification. By describing the embodiments of the present disclosure in detail in conjunction with the accompanying drawings, the above and other purposes, features, and advantages of the present disclosure will become more apparent.

[0101] Figure 1 It is the process method for intelligent transfer of 12345 hotline.

[0102] Figure 2 is a flow chart of the present disclosure.

[0103] Figure 3 It is a model architecture diagram of the present disclosure.

[0104] Figure 4 It is a label hierarchy tree of the present disclosure.

[0105] Figure 5 It is the pre-training module of the disclosed model and the self-attention matrix diagram.

[0106] Figure 6 This is the architecture diagram of the P-Tuning V2 fine-tuning model disclosed in this disclosure. DETAILED DESCRIPTION

[0107] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0108] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0109] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0110] The terms used in the present disclosure are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups are not excluded from existence or addition. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0111] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0113] One of the purposes of the present disclosure is to provide a method for intelligently transferring texts of 12345 hotline based on generative models and hierarchical training. Taking the texts of 12345 hotline as an example, the method for intelligently transferring texts of 12345 hotline is as follows: Figure 1 As shown. First, the public or enterprises call the 12345 hotline to explain the problem. The problem text will be put into the transfer rule pool. If the undertaking unit corresponding to the problem text is clear enough and meets the transfer rules, it will be directly transferred; if the problem text is not clear enough, there are two situations. According to the rules, only transfer prompts are allowed and automatic transfer is not restricted by the rules. The texts corresponding to these two situations will be intelligently transferred. The texts that allow transfer prompts will be prompted through the defined rules. The texts that are not restricted by the rules will be directly intelligently transferred. If the text meets the automatic transfer standards, it will be automatically transferred. If not, it will be transferred to the seat for manual transfer. The specific flow chart of the present disclosure is as follows. Figure 2As shown, first there is the data preprocessing stage, where the data is cleaned, including deduplication, removal of extra spaces and line breaks, and finally converted into JSON format data; secondly, it enters the model pre-training stage, where a Chinese-English bilingual dataset with about 1T of identifiers is loaded for autoregressive blank filling; after pre-training, data augmentation is first performed, where data with the same label in the original dataset is paired to form a new dataset, and this dataset is used for fine-tuning training, and the trained model is used for data augmentation; then it enters the model layering and fine-tuning stage, where the augmented dataset is layered according to the label level, and the layered JSON dataset is used for fine-tuning respectively; finally, it enters the model prediction stage, where samples are input into different models to obtain results corresponding to different levels, and then spliced ​​to obtain the final label, which is the flow path of the event. The model architecture diagram of the present disclosure is shown in FIG. Figure 3 As shown in the figure, the model architecture is mainly divided into three parts: the pre-training module of the GLM pre-trained language model, the data augmentation module, and the hierarchical fine-tuning training module. The pre-training module of the GLM pre-trained language model mainly performs the pre-training of the GLM pre-trained language model and initially builds a model with better results. The data augmentation module mainly processes the unbalanced data in the data set and expands a small amount of data. The hierarchical fine-tuning training module mainly trains multiple models in layers to achieve better results. The specific steps are as follows:

[0114] The first step is to prepare the dataset

[0115] The data of this method mainly comes from the hotline text data recorded by the "12345" citizen hotline in a certain district of a city, mainly from November 1, 2019 to June 20, 2023, the hotline text data stored under the new working mechanism of "handling complaints immediately" in a certain district of a city, totaling 48,556. Each hotline text data contains two parts: "text" and "label". The text is the information such as complaints and feedback received by the citizen hotline in a certain district of a city, and the label is generally the flow path recorded by the functional department.

[0116] The second step is data preprocessing

[0117] Step 2.1, data cleaning. The present invention cleans the hotline text data set, and a python program can be used to clean the hotline text data set. First, redundant characters are processed, including spaces, tabs, and line breaks. Redundant characters basically exist in the "text" of each hotline text data sample, so the "text" and "label" of each data are extracted and processed separately. Considering that there is a line break in the middle of the "text" in the sample, the "text" is divided into two lines. Therefore, first determine whether the "label" extracted from each sample is an empty character. If it is an empty character, record the "text" and repeat the operation for the next sample until the extracted "label" is a normal character, merge the recorded "text" into a new sample, and finally delete the tabs, spaces, and non-last line breaks in the "text".

[0118] The next step is to deduplicate the data set. The duplicate data in the data set can be divided into two categories. One is that there are two identical data. In this case, only the first sample is retained. The second is that the content of the sample's "label" is the same as its "text" content, or the "label" is other garbled code. In this case, these samples are extracted separately for manual labeling. If manual labeling is not possible, they are deleted.

[0119] Step 2.2: Label stratification. Labels record the flow paths of departments. The process of label stratification is the process of constructing a hierarchical structure tree of flow departments. Taking the above hotline text data as an example, after statistics, there are 622 different flow paths in the data set. Due to the complexity of the flow paths and the diversity of department combinations, simple word segmentation cannot be used. This disclosure first constructs a hierarchical structure tree for all labels, such as Figure 4 As shown. A Python program can be used to construct a hierarchical tree for all tags. However, the initial hierarchical tree is too deep, such as Figure 4 As shown in the figure above. According to observation statistics, some departments appear in bundles, so when layering, these bundled departments are regarded as labels of the same level, and the constructed hierarchical structure tree is processed and traversed in order. If the current node has only one child node, then the child node of the node is merged with it. After processing, the depth of the hierarchical structure tree is 4, as shown in Figure 4 As shown in the figure below.

[0120] Step 2.3, build the JSON dataset. First, split the dataset. Since there is only one root node in the hierarchical structure tree, the dataset is divided into three datasets with the same "text" but different "labels" according to the 2nd, 3rd, and 4th layers of the hierarchical structure tree, named dataset 1, dataset 2, and dataset 3, respectively. Then transcode the three datasets separately. You can use the python program for transcoding. Each sample in the JSON dataset has two keys, namely the "text" key and the "label" key, which correspond to the "text" and "label" in the original hotline text dataset.

[0121] Step 3: Model training and reasoning

[0122] This model is implemented based on the GLM pre-trained language model framework independently developed by Tsinghua University. The GLM pre-trained language model uses a single Transformer converter model architecture and makes some modifications to the Transformer converter model architecture. The GLM pre-trained language model rearranges the order of layer normalization and residual connection, and uses a single linear layer to predict output words, and replaces the ReLU linear rectification activation function with the GeLUs Gaussian error linear unit activation function. The present disclosure has made optimizations and innovations on this basis, and improved the hallucination problem generated by the model during prediction without losing prediction accuracy. The following is a detailed introduction to the model training process.

[0123] Step 3.1: Model pre-training. This model is trained by optimizing an autoregressive blank filling objective. Given an input text x = [x1,...,x n ], where x is the hotline text data vector, x1 is the first item of the vector, and x n is the nth item of the vector, n is the number of items of the input text, from which multiple text segments {s1,...,s m}, where each fragment s i (i=1,…,m), corresponding to a series of consecutive words in x [s i,1 ,...,s i,l ], where i is an integer between 1 and m, m is the number of text segments, s i,1 Indicates the first word, s i,l represents the lth word. Each fragment is replaced by a single [MASK] symbol to form a corrupted text x corruptThe model predicts missing words from the corrupted text in an autoregressive manner, which means that when predicting missing words in a segment, the model has access to the corrupted text and previously predicted segments. In order to fully capture the interdependencies between different segments, we randomly shuffle the order of the segments, similar to the permutation language model. Let Z correspond to the number of previous segments. m is an index sequence of length m, let s z<i Represents the fragment corresponding to the index sequence whose index length is less than i, where The first item, is the i-1th item. The pre-training objective function can be expressed as:

[0124]

[0125] in, It means to find the maximum value of expectation, θ represents the variable that needs to be traversed to obtain the optimal value when taking the maximum value, where z~Z m Indicates that Z m The index sequence is the expected variable z, m is the length of the index sequence, p is the expected variable z, θ Indicates the generated fragment s i The probability of Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and should be less than the number j.

[0126] Generate the words in each blank in order from left to right, that is, generate fragment s i The probability can be decomposed into:

[0127]

[0128] in, Indicates the existence of x corrupt When generating fragments i The probability of Indicates the existence of x corrupt When generating fragments i The words in i,j The product of the probabilities of i Representative fragments i Length, Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and should be less than the number j, si, < j means fragment s i The number of consecutive words in is also limited, and should be less than the number j.

[0129] Specifically, the input text x is divided into two parts, denoted as Part A and Part B, where Part A is the damaged text x. corrupt , Part B is the masked segment. Words in Part A can see each other, but cannot see any words in Part B. Words in Part B can see the preposition words in Part A and Part B, but cannot see the subsequent words in Part B. To achieve autoregressive generation, each segment is padded with special symbols start marker [START] and end marker for input and output respectively. In this way, the model automatically learns a bidirectional encoder (for Part A) and a unidirectional decoder (for Part B) in a unified model.

[0130] The details of the autoregressive gap filling task are as follows Figure 5 As shown:

[0131] (1) Taking the input text x=[x1, x2, x3, x4, x5, x6] as an example, the input text is randomly and continuously masked [MASK]. In the figure, the [MASK] mask blocks the third input word [x3] and the fifth and sixth input words [x5, x6].

[0132] (2) Replace the third input word [x3] and the fifth and sixth input words [x5, x6] with the [M] flag. In order to capture the intrinsic connection between spans, the order of spans is randomly swapped.

[0133] (3) Part A is input into the model as Part B, and the model generates Part B by autoregression. The [S] flag is inserted at the head of each fragment during input, and the [E] flag is inserted at the tail during output. The self-attention mask used by the model during generation is different from the traditional Transformer converter. A part of it is masked, and the words in Part A can obtain all of their own information, but cannot obtain the information of Part B. The words in Part B can see the information before Part A and the words in the sequence. In addition, during input, Part A will also insert two position codes, recorded as Position1 and Position2, where Position1 and Position2 are the two-dimensional codes of the input. Position1 records the relative position of the fragment in Part B in Part A, and Position2 records the relative position of each fragment in Part B.

[0134] Step 3.2, data augmentation. Since the data imbalance in the dataset is serious, that is, the data corresponding to some labels are too sparse, and the input to the model training is prone to overfitting, it is extremely critical to use the pre-trained model to balance the dataset. The first step is to build a special dataset required for data augmentation. The specific operation is to extract all the texts corresponding to each label in dataset 3, pair them up in pairs, and form a new dataset, which is recorded as a text dataset. The two key values ​​of the text dataset are recorded as "text1" and "text2", which correspond to the two successfully paired texts respectively. Each pair of texts only generates one new data. Among them, some labels are too sparse with the corresponding texts, and even cannot be paired. For such texts, we will look for the parent label of the corresponding text in dataset 2 and use its parent label for pairing. After pairing each label, a new dataset is generated.

[0135] The text dataset is passed into the pre-trained model for fine-tuning, and the parameters are set to PRE_SEQ_LEN = 256, LR = 2e-2, max_source_length = 512, and max_target_length = 512. PRE_SEQ_LEN and LR are the prompt template length and the learning rate of training, respectively. Max_source_length and max_target_length are the maximum input length and the maximum output length of the input text. The length of the "text" in the statistical dataset is about 120-220, so it is more appropriate to choose an input and output length of 256.

[0136] In order to balance the data, only the data with the number of corresponding data of the label below 300 are selected for augmentation. The selected data are passed into the fine-tuned model for data augmentation. After augmentation, the new data are cleaned and the new data are paired with labels and inserted into the three datasets respectively.

[0137] Step 3.3: fine-tune the model layer by layer. The three data sets are respectively passed into the model for fine-tuning. The fine-tuning method adopts p-tuning v2 prompt fine-tuning. The process and model architecture of p-tuning v2 prompt fine-tuning are as follows: Figure 6 As shown, for the input text [x1,…,x n ], add the start mark [CLS] at the beginning of the sentence as the intermediate input text Next, freeze the model parameters and Add the prompt embedding hint vector to get the original input text X of the model:

[0138]

[0139] prompt embedding=[h0,...,h i ]

[0140] X=[h0,...,h i ][CLS][x1,…,x n ]

[0141] Among them, [CLS] indicates the start mark, Represents the input text [x1,…,x n ] plus the start mark [CLS] to form the middle input text, [h0,...,h i ] represents multiple word vectors in the prompt embedding prompt vector, and X represents the original input text of the model.

[0142] During fine-tuning, the model only adjusts and optimizes the parameters in the prompt embedding prompt vector. Each output Y passes through a linear layer and optimizer to adjust the parameters in the prompt embedding prompt vector.

[0143] Y=GLM(X)

[0144] prompt embedding=optimization(a·Y+b)

[0145] Where Y represents the vector output by the model's original input text X after the pre-trained language model GLM, optimization(a·Y+b) represents the linear optimizer in p-tuning v2 prompt fine-tuning, and a and b represent the optimization parameters of the linear optimizer.

[0146] Among them, the fine-tuning parameters are set to the fine-tuning model corresponding to dataset 1, PRE_SEQ_LEN=64, LR=2e-2, max_source_length=256, max_target_length=64, PRE_SEQ_LEN and LR are the prompt template length and the learning rate of training respectively, and max_source_length and max_target_length are the maximum input length and the maximum output length of the input text. According to statistics, the label length in dataset 1 is relatively short, between 8-56, so the output length is set to 64.

[0147] Dataset 2 corresponds to the fine-tuning model, PRE_SEQ_LEN = 128, LR = 2e-2, max_source_length = 256, max_target_length = 128. Dataset 3 corresponds to the fine-tuning model, PRE_SEQ_LEN = 128, LR = 2e-2, max_source_length = 256, max_target_length = 128. PRE_SEQ_LEN and LR are the prompt template length and the learning rate of training, respectively. Max_source_length and max_target_length are the maximum input length and the maximum output length of the input text. The label lengths in Datasets 2 and 3 are relatively long, ranging from 25 to 89, so it is more appropriate to choose 128 for the output length.

[0148] Regarding the selection strategy of PRE_SEQ_LEN, PRE_SEQ_LEN indicates the length of the prompt template during fine-tuning, which represents the number of parameters that can be adjusted and optimized in model layer fine-tuning. However, tests have shown that the larger the parameter, the better. For example, in dataset 1, the data is well balanced, and using 128 parameters reduces the fine-tuning effect, and the computational cost also increases.

[0149] After fine-tuning, pre-trained models corresponding to the data sets are obtained; specifically, in one embodiment, after fine-tuning, three pre-trained models corresponding to data set 1, data set 2, and data set 3 are obtained, which are respectively called model 1, model 2, and model 3.

[0150] Step 3.4: Model inference. The validation set is input into the pre-trained model. The model architecture is similar to the Transformer converter and uses the self-attention mechanism. The inference output process is as follows: As shown, Q represents the matrix composed of query vectors, K represents the matrix composed of key vectors, and K T represents the transposed matrix of K, V represents the matrix composed of value vectors, softmax represents the normalized exponential function, d k is the dimension of the K matrix; the self-attention calculation maps a set of key-value pairs consisting of the query vector query, the key vector and the value vector to the output, and the output is calculated as the weighted sum of the value vectors values, where the weight assigned to each value vector value is calculated by the similarity function between the query vector query and the corresponding key vector key; then the feature vector λ processed by the multi-head self-attention is sent to the feedforward layer, which consists of two linear transformations, and the activation function is as follows:

[0151]

[0152] Where λ represents the eigenvector λ and erf represents the error function.

[0153] The labels output by the three models are concatenated to obtain the final label and verified. The evaluation indicators are the recall-oriented evaluation indicator ROUGE-N and the bilingual replacement evaluation indicator BLEU. In order to compare with the discriminant model, this disclosure also introduces three indicators: classification accuracy Acc, macro-average F1 score Macro-F1, and weighted F1 score Weighted-F1.

[0154] Table 1 Comparison with advanced methods in discriminant models

[0155]

[0156] Table 2 Comparison with other advanced Chinese language model methods

[0157]

[0158] Any process or method description in the flowchart of the present disclosure or described in other ways herein can be understood as a module, fragment or part of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the technical field to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0159] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0160] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, a device on the Internet, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or, if necessary, processing in another suitable manner, and then stored in a memory.

[0161] It should be understood that the various parts of the present disclosure can be implemented with hardware, software or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented with software stored in a memory and executed by a suitable instruction execution system. For example, if implemented with hardware, as in another embodiment, it can be implemented with any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] A person of ordinary skill in the art of the present disclosure may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0163] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0164] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above method.

[0165] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above method.

[0166] In summary, the method disclosed in the present invention has the following advantages:

[0167] 1. A new scenario application based on generative model is implemented to solve the problem of intelligent transfer of 12345 hotline.

[0168] 2. To address the data imbalance problem in the data set, this method performs data augmentation by pairing data with the same label based on a generative model, diversifying samples with a small number of labels, thereby effectively alleviating the problem of data imbalance.

[0169] 3. Since the emergence of Transformer, the difficulty of text classification has been greatly reduced, but the traditional Transformer-based discriminant model can only focus on the individual information of each sample, and cannot focus on the connection between each sample, such as the similarity of context. Therefore, this method uses a generative model to model the data distribution, which can better understand the sample content in the data set and the context information of each sample. Therefore, when classifying, it can take into account the global information in the text, and can better identify the key features and patterns in the text, thereby improving the accuracy of classification.

[0170] 4. In traditional text classification methods based on generative models, "hallucination problems" often occur, that is, the departments generated by the model do not exist, or the generated departments have no correlation with the upper-level departments. Therefore, this method proposes a hierarchical training model, and when training the model corresponding to the lower-level department, the data set will include the department information of the previous level, so that the model can focus more on the department information of the current level and the correlation with the department of the previous level, improving the hallucination problem caused by the long department circulation path.

[0171] It can be seen that the present disclosure has achieved good results in actual use, and can realize timely, effective and accurate allocation of hot issues involved in the hotline case text data information to the corresponding social function management departments, and realize accurate and refined allocation of these special text data to social function management departments.

[0172] So far, the technical solutions of the present disclosure have been described in conjunction with the preferred implementation methods shown in the accompanying drawings. However, those skilled in the art should understand that the above implementations are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. The protection scope of the present disclosure is obviously not limited to these specific implementations. Without departing from the principles of the present disclosure, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions are still within the scope of the present disclosure.

Claims

1. A hotline text intelligent transfer method based on generative model and hierarchical training, It is characterized in that The steps include: The first step is text preprocessing, which includes: Step 1.1, obtain a hotline text data set; the data of the hotline text data set comes from the hotline text data recorded by the hotline phone or web page; each hotline text data contains two parts: "text" and "label"; Step 1.2: Data cleaning: Cleaning the hotline text dataset includes processing redundant characters and removing duplicates from the dataset. Step 1.3: Label stratification; labels record the flow path of the department, and a hierarchical structure tree is constructed for all labels; Step 1.4, constructing a JSON dataset: First, the dataset is split into several datasets with the same "text" but different "labels" according to the layers of the hierarchical structure tree, and then the datasets are transcoded respectively; The second step is model training, which includes: Step 2.1: Model pre-training: The GLM pre-trained language model framework independently developed by Tsinghua University is used as the basic model; the model is trained and improved by optimizing an autoregressive blank filling objective; Step 2.2, data augmentation: use the pre-trained model to balance the data set; Step 2.3: fine-tune the model layer by layer: pass all data sets into the model for fine-tuning; The third step is model reasoning, which includes: The validation sets are input into the pre-trained model. The model architecture is similar to the Transformer converter and uses the self-attention mechanism. The inference output process is as shown in the formula As shown, Q represents the matrix composed of query vectors, K represents the matrix composed of key vectors, and K T represents the transposed matrix of K, V represents the matrix composed of value vectors, softmax represents the normalized exponential function, d k is the dimension of the K matrix; the self-attention calculation maps a set of key-value pairs consisting of the query vector query, the key vector and the value vector to the output, and the output is calculated as the weighted sum of the value vectors values, where the weight assigned to each value vector value is calculated by the similarity function between the query vector query and the corresponding key vector key; then the feature vector λ processed by the multi-head self-attention is sent to the feedforward layer, which consists of two linear transformations, and the activation function is as follows: Where λ represents the eigenvector λ and erf represents the error function.

2. According to claim 1, a hotline text intelligent transfer method based on generative model and hierarchical training, It is characterized in that The "text" is the complaint or feedback information received by the hotline or webpage, and the "tag" is the flow path of the hotline text data recorded by the functional department.

3. A hotline text intelligent transfer method based on a generative model and hierarchical training according to claim 1 or 2, It is characterized in that The hotline text data set is cleaned using a python program; the redundant characters include spaces, tabs, and line breaks; The processing of redundant characters includes extracting and processing the "text" and "label" of each data separately, first determining whether the "label" extracted from each sample is an empty character, if it is an empty character, recording the "text" and repeating the operation for the next sample until the extracted "label" is a normal character, merging the recorded "text" to form a new sample, and finally deleting the tab character, space, and non-last line break character in the "text"; The deduplication processing of the data set includes:

1. If there are two completely identical data, only the sample that appears for the first time is retained; 2. If the content of the "label" of the sample is the same as its "text" content, or the "label" is other garbled characters, these samples are extracted separately for manual annotation or deletion.

4. A hotline text intelligent transfer method based on a generative model and hierarchical training according to claim 1 or 2, It is characterized in that A python program is used to construct a hierarchical structure tree for all labels. The departments that appear in the bundle are regarded as labels at the same level, and the constructed hierarchical structure tree is traversed in order. If the current node has only one child node, the child node of the node is merged with it.

5. A hotline text intelligent transfer method based on a generative model and hierarchical training according to claim 1 or 2, It is characterized in that The hierarchical structure tree is a four-layer structure with only one root node. The data set is divided into three data sets with the same "text" but different "labels" only according to the 2nd, 3rd and 4th layers of the hierarchical structure tree, and are named as data set 1, data set 2 and data set 3 respectively, and then the three data sets are transcoded respectively.

6. A hotline text intelligent transfer method based on a generative model and hierarchical training according to claim 1 or 2, It is characterized in that Use Python program for transcoding. Each sample in the JSON dataset has two keys, "text" and "label", which correspond to the "text" and "label" in the original hotline text dataset.

7. A hotline text intelligent transfer method based on a generative model and hierarchical training according to claim 1 or 2, It is characterized in that Given an input text x = [x 1 ,...,x n ], x is the hotline text data vector, x 1 is the first term of the vector, x n is the nth item of the vector, n is the number of items of the input text, from which multiple text fragments {s 1 ,...,s m }, where each fragment s i (i=1,…,m), corresponding to a series of consecutive words in x [s i,1 ,...,s i,l ], where i is an integer between 1 and m, m is the number of text segments, s i,1 Indicates the first word, s i,l represents the lth word; each segment is replaced by a single [MASK] symbol, forming a corrupted text x corrupt ; The model predicts missing words from the damaged text in an autoregressive manner. When predicting missing words in a segment, the model accesses the damaged text and the previously predicted segments. In order to fully capture the interdependencies between different segments, the order of the segments is randomly shuffled. Corresponding to the number of previous segments, let Z m is an index sequence of length m, let s z<i Represents the fragment corresponding to the index sequence whose index length is less than i, where The first item, is the i-1th item; the pre-training objective function is expressed as: in, It means to find the maximum value of expectation, θ represents the variable that needs to be traversed to obtain the optimal value when taking the maximum value, where z~Z m Indicates that Z m Index sequence as expected expected variable z,p θ Indicates the generated fragment s i The probability of Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and its length is less than the number j; Generate the words in each blank in order from left to right, that is, generate fragment s i The probability can be decomposed into: in, Indicates the existence of x corrupt When generating fragments i The probability of Indicates the existence of x corrupt When generating fragments i The words in i,j The product of the probabilities of i Representative fragments i Length, Represents an index sequence z with index length i i The corresponding fragment, x corrupt Indicates that there is a corrupted text with a [MASK] symbol mask in the fragment, Indicates that the length of the index sequence is limited and should be less than the number j, s i,<j Represents fragments i The number of consecutive words in is also limited, and its number is less than the number j.

8. A hotline text intelligent transfer system based on generative model and hierarchical training, It is characterized in that include: Text preprocessing unit, specifically including: Acquire a hotline text data set unit; the data of the hotline text data set comes from the hotline text data recorded by the hotline phone or web page; each of the hotline text data includes two parts: "text" and "label"; Data cleaning unit: Cleaning the hotline text data set includes processing redundant characters and deduplication of the data set; Label hierarchical unit; labels record the flow path of the department and construct a hierarchical structure tree for all labels; Constructing a JSON dataset unit: First, split the dataset into several datasets with the same "text" but different "labels" according to the layers of the hierarchical structure tree, and then transcode the datasets respectively; Model training unit, including: Model pre-training unit: Based on the GLM pre-training language model framework independently developed by Tsinghua University; train and improve the model by optimizing an autoregressive blank filling objective; Data augmentation unit: Use the pre-trained model to balance the data set; Model hierarchical fine-tuning unit: all data sets are passed into the model for fine-tuning; Model reasoning unit, specifically including: The validation sets are input into the pre-trained model. The model architecture is similar to the Transformer converter and uses the self-attention mechanism. The inference output process is as shown in the formula As shown, Q represents the matrix composed of query vectors, K represents the matrix composed of key vectors, and K T represents the transposed matrix of K, V represents the matrix composed of value vectors, softmax represents the normalized exponential function, d k is the dimension of the K matrix; the self-attention calculation maps a set of key-value pairs consisting of the query vector query, the key vector and the value vector to the output, and the output is calculated as the weighted sum of the value vectors values, where the weight assigned to each value vector value is calculated by the similarity function between the query vector query and the corresponding key vector key; then the feature vector λ processed by the multi-head self-attention is sent to the feedforward layer, which consists of two linear transformations, and the activation function is as follows: Where λ represents the eigenvector λ and erf represents the error function.

9. An electronic device, It is characterized in that include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory so that the processor executes the method according to any one of claims 1 to 7.

10. A readable storage medium, It is characterized in that The readable storage medium stores execution instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images