Government affair hotline work order intelligent allocation method based on large language model

Abstract

The invention provides a government affair hotline work order intelligent allocation method based on a large language model, the allocation party comprises fine adjustment of the large language model and government affair case distribution of the fine adjustment model, and the fine adjustment of the large language model comprises the following steps: arranging case distribution record information and constructing a case distribution knowledge base; constructing fine tuning data sets in batches; cleaning the fine-tuning data set; the large language model is fine-tuned using the fine-tuning dataset. According to the method, the logical reasoning ability of a large language model is utilized, the logic of manual order distribution is imitated, the work order distribution rule followed by an order distributor in the actual distribution process in business is sorted out, intelligent distribution is completed, high flexibility is achieved, under the condition that the work order distribution rule changes slightly, the model does not need to be trained again, and the work order distribution efficiency is improved. Updating of the intelligent distribution module can be completed only by updating the work order distribution rule document; and meanwhile, the method also has interpretability, can provide an original partial wave model, and is easier to understand and optimize the whole module.

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a method for intelligently allocating government hotline work orders based on a large language model. Background Art

[0002] The Big Language Model is a natural language processing technology based on artificial intelligence that can understand and generate natural language. By learning from massive amounts of text data, the Big Language Model can master the grammar, semantics, and contextual relationships of a language, and complete various language tasks on this basis. It has achieved remarkable results in many fields such as text classification, sentiment analysis, and question-answering systems, and can provide intelligent and automated services. In the intelligent classification and distribution of government hotline work orders, the Big Language Model can effectively analyze the content of user complaints, automatically recommend processing departments based on customized distribution rules, and improve work efficiency.

[0003] Existing technologies, such as the Chinese patent document CN115935245A, provide a method for automatically classifying and allocating government hotline cases. The method is based on the Bert pre-training model. The case classification model improves the classification accuracy of similar case categories by focusing on training the differences between similar categories. The department allocation model improves the accuracy of case allocation by integrating administrative division information and focusing on training the differences between the competent departments of similar cases. However, the above technology only uses the Bert-based case classification and allocation model to complete the intelligent classification and allocation of cases. It can only provide case classification and case handling departments, but cannot provide reasons or basis for allocation.

[0004] Most of the existing methods are based on BERT's classification model. This method can directly give the case classification and handling department after inputting the case text content. However, due to the limitation of training data, this method basically has a good effect in the early stage of going online, but as time goes by, the effect will show a downward trend, and it is necessary to update the training data and train the model more frequently. And when there is a change in the allocation rules, such as the same situation in the past, the case was assigned to department A, but department A now does not accept this part of the case and assigns it to department B instead, in this case, adding these data to the training will cause great interference to the model, and there will be conflicts in the data. However, the work of screening and removing these data is time-consuming and labor-intensive, and it is even difficult to find these data conflicts. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for intelligent distribution of government hotline work orders based on a large language model to solve the problems raised in the above-mentioned background technology. The results generated by the present invention have the advantages of explainability, derivability, flexibility and strong adaptability.

[0006] In order to achieve the above object, the present invention is implemented by the following technical scheme: a method for intelligently allocating government hotline work orders based on a large language model, the allocation method comprising fine-tuning the large language model and dispatching government cases of the fine-tuned model, wherein the fine-tuning of the large language model comprises the following steps: S1.1. Organize case dispatch record information and build case dispatch knowledge base; S1.2, batch construct fine-tuning dataset; S1.3, fine-tuning dataset cleaning; S1.4. Fine-tune the large language model using the fine-tuning dataset; The government affairs case dispatching of the fine-tuning model includes the following steps: S2.1. Regularly read unassigned government hotline cases; S2.2, classify cases using BERT classification model; S2.3, through the case classification obtained by the classification model, the classified work order dispatching rules are retrieved from the knowledge base; S2.4. Build a large model input by combining the prompt word template + case information + work order dispatch rule document to obtain the final model input; S2.5. Extract information from the json format output by the model and allocate cases according to the corresponding information.

[0007] Furthermore, in step S1.1, the case content, case classification, case address, and handling department information collected in the case dispatch record are manually sorted out according to the dispatch rules of all categories.

[0008] Furthermore, each category uses "xxx work order dispatching rules.docx" as its name, and the dispatching rules are written into the document to form a knowledge base.

[0009] Furthermore, step S1.1 specifically includes the following process: S1.2.1. Construct a simple program with case content, case classification, and case address as input; S1.2.2. According to the case classification, retrieve the work order dispatch rule document currently used for input; S1.2.3. Based on the document and case content, the case address constructs a large model input; S1.2.4. Input the Qwen2.5-72B large model and save the model output; S1.2.5. Execute S1.2.1 to S1.2.4 in batches to obtain a preliminary data set.

[0010] Furthermore, the large model input includes prompt word templates and samples.

[0011] Furthermore, in step S1.3, data filtering is performed according to the handling department in the case dispatch record, combined with the work order dispatch rules and the saved model output to obtain a cleaned fine-tuning data set.

[0012] Furthermore, the fine-tuning dataset consists of model input and annotation. The model input is composed of prompt word template + work order dispatch rule document + work order data, and the annotation is the json data of dispatch reason and handling department.

[0013] Furthermore, in step S1.4, the model input data is input into the large language model, the loss is calculated through the loss function according to the output content and annotations of the large model, and the loss function is reduced based on the optimizer of the large language model, so that the output of the large language model gradually approaches the expected output content.

[0014] Furthermore, the specific process of step S1.4 is as follows: S1.4.1. Large language models are pre-trained with a large amount of text and are used to generate text, summarize content, and perform translation, rewriting, classification, categorization, and analysis; S1.4.2. The method of fine-tuning the large language model is the QLora fine-tuning method; S1.4.3. Set the number of training steps.

[0015] Furthermore, the large language model is an open source large language model, including ChatGLM3-6b, Qwen2.5 series; the number of training steps is set to 5000.

[0016] Beneficial effects of the present invention: 1. This method of intelligent distribution of government hotline work orders based on a large language model automatically distributes cases based on case information and business information (work order distribution rules, departmental rights and responsibilities list, and other additional accumulated document information), and can provide a certain degree of explainability for the distribution behavior. This improves the flexibility of intelligent distribution. In the case of changes in business information, adaptation can also be completed at zero cost.

[0017] 2. This method of intelligent dispatching of government hotline work orders based on a large language model makes full use of the logical reasoning ability of the large language model, imitates the logic of manual dispatching, sorts out the work order dispatching rules followed by dispatchers in the actual dispatching process, and completes intelligent dispatching.

[0018] 3. The intelligent distribution method of government hotline work orders based on a large language model uses external documents (work order distribution rules) to complete intelligent distribution, which has high flexibility. When there are slight changes in the work order distribution rules, there is no need to retrain the model. Only the work order distribution rule document needs to be updated to complete the update of the intelligent distribution module. At the same time, it is also interpretable and can provide interpretability that the original wave distribution model (BERT-based classification model) cannot provide. On the basis of providing the distribution department, it can provide the reason for distribution to a certain department, which makes it easier to understand and optimize the effect of the overall module. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of fine-tuning a large language model according to the present invention; Figure 2 This is a schematic diagram of a fine-tuning large language model example in an embodiment of the present invention; Figure 3 The present invention is a schematic diagram of an intelligent distribution method for government hotline work orders based on a large language model. DETAILED DESCRIPTION

[0020] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0021] See also Figures 1 to 3 The present invention provides the following technical solution: a method for intelligently allocating government hotline work orders based on a large language model, the method comprising two parts: fine-tuning the large language model and dispatching government cases in combination with the fine-tuned model, wherein the fine-tuning of the large language model specifically comprises the following steps: 1.1 Organize case dispatch record information and build a case dispatch knowledge base Collect case content, case classification, case address, and handling department information in the case dispatch records, manually organize the dispatch rules for all categories, use "xxx Work Order Dispatching Rules.docx" as the name for each category, and write the dispatch rules into the document to form a knowledge base. 1.2 Batch construction of fine-tuning dataset 1.2.1 Construct a simple program with case content, case classification, and case address as input; 1.2.2 According to the case classification, retrieve the work order dispatch rule document currently used for input; 1.2.3 Based on the document and case content, the case address constructs a large model input (prompt word template + sample); 1.2.4 Input the Qwen2.5-72B large model and save the model output; 1.2.5 Execute steps 1-4 in batches to obtain a preliminary data set.

[0022] 1.3 Fine-tuning Dataset Cleaning According to the handling department in the case dispatch record, combined with the work order dispatch rules and the saved model output, the data is filtered to obtain the cleaned fine-tuning data set. (It is enough to filter out about 3k data) The fine-tuning data set consists of model input and annotation. The model input is the prompt word template + work order dispatch rule document + work order data, and the annotation is the json data of the dispatch reason and the handling department. 1.4 Fine-tune the large language model using the fine-tuning dataset The model input data is input into the large language model (Qwen2.5-7B). According to the output content and annotations of the large model, the loss is calculated through the loss function, and the loss function is reduced based on the optimizer that comes with the large language model, so that the output of the large language model gradually approaches the expected output content.

[0023] 1.4.1 Large language models can understand and use language. After a large amount of text pre-training, such language models can generate text, summarize content, and translate, rewrite, classify, categorize and analyze. The large language model described in the present invention refers to an open source large language model, such as ChatGLM3-6b, Qwen2.5 series, etc. The present invention selects Qwen2.5-7B as the fine-tuning model of the present invention; 1.4.2 The method of fine-tuning the large language model in the present invention is the QLora fine-tuning method; 1.4.3 The number of training steps in this embodiment is set to 5000. In this embodiment, an intelligent allocation process is also provided. Figure 3 As shown, the specific process includes the following: 2.1 Regularly read unassigned government hotline cases. 2.2 Use the BERT classification model to classify cases. (Here we use the government hotline case classification model based on patent CN115935245A) 2.3 The case classification is obtained through the classification model, and the classified work order dispatching rules are retrieved from the knowledge base.

[0024] For example: Bus management work order distribution rules.docx Bus management work order dispatch rules Rule 1: Buses starting with 8 are operated by xx company and should be transferred to xx department for processing.

[0025] Rule 2: The core demands are about applying to open bus routes, add or cancel bus stops, and change routes or stops, all of which are handled by the xx Bureau.

[0026] Rule Three: For municipal-managed routes, other issues except applying for bus routes, adding or canceling bus stops, such as few bus departures, long departure times, fare discounts, etc., should be transferred to xx company for processing.

[0027] Municipally managed routes: All bus lines that do not begin with 8 digits, and those that begin with K, D or YX.

[0028] ......(the following content is omitted) 2.4 Build a large model input by combining the prompt word template + case information + work order dispatch rule document to obtain the final model input.

[0029] like: You are an assistant who can assist in dispatching bus management work orders according to document rules.

[0030] # Knowledge Base ## Content from [File](Bus Management Work Order Distribution Rules.docx): ```Bus management work order distribution rules Rule 1: Buses starting with 8 are operated by xx company and should be transferred to xx department for processing.

[0031] Rule 2: The core demands are about applying to open bus routes, add or cancel bus stops, and change routes or stops, all of which are handled by the xx Bureau.

[0032] Rule Three: For municipal-managed routes, other issues except applying for bus routes, adding or canceling bus stops, such as few bus departures, long departure times, fare discounts, etc., should be transferred to xx company for processing.

[0033] Municipally managed routes: All bus lines that do not begin with 8 digits, and those that begin with K, D or YX.

[0034] ......``` Complaint content: Residents of xx community have difficulty traveling. More than 3,000 households can only walk to xx station 1 kilometer away to take the bus. It is strongly recommended to set up a new bus stop at the south gate of xx community to solve the difficulties faced by residents of xx community.

[0035] Complaint address: xx Street, xx Community (for reference only, the address in the case content shall be used first) Provides JSON results in the following format: {{"reason":"Reason for distribution (within 50 words)","dept":"Name of the specific department that distributed the item"}} 2.5 Extract information from the JSON format output by the model and assign cases according to the corresponding information.

[0036] like: {"reason":"This involves adding new sites, which falls within the jurisdiction of xx Bureau, so please transfer it to xx Bureau for processing.","dept":"xx Bureau"} After parsing JSON, it is allocated to the xx bureau and the reason for the allocation is provided.

[0037] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0038] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for intelligently allocating government hotline work orders based on a large language model, characterized in that: The allocation method includes fine-tuning a large language model and allocating government affairs cases of the fine-tuned model, wherein the fine-tuning of the large language model includes the following steps: S1.

1. Organize case dispatch record information and build case dispatch knowledge base; S1.2, batch construct fine-tuning dataset; S1.3, fine-tuning dataset cleaning; S1.

4. Fine-tune the large language model using the fine-tuning dataset; The government affairs case dispatching of the fine-tuning model includes the following steps: S2.

1. Regularly read unassigned government hotline cases; S2.2, classify cases using BERT classification model; S2.3, through the case classification obtained by the classification model, the classified work order dispatching rules are retrieved from the knowledge base; S2.

4. Build a large model input by combining the prompt word template + case information + work order dispatch rule document to obtain the final model input; S2.

5. Extract information from the json format output by the model and allocate cases according to the corresponding information.

2. According to claim 1, a method for intelligently allocating government hotline work orders based on a large language model is characterized by: In step S1.1, the case content, case classification, case address, and handling department information in the case dispatch record are collected and manually sorted out according to the dispatch rules of all categories.

3. According to claim 2, a method for intelligently allocating government hotline work orders based on a large language model is characterized in that: Each category is named "xxx Work Order Distribution Rules.docx", and the distribution rules are written into the document to form a knowledge base.

4. According to the method of intelligent distribution of government hotline work orders based on large language model in claim 1, it is characterized in that: In step S1.1, The process includes: S1.2.

1. Construct a simple program with case content, case classification, and case address as input; S1.2.

2. According to the case classification, retrieve the work order dispatch rule document currently used for input; S1.2.

3. Based on the document and case content, the case address constructs a large model input; S1.2.

4. Input the Qwen2.5-72B large model and save the model output; S1.2.

5. Execute S1.2.1 to S1.2.4 in batches to obtain a preliminary data set.

5. According to claim 4, a method for intelligently allocating government hotline work orders based on a large language model is characterized in that: The large model input includes prompt word templates and samples.

6. According to the method of intelligent distribution of government hotline work orders based on large language model in claim 1, it is characterized in that: In step S1.3, data is filtered according to the handling department in the case dispatch record, combined with the work order dispatch rules and the saved model output to obtain a cleaned fine-tuning data set.

7. According to claim 6, a method for intelligently allocating government hotline work orders based on a large language model is characterized by: The fine-tuning dataset consists of model input and annotation. The model input is composed of prompt word template + work order dispatch rule document + work order data, and the annotation is the json data of dispatch reason and handling department.

8. According to claim 1, a method for intelligently allocating government hotline work orders based on a large language model is characterized in that: In step S1.4, the model input data is input into the large language model, and the loss is calculated through the loss function according to the output content and annotations of the large model. The loss function is reduced based on the optimizer of the large language model, so that the output of the large language model gradually approaches the expected output content.

9. According to the method of intelligent distribution of government hotline work orders based on large language model in claim 1, it is characterized in that: The specific process of step S1.4 is as follows: S1.4.

1. Large language models are pre-trained with a large amount of text and are used to generate text, summarize content, and perform translation, rewriting, classification, categorization, and analysis; S1.4.

2. The method of fine-tuning the large language model is the QLora fine-tuning method; S1.4.

3. Set the number of training steps.

10. The method for intelligently allocating government hotline work orders based on a large language model according to claim 9 is characterized in that: The large language model is an open source large language model, including ChatGLM3-6b, Qwen2.5 series; the number of training steps is set to 5000.

Images