Work order automatic generation method and system based on large language model
Through large language model technology, customer service call audio is converted into text, keywords and business entities are extracted, and a complete work order is generated, which solves the problem of poor standardization of spoken information in customer service call text, and realizes the completeness and accuracy of work order content. sex.
Patent Information
- Application Number
- CN202410811266.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art cannot effectively solve the problem of numerous oral information and poor standardization in customer service call texts obtained by voice recognition, resulting in low quality of work order writing.
The large language model is used to convert customer service call audio into text, extract work order keywords, extract business entities and sample samples through user awareness identification model, predict demand types, and enter work order generation model to generate complete and accurate work orders.
Effectively identify key business information in the call text, generate work orders with complete and accurate content, accurately locate user demands through business entity discovery and demand classification models, improve the quality and accuracy of work orders, and solve the problem of colloquial information matching.
Smart Images

Figure CN119025632A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and system for automatically generating a work order based on a large language model, and relates to the field of computer technology. Background Art
[0002] Customer service work orders are an important means for operators to reflect user demands and record service processes. Currently, the customer service work orders of operators are mainly summarized and written by customer service personnel based on the conversation process with the calling customers. The quality of the work orders is highly dependent on the business capabilities and writing skills of the customer service personnel.
[0003] Patent application CN202311690079.0 (application name: A work order filling method based on generative artificial intelligence, applicants: Fujian Yirong Information Technology Co., Ltd., State Grid Information and Communication Industry Group Co., Ltd., application date: 2023.12.08) relates to a work order filling method based on generative artificial intelligence, including the following steps: inputting voice information through the work order filling terminal, and the work order filling terminal converts the input voice information into text information through a voice recognition model; the work order filling terminal inputs the text information into a self-training intention recognition model to identify the work order task type, and after identifying the work order task type, the work order filling terminal jumps to the work order filling page of the corresponding task type; the work order filling page extracts the main content of the text information through an information extraction model and automatically fills the extracted content into the work order; after the work order is filled in, the error correction model is used to correct the completed work order content, and after the error correction is completed, the user confirms whether the information is correct, and finally submits the work order. This technical solution is mainly limited to standardized voice content, for example: "I want to book conference room 403 on July 5, 2023. The theme of the meeting is the office weekly meeting", and the work order task types and work order filling pages are all pre-set. However, since the information of customer calls is scattered and non-standardized, the corresponding demand types and work order filling pages are also difficult to pre-set, that is, they cannot be summarized into a few simple types. Therefore, this technical solution cannot solve the problem of a large amount of colloquial information and poor standardization in the customer service call text obtained by voice recognition, and therefore cannot effectively guarantee the quality of work order writing.
[0004] Therefore, with the development of artificial intelligence technology, how to build the ability to automatically generate work orders based on artificial intelligence technology and effectively ensure that the work order content is complete, accurate and effective has become a technical problem that technical personnel urgently need to solve. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a method and system for automatically generating work orders based on a large language model, which can build automatic work order generation capabilities based on artificial intelligence technology and effectively ensure that the work order content is complete, accurate and effective.
[0006] In order to achieve the above object, the present invention provides a method for automatically generating a work order based on a large language model, comprising:
[0007] Step 1: Convert the customer service call audio into call text;
[0008] Step 2: Extract work order keywords from the call text and input the work order keywords into the user awareness recognition model. The user awareness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains several pre-set example samples under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples;
[0009] Step 3: The call text, work order keywords and requirement type are combined into prompt words, which are input into the work order generation model. Finally, the work order generation model generates a customer service work order.
[0010] In order to achieve the above object, the present invention also provides a work order automatic generation system based on a large language model, comprising:
[0011] A text conversion device, used to convert the customer service call audio into call text;
[0012] A consciousness recognition device is used to extract work order keywords from the call text and input the work order keywords into the user consciousness recognition model. The user consciousness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains a number of example samples pre-set under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples;
[0013] The work order generation device is used to form prompt words from the call text and its corresponding work order keywords and demand types, input them into the work order generation model, and finally generate a customer service work order by the work order generation model.
[0014] In order to achieve the above object, the present invention further provides a computing device, comprising:
[0015] Memory and processor;
[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for automatically generating work orders based on a large language model are implemented.
[0017] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method for automatically generating work orders based on a large language model are implemented.
[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention can effectively solve the problem of a large amount of colloquial information and poor standardization in the customer service call text obtained by speech recognition, identify key business information in the call text, and generate complete and accurate valid work orders based on artificial intelligence technology; the present invention further accurately locates user demands through a business entity discovery model and a demand classification model, and adds them to the prompt to strengthen the large language model's understanding of the customer service call process, thereby ensuring that the work order content is complete, accurate and effective; the present invention also specially designs a set of text similarity calculation methods suitable for this application scenario, which can effectively solve the matching problem of dialogue-style texts, and introduces part-of-speech weights to deepen the accuracy of text matching in customer service scenarios, which can effectively handle the matching problem of business-specific nouns. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The figure is a flowchart of a method for automatically generating a work order based on a large language model, shown in an exemplary embodiment of the present invention.
[0020] Figure 2 An exemplary embodiment of the present invention is shown Figure 1 Specific implementation flow chart of step 2.
[0021] Figure 3 It is a specific implementation flow chart of calculating the similarity between a call text D and any example sample Q according to an exemplary embodiment of the present invention.
[0022] Figure 4 It is a structural diagram of a work order automatic generation system based on a large language model shown in an exemplary embodiment of the present invention.
[0023] Figure 5 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.
[0025] like Figure 1 As shown, the present invention provides a method for automatically generating a work order based on a large language model, comprising:
[0026] Step 1: Convert the customer service call audio into call text;
[0027] Step 2: Extract work order keywords from the call text and input the work order keywords into the user awareness recognition model. The user awareness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains several pre-set example samples under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples;
[0028] Step 3: The call text, work order keywords and requirement type are combined into prompt words, which are input into the work order generation model. Finally, the work order generation model generates a customer service work order.
[0029] The work order generation model in step 3 can be implemented using a large language model. In order to improve the accuracy and effectiveness of the large language model in extracting key information from the work order, the present invention can extract user intention information through the user awareness recognition model in step 2 and add it to the prompt to enhance the large language model's understanding of the customer service call process. The user intention recognition model can be further divided into two sub-models: a business entity discovery model and a demand classification model, such as Figure 2 As shown, step 2 may further include:
[0030] Step 21: Extract work order keywords from the call text. The work order keywords include customer information, customer demands, customer service answers, and service guidance.
[0031] The prompt-guided model can be used to extract ticket keywords from the conversation text between customers and customer service staff;
[0032] Step 22: Input the work order keywords into the business entity discovery model. The business entity discovery model extracts the business entity to which the work order belongs from the work order keywords. The business entity discovery model can be implemented by using the pre-trained language model XLNet + Conditional Random Field (CRF) algorithm.
[0033] Using work order keywords as input for the business entity discovery model instead of directly using call text as input can significantly improve the training efficiency of the large language model by strengthening data features. The business entity discovery model extracts entity information related to the operator's business from the customer's call content, such as "broadband", "traffic", "XX package", etc. As the services of operators are constantly changing and developing, business entities are also continuously updated. Therefore, the business entity discovery model adopts the implementation scheme of "pre-trained language model XLNet + conditional random field algorithm", which can automatically discover business entities and optimize the effect of entity information extraction. Among them, XLNet is a model improved based on Transformer-XL. XLNet inherits the segmented recursion and relative position encoding mechanism of Transformer-XL, so that it can theoretically receive input text of infinite length and can perform fast reasoning. This feature of XLNet enables it to effectively process long call texts and break through the maximum input length limit commonly found in other pre-trained models. CRF uses the embedding result of each character of XLNet as input, models the context by capturing the dependency between adjacent elements in the sequence, and marks the location of the business entity. Compared with the HMM algorithm, CRF considers the joint probability distribution of the entire sequence, not just the local features, and has a better sequence labeling capability;
[0034] Step 23: Input the call text and the business entity to which the work order belongs into the demand classification model. The demand classification model is pre-set with several example samples for each business entity. The example samples are used to guide the demand classification model to output accurate demand types, and determine whether the business entity to which the work order belongs is in the common business entity table. If yes, extract several example samples under the business entity to which the work order belongs, and then predict the demand type of the work order based on the work order keywords, the business entity to which the work order belongs, and the extracted several example samples; if not, calculate the similarity between the call text and the example samples under all business entities, and select several example samples with high similarity, and then predict the demand type of the work order based on the work order keywords, the business entity to which the work order belongs, and the selected several example samples. The common business entity table is used to store multiple commonly used business entities.
[0035] The demand classification model further classifies the detailed needs of users in the business field based on the output of the business entity discovery model. The output example is as follows:
[0036] Business Entity Requirement Type Data Package Upgrade capacity Home broadband Improve your internet speed Lucky Number Number Portability
[0037] The demand classification model can adopt the few-shot learning technology of the large language model. For several common business entities (such as traffic, calls, broadband, etc.), several examples are pre-set. When predicting the demand type of a new sample, appropriate example samples are selected according to the call text and the business entity to which the work order belongs. In this way, a small number of example samples are given during model inference to guide the large language model to output accurate demand types, thereby improving the standardization of the classification results output by the large language model.
[0038] Since the business scenarios of operators contain a large number of proper nouns, the semantic features of these proper nouns have not been fully learned during the training phase of the text vectorization model. Therefore, it is difficult to accurately match the input text with the example sample only by relying on the semantic features of the text. When calculating the similarity between the call text and the example sample, the present invention innovatively designs the calculation of key information similarity. The key information similarity is used to represent the literal similarity between the call text and the example sample. By integrating the semantic similarity and the key information similarity, the call text can be accurately matched with the corresponding example sample. Figure 3 As shown, calculating the similarity between the call text D and any example sample Q may further include:
[0039] Step A1, calculating the semantic similarity Semantic_score(Q, D) between the call text D and the sample Q;
[0040] The text semantic vectors of the call text D and the sample Q can be calculated by using a text vectorization model such as BGE or Sentence-BERT, and then the cosine similarity is calculated by using the text semantic vectors to obtain the semantic similarity between the call text D and the sample Q.
[0041] Step A2: Calculate the key information correction factor according to the text length of the call text D and the example sample Q: G=1n(1+|D| / |L|), where |D| and |L| are the text lengths of the call text D and the example sample Q, respectively;
[0042] Step A3: divide the call text D and the sample Q into multiple words respectively, and mark the part of speech of each word, and then calculate the part of speech-frequency value of each word of the sample Q in the call text D: q i ,q j is the i-th and j-th word of the sample Q, Q-NUM is the total number of words in Q, Count(q i ,D)、Count(q j ,D) is q i ,q j The number of occurrences in the call text D, w qi 、w qj Yesi ,q j The part-of-speech weight;
[0043] Step A4: Calculate the similarity of key information between the call text D and the sample Q:
[0044] Step A5: Calculate the similarity between the call text D and the example sample Q: score = a*WordSim_score(Q,D)+(1-a)*semantic_score, where a is a weight coefficient, which can be set according to actual business needs, for example, a = 0.65.
[0045] In combination with the application scenario of the present invention, nouns and verbs in the customer service call text are more important, because nouns generally refer to the core elements of the operator's products, packages, etc., and verbs can describe the specific behavior of users. The nouns and verbs in the text together constitute the key content of the customer service call. Therefore, the calculation formula of the part-of-speech weight in step A3 is as follows: w l is the part-of-speech weight of word l.
[0046] like Figure 4 As shown, the present invention provides a work order automatic generation system based on a large language model, comprising:
[0047] A text conversion device, used to convert the customer service call audio into call text;
[0048] A consciousness recognition device is used to extract work order keywords from the call text and input the work order keywords into the user consciousness recognition model. The user consciousness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains a number of example samples pre-set under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples;
[0049] The work order generation device is used to form prompt words from the call text and its corresponding work order keywords and demand types, input them into the work order generation model, and finally generate a customer service work order by the work order generation model.
[0050] See also Figure 5 , Figure 5 5 is a block diagram of a computing device 500 shown in an exemplary embodiment of this specification. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0051] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.
[0052] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0053] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 500 may also be a mobile or stationary server or a cloud server, etc.
[0054] The processor 520 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned method for automatically generating work orders based on a large language model.
[0055] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned method for automatically generating work orders based on a large language model belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the above-mentioned method for automatically generating work orders based on a large language model.
[0056] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned method for automatically generating work orders based on a large language model.
[0057] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned method for automatically generating work orders based on a large language model belong to the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned method or system for automatically generating work orders based on a large language model.
[0058] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned method for automatically generating work orders based on a large language model.
[0059] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned method for automatically generating work orders based on a large language model belong to the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the above-mentioned method or system for automatically generating work orders based on a large language model.
[0060] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0061] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0062] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for automatically generating work orders based on a large language model, characterized in that: Included are: Step 1: Convert the customer service call audio into call text; Step 2: Extract work order keywords from the call text and input the work order keywords into the user awareness recognition model. The user awareness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains several pre-set example samples under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples; Step 3: The call text, work order keywords and requirement type are combined into prompt words, which are input into the work order generation model. Finally, the work order generation model generates a customer service work order.
2. The method according to claim 1, characterized in that: The work order generation model is implemented using a large language model. The user intent recognition model is divided into two sub-models: business entity discovery model and demand classification model. Step 2 further includes: Step 21: Extract work order keywords from the call text. The work order keywords include customer information, customer demands, customer service answers, and service guidance. Step 22: Input the work order keywords into the business entity discovery model, and the business entity discovery model extracts the business entity to which the work order belongs from the work order keywords; Step 23, input the call text and the business entity to which the work order belongs into the demand classification model. The demand classification model pre-sets a number of example samples for each business entity, and determines whether the business entity to which the work order belongs is in the common business entity table. If so, extract a number of example samples under the business entity to which the work order belongs, and then predict the demand type of the work order based on the work order keywords, the business entity to which the work order belongs and the extracted number of example samples; if not, calculate the similarity between the call text and the example samples under all business entities, and select a number of example samples with high similarity, and then predict the demand type of the work order based on the work order keywords, the business entity to which the work order belongs and the selected number of example samples.
3. The method according to claim 2, characterized in that In step 23, calculating the similarity between the call text D and the example sample Q further includes: Step A1, calculating the semantic similarity Semantic_score(Q, D) between the call text D and the example sample Q; Step A2: Calculate the key information correction factor according to the text length of the call text D and the example sample Q: G = ln(1 + |D| / |L|), where |D| and |L| are the text lengths of the call text D and the example sample Q, respectively; Step A3: divide the call text D and the sample Q into multiple words respectively, and mark the part of speech of each word, and then calculate the part of speech-frequency value of each word of the sample Q in the call text D: q i ,q j is the i-th and j-th word of the sample Q, Q-NUM is the total number of words in Q, Count(q i ,D)、Count(q j , D) is q i ,q j The number of occurrences in the call text D, Yes i ,q j The part-of-speech weight; Step A4: Calculate the similarity of key information between the call text D and the sample Q: Step A5: Calculate the similarity between the call text D and the example sample Q: score = a*WordSim_score(Q, D)+(1-a)*Semantic_score, where a is the weight coefficient.
4. The method according to claim 3, characterized in that The calculation formula for the part-of-speech weight in step A3 is as follows: wl is the part-of-speech weight of word l.
5. A work order automatic generation system based on a large language model, characterized in that: Included are: A text conversion device, used to convert the customer service call audio into call text; A consciousness recognition device is used to extract work order keywords from the call text and input the work order keywords into the user consciousness recognition model. The user consciousness recognition model first extracts the business entity to which the work order belongs from the work order keywords, then obtains a number of example samples pre-set under the business entity, and finally predicts the demand type of the work order based on the work order keywords, business entity and example samples; The work order generation device is used to form prompt words from the call text and its corresponding work order keywords and demand types, input them into the work order generation model, and finally generate a customer service work order by the work order generation model.
6. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for automatically generating work orders based on a large language model as described in any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method for automatically generating work orders based on a large language model as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Work order filling method based on generative artificial intelligence
CN117634442A
Cited By
Automatic voice work order generation system and method based on multi-modal processing
CN120766676A
Business work order processing method and system and electronic equipment
CN121616054A
