Work order processing method, device and equipment based on large model, medium and product

Abstract

The embodiment of the invention provides a work order processing method and device based on a large model, equipment, a medium and a product. The method comprises the following steps: converting call voice corresponding to a to-be-processed work order into a to-be-processed text; a cue word is constructed, the cue word is used for indicating the large model to execute work order processing on the to-be-processed text, and work order processing comprises at least one of element extraction, work order classification and work order summarization; and calling the large model, inputting the cue word into the large model, so that the large model performs corresponding processing on the to-be-processed text based on the cue word to obtain a work order processing result, and through combination of voice-to-text processing and large model processing, the processing period of a single work order is greatly shortened, the processing efficiency of the work order is improved, and nearly real-time processing capability is realized. Therefore, the throughput of the whole customer service or operation and maintenance system is remarkably improved, the large model is guided through the elaborately designed cue words, standardization and normalization of the processing process can be ensured, and the accuracy of work order processing is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a work order processing method, apparatus, equipment, medium and product based on a large model. Background Technology

[0002] The application of natural language processing (NLP) technology in intelligent customer service systems has become an important means to improve customer service efficiency and quality. Traditional intelligent customer service systems mainly rely on rule-based methods or early machine learning techniques, using NLP modules to parse user input and extract user intent and key entity information. In recent years, with the development of deep learning technology, pre-trained models such as recurrent neural networks, long short-term memory networks, and BERT (Bidirectional Encoder Representations from Transformers) have gradually become the mainstream technical solutions for achieving semantic understanding. These methods typically map user input text to a high-dimensional vector space to capture semantic features, contextual relationships, and sentiment information, thereby completing tasks such as intent classification and slot filling.

[0003] However, the above methods have many shortcomings when dealing with complex customer service scenarios, including inaccurate recognition of user intent, knowledge base matching deviation, weak contextual understanding ability, and poor performance in processing long texts. Summary of the Invention

[0004] This application provides a work order processing method, apparatus, equipment, medium, and product based on a large model, to improve the efficiency and accuracy of customer service work order processing.

[0005] In a first aspect, embodiments of this application provide a work order processing method based on a large model, the method comprising:

[0006] Convert the voice recordings corresponding to the work orders to be processed into text to be processed.

[0007] Construct prompt words, which are used to instruct the large model to perform work order processing on the text to be processed. The work order processing includes at least one of the following: feature extraction, work order classification, and work order summary.

[0008] The large model is invoked, and the prompt words are input into the large model so that the large model performs corresponding processing on the text to be processed based on the prompt words, thereby obtaining the work order processing result.

[0009] Optionally, the method further includes:

[0010] In the experimental environment, the large model is pre-trained and tuned based on the training data, the tuned large model is evaluated, and the model parameters of the large model that has passed the evaluation are imported into the test environment.

[0011] In the test environment, inference images are built through continuous integration and continuous delivery processes to evaluate model performance and review the content generated by the model, and the large model that passes the review is pushed to the production environment.

[0012] In a production environment, model inference images are packaged through continuous integration and continuous delivery processes for model release, inference, and monitoring.

[0013] Optionally, the step of pre-training and optimizing a large model based on training data in the experimental environment specifically includes:

[0014] Acquire training data, which is based on work order data generated by the production system and work order processing results output by the large model in the past. The training data includes work order types and work order summaries.

[0015] The call samples in the training data are converted into training text, and the large model is invoked and instructed to perform work order processing through prompt words. Specifically, for work order processing of the type of feature extraction, the corresponding prompt words are used to indicate the type of feature that the large model needs to focus on; for work order processing of the type of work order classification, the corresponding prompt words are used to indicate the category definition and description of each category in the large model.

[0016] The large model is then optimized based on the processing results of the large model's output of the training text.

[0017] Optionally, the step of optimizing the large model based on the processing results of the large model's output of the training text includes:

[0018] For the pre-defined complex business work order classification, the large model is optimized based on the historical work order classification data recorded by the agents;

[0019] For work order summaries, the large model is optimized based on historical work order summary data recorded by agents.

[0020] Optionally, the optimization of the large model includes:

[0021] Based on the pre-trained large model, the low-rank adaptive fine-tuning algorithm is used to fine-tune the large model according to the defined loss function.

[0022] Optionally, the monitoring of the model includes:

[0023] The performance of the large model is evaluated based on its performance parameters, including accuracy, recall, and ROUGE score.

[0024] Based on monitoring tools and preset thresholds, the system operation status of the large model is monitored, and if an anomaly is detected, an alarm is triggered.

[0025] Secondly, embodiments of this application provide a work order processing device based on a large model, the device comprising:

[0026] The conversion module is used to convert the voice call corresponding to the work order to be processed into the text to be processed;

[0027] A construction module is used to construct prompt words, which are used to instruct the large model to perform work order processing on the text to be processed. The work order processing includes at least one of the following: feature extraction, work order classification, and work order summary.

[0028] The calling module is used to call the large model, input the prompt words into the large model, so that the large model performs corresponding processing on the text to be processed based on the prompt words, and obtains the work order processing result.

[0029] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0030] The memory stores computer-executed instructions;

[0031] The processor executes computer execution instructions stored in the memory, causing the processor to perform various possible implementations as described above.

[0032] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement various possible implementations as described in any of the above aspects.

[0033] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements various possible implementations as described in any of the above aspects.

[0034] This application provides a method, apparatus, device, medium, and product for processing work orders based on a large model. The method includes: converting the voice of a call corresponding to a work order to be processed into text to be processed; constructing prompt words, which instruct the large model to perform work order processing on the text to be processed, wherein the work order processing includes at least one of the following: element extraction, work order classification, and work order summary; invoking the large model by inputting the prompt words into the large model, so that the large model performs corresponding processing on the text to be processed based on the prompt words to obtain the work order processing result. This method combines voice-to-text conversion with large model processing, realizing end-to-end automation, greatly shortening the processing cycle of a single work order, improving the processing efficiency of work orders, and achieving near real-time processing capabilities, thereby significantly improving the throughput of the entire customer service or operation and maintenance system. Furthermore, by guiding the large model with carefully designed prompt words, the standardization and normalization of the processing process can be ensured, improving the accuracy of work order processing. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] Figure 1 An application scenario diagram provided for an embodiment of this application;

[0037] Figure 2 A flowchart illustrating a work order processing method based on a large model, provided for an embodiment of this application;

[0038] Figure 3 A schematic diagram illustrating the collaborative development process of an AI model, provided as an embodiment of this application;

[0039] Figure 4 A schematic diagram of system interaction provided for an embodiment of this application;

[0040] Figure 5 A schematic diagram of the structure of a work order processing device based on a large model provided in this application;

[0041] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] The application of natural language processing (NLP) technology in intelligent customer service systems has become an important means to improve customer service efficiency and quality. Traditional intelligent customer service systems mainly rely on rule-based methods or early machine learning techniques, using a natural language understanding module to parse user input and extract user intent and key entity information. Rule-based methods heavily rely on domain experts manually writing a large number of grammar and dictionary rules. While they can guarantee accuracy in closed, fixed scenarios, they lack flexibility and scalability, making it difficult to cope with the diversity and variability of user expressions. Early machine learning methods, such as support vector machines and decision trees, while achieving automated learning to some extent, heavily depend on manually defined feature engineering for performance, and their model capabilities have a limited upper limit, making it difficult to capture the complex semantics and contextual relationships in natural language.

[0045] In recent years, with the development of deep learning technology, recurrent neural networks (RNNs), long short-term memory (LSTM) networks, and pre-trained models such as BERT have gradually become mainstream technologies for achieving semantic understanding. These methods typically map user input text to a high-dimensional vector space to capture semantic features, contextual relationships, and sentiment information, thereby completing tasks such as intent classification and slot filling. Specifically, RNNs and their variants (such as LSTM networks) can model sequential data and handle a certain degree of contextual dependence; while pre-trained models based on the Transformer architecture, represented by BERT, can more effectively capture global semantic information of text through self-attention mechanisms, achieving breakthroughs in multiple natural language understanding tasks.

[0046] However, despite the significant improvement over traditional methods, the aforementioned deep learning-based solutions still reveal many inherent technical bottlenecks and shortcomings in practical deployment and application, especially when dealing with complex and ever-changing real-world customer service scenarios. These include inaccurate user intent recognition, knowledge base matching bias, weak contextual understanding, and poor long text processing performance.

[0047] In view of this, this application provides a work order processing method based on a large model. The customer service system converts the voice call corresponding to the work order to be processed into text to be processed, and then constructs prompt words. The prompt words are used to instruct the large model to perform work order processing on the text to be processed. Work order processing includes at least one of the following: element extraction, work order classification, and work order summary. Then, the prompt words are input into the large model, and the large model performs corresponding processing on the text to be processed based on the prompt words to obtain the work order processing result. This method combines voice-to-text conversion with large model processing, realizing end-to-end automation, greatly shortening the processing cycle of a single work order, improving the processing efficiency of work orders, and achieving near real-time processing capabilities, thereby significantly improving the throughput of the entire customer service or operation and maintenance system. Furthermore, by guiding the large model with carefully designed prompt words, the standardization and normalization of the processing process can be ensured, improving the accuracy of work order processing.

[0048] Figure 1 An application scenario diagram provided for an embodiment of this application, such as... Figure 1 As shown, the customer service system converts the voice recordings of pending work orders into text to be processed. Then, the customer service system constructs prompts, which instruct the large model to perform work order processing on the text. Work order processing includes at least one of the following: feature extraction, work order classification, and work order summary. Finally, the large model is invoked, and the constructed prompts are input into the large model. The large model can then perform corresponding processing on the text based on the prompts, obtain the work order processing results, and send the results back to the customer service system.

[0049] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0050] Figure 2 This application provides a flowchart illustrating a work order processing method based on a large model. The executing entity in this embodiment can be a device with data processing capabilities. This application uses a customer service system as an example for specific illustration. Figure 2 As shown in the embodiments of this application, a work order processing method based on a large model may include:

[0051] Step 201: Convert the voice recording of the call corresponding to the work order to be processed into text to be processed.

[0052] Among them, the call audio corresponding to the work order to be processed is the call audio between the agent and the customer.

[0053] In one alternative implementation, after the call between the agent and the customer ends, the customer service system uses a speech recognition module to convert the voice recordings of the call into text to be processed. Specifically, the speech recognition module is used to convert the voice signal into a text sequence.

[0054] In another alternative implementation, during a call between an agent and a customer, the customer service system's speech recognition module converts the agent's voice into text to be processed in real time.

[0055] Step 202: Construct prompt words. Prompt words are used to instruct the large model to perform work order processing on the text to be processed. Work order processing includes at least one of the following: feature extraction, work order classification, and work order summary.

[0056] Among them, prompt words refer to a series of text instructions and contextual information that are combined together and input into the large language model to guide, constrain, and control the model to perform specific "work order processing" tasks and output results that meet the expected format and content requirements.

[0057] Feature extraction involves extracting features from the text to be processed. The prompts for feature extraction are used to instruct the large model to complete the feature extraction task and to identify the types of features to focus on. For example, feature types may include: customer name, contact information, responsible unit, etc.

[0058] Work order classification involves categorizing the text to be processed. The prompts for work order classification are used to instruct the large model to complete the work order classification task, as well as the type of work order classification. For more technical category names or category names that differ from their literal meaning, it is necessary to supplement the definition and description of the category name.

[0059] The work order summary is a summary of the text to be processed, that is, extracting a text summary of the text to be processed. The prompt words corresponding to the work order summary are used to instruct the large model to complete the work order summary task.

[0060] Step 203: Call the large model and input the prompt words into the large model so that the large model can perform corresponding processing on the text to be processed based on the prompt words and obtain the work order processing result.

[0061] Specifically, the customer service system calls the large model and inputs the constructed prompt words into the large model. The large model performs corresponding processing on the text to be processed based on the prompt words, and can obtain the work order processing result. The work order processing result is displayed on the user interface so that the agent can confirm whether the work order processing result is accurate and correct any inaccurate work order processing results.

[0062] The work order processing method based on a large model provided in this application converts the voice call corresponding to the work order to be processed into text to be processed, constructs prompt words, and instructs the large model to perform work order processing on the text to be processed. Work order processing includes at least one of the following: element extraction, work order classification, and work order summary. The large model is invoked, and the prompt words are input into the large model so that the large model can perform corresponding processing on the text to be processed based on the prompt words to obtain the work order processing result. This method combines voice-to-text conversion with large model processing to achieve end-to-end automation, greatly shortens the processing cycle of a single work order, improves the processing efficiency of work orders, and achieves near real-time processing capabilities, thereby significantly improving the throughput of the entire customer service or operation and maintenance system. Furthermore, by using carefully designed prompt words to guide the large model, the standardization and normalization of the processing process can be ensured, improving the accuracy of work order processing.

[0063] Optionally, the work order processing method based on a large model provided in this application also includes:

[0064] In the experimental environment, the large model is pre-trained and tuned based on the training data, the tuned large model is evaluated, and the model parameters of the large model that has passed the evaluation are imported into the test environment.

[0065] In the test environment, inference images are built through continuous integration and continuous delivery processes to evaluate model performance and review the content generated by the model, and the large model that passes the review is pushed to the production environment.

[0066] In a production environment, model inference images are packaged through continuous integration and continuous delivery processes for model release, inference, and monitoring.

[0067] Among them, the continuous integration and continuous delivery process is the CICD (Continuous Integration / Continuous Delivery / Continuous Deployment) process.

[0068] Figure 3 This application provides a schematic diagram of a collaborative AI model development process, as illustrated in the embodiments of this application. Figure 3As shown, in the laboratory environment, the first step is to prepare the model data, including data collection, cleaning, and preprocessing. Next, model training is performed, involving the design and implementation of corresponding algorithms and model structures, followed by training and optimization. Finally, the trained models are evaluated, and high-performing models are selected and imported into a large model parameter repository and a model inference image repository. The large model parameter repository stores model parameter information and supports export to the model artifact repository in the testing environment. In the testing environment, an inference image is built using a CI / CD process to perform model scenario performance testing and model-generated content review. Due to the nature of text generation tasks, issues such as illusions or sensitive words are unavoidable. A model content review platform is used to screen the content generated by the large model, detecting any sensitive or illegal information. Models that fail the review are returned to the laboratory stage for reinforcement learning optimization based on human feedback until the generated content reaches a safe and usable state. The model is then pushed to the model artifact repository in the production environment. To ensure that the models in the production environment are always at their best performance, the highest-performing model is selected from the model artifact repository, a Docker image is built using a CI / CD process, and the new image is deployed to the production environment. Furthermore, a version control system is used to control and track model versions and configuration files. By creating clear version naming conventions for the models, the traceability of production inference models can be ensured quickly. Using a version control system for collaborative development and asset management of model source code, configuration files, training scripts, etc., allows for recording the history of every code or configuration change, facilitating code review, issue tracing, and version rollback for team members. During the model inference phase, a model serialization library is used to save the model in a portable file format for loading and use in different environments. After image deployment, connectivity testing of the new model instance is performed using automated test cases. After the model update is completed and the connectivity test passes, the production environment nodes and traffic are switched over. Finally, data from model monitoring and event tracking is used to assess whether the model has the conditions for self-learning. After collecting a significant amount of data (several thousand data points), fine-tuning training and performance evaluation of the model can be initiated, thereby achieving self-learning and automatic updates. In addition, based on the model exit mechanism set by the customer service system and business departments, the relevant model exit procedures are executed for models that meet the exit mechanism.

[0069] In this way, by using AI (Artificial Intelligence) models to collaboratively develop a pipeline, we can create a guided, standardized, and process-oriented R&D pipeline, achieve full lifecycle management of models, improve model development efficiency, and quickly support the implementation of application scenarios.

[0070] Figure 4 A system interaction diagram provided for an embodiment of this application, such as Figure 4As shown, the customer service system transcribes call recordings between agents and customers in real time, using the text to be processed as input to a large-scale model. This model extracts elements, categorizes work orders, and summarizes them, automatically filling the system with the work order processing results. This application integrates data annotation into the agent's workflow. Agents verify the automatically entered data from the large-scale model and correct any categorization errors or inaccurate work order summaries. When an agent modifies and submits a work order, the system's tracking mechanism is triggered to record and store the agent's modifications, enabling the discovery and annotation of production problem data.

[0071] Optionally, in the experimental environment, the large model can be pre-trained and tuned based on the training data, specifically including:

[0072] The training data is obtained based on the work order data generated by the production system and the work order processing results output by the large model in the past. The training data includes work order types and work order summaries.

[0073] The call samples in the training data are converted into training text, and the large model is invoked and instructed to perform work order processing through prompt words. Specifically, for work order processing of the type of feature extraction, the corresponding prompt words are used to indicate the type of feature that the large model needs to focus on; for work order processing of the type of work order classification, the corresponding prompt words are used to indicate the category definition and description of each category in the large model.

[0074] The large model is then optimized based on the processing results of the training text output.

[0075] The training data can come from work order data generated by the production system. After a call with a customer, the agent needs to record various aspects of the call, including key elements of the conversation, work order type, and work order summary, and submit this information to the work order system. In the training data: the text of the conversation between the agent and the customer can serve as samples, while the key elements of the conversation, work order type, and work order summary recorded by the agent can serve as labels.

[0076] Training data can also be based on the historical output of the large model of work order processing results. The work order processing results output by the large model will be submitted to the work order system after being reviewed by the agents. In this case, the training data can include: the text of the dialogue between the agent and the customer as samples of the training data, and the key elements of the dialogue content after the agent's review, the work order type, the work order summary and other elements as labels.

[0077] After obtaining the training data, data preprocessing and data segmentation are also required.

[0078] The main contents of data preprocessing include: noise reduction, deduplication, desensitization, and data construction.

[0079] (1) Noise Reduction: Since the text content of the conversation between the agent and the customer can be transcribed by the ASR system, there may be speech recognition errors, special characters and punctuation marks, etc. To address the above issues, this system uses a text correction script tool to perform data noise reduction.

[0080] (2) Deduplication: For long text data such as call records, there may be duplicate or nearly duplicate content. This application uses text similarity calculation technology and hash-based duplicate text retrieval method to identify and remove duplicate content, avoiding excessive focus on duplicate data during model training and reducing its generalization ability.

[0081] (3) Desensitization: The data involved in this application are all real data from the production environment, which poses a risk of exposing user privacy. For regulatory compliance considerations, it is necessary to desensitize key user information. In addition, masking user personal information can improve the efficiency of training large models. This system uses a natural language processing model to perform entity recognition on elements in the data, including information such as "name", "organization name", "location", "telephone number", "bank card number", "date", "email address", and "ID card number". Based on the desensitization elements configured in the system, the system automatically completes the information desensitization, replacing key information with "*" or category labels, such as [name].

[0082] (4) Data construction: The cleaned and processed data format is converted to construct a data format that meets the training requirements of the model.

[0083] Data segmentation involves dividing labeled and preprocessed training data into training, testing, and validation sets for model training and performance evaluation.

[0084] Optionally, periodically updating the test set is one way to ensure the robustness of model performance. A reasonable update frequency should be determined based on the evolution of business needs and the characteristics of data distribution changes to maintain the representativeness of the test set. For the large-scale model fine-tuning scenario of this invention, the amount of fine-tuning data required for a single customer service scenario is in the thousands to tens of thousands. Adding more than 5,000 new data entries to the existing fine-tuning data is sufficient to initiate training and test set updates for large-scale model fine-tuning training. The ratio of training to test set data in the newly added data is generally 9:1. Furthermore, the frequency of test set updates and large-scale model training can be adjusted based on the daily transaction data volume of the customer service system, whether it is done daily, weekly, or monthly.

[0085] This invention establishes an automated data pipeline to automate the process of integrating new data sources and updating test sets. It utilizes workflow management tools to automate data collection, processing, and distribution. By configuring multiple processors to connect input and output ports, it achieves an automated data flow processing pipeline within the data warehouse. Furthermore, after deploying the workflow management tool, system administrators can view data flow performance metrics and real-time logs on a web interface, ensuring the entire automated data processing workflow operates normally.

[0086] Specifically, the call samples in the training data are converted into training text through preprocessing. The call samples are the text of the conversation between the agent and the customer, and the training text is the text of the conversation between the agent and the customer after preprocessing.

[0087] For work orders of the feature extraction type, the training text is sent to the large model, and a Prompt is used to inform the model of the feature extraction task and the types of features to focus on. The large model extracts key features and automatically populates the form into the system. An example of a Prompt for feature extraction from the large model is as follows:

[0088] You are a bank customer service representative. Based on the conversation between you and a customer, extract specified information, fill in missing fields in a JSON file, and return the result. The input is ${doc_text}, and the output format is...

[0089] {

[0090] Name: "To be added"

[0091] Responsible unit: "To be supplemented"

[0092] Complaint channels: "To be added"

[0093] }

[0094] A sample example is as follows: ${doc_text} is: [|zx: Hello, what's your name? How can I help you? |kh: Hello, this is Ms. Li. I've been going to Branch A for over six months now. Every time I go to withdraw money on Sundays, they insist on doing it at the counter, not at the ATM. At other times, I have to use the ATM because it's my retirement pension from a certain year, and there's some change in it, so they won't let me withdraw it. This has been going on for about seven months now, every single month. |zx: Uh, which branch are you referring to? |kh: Branch A. |zx: In which city is Yinhua located? |kh: Next to the TV tower... |kh: Okay, thank you! |zx: You're welcome. Do you have any other questions? Okay, Ms. Li, thank you for your call.] The large model output is:

[0095] {

[0096] Name: "Ms. Li"

[0097] Responsible unit: "Branch A"

[0098] Complaint channel: "Telephone"

[0099] }

[0100] For work order classification, the training text is sent to the large model, and the Prompt project is used to tell the large model that it needs to complete the work order classification task. The Prompt needs to tell the large model the classification category. For more technical category names or category names that are different from their literal meaning, the definition and description of the category name need to be added to guide the large model in learning the meaning of the category label.

[0101] For work order processing of type "work order summary", the training text is sent to the large model, and the Prompt project is used to tell the large model that it needs to complete the work order summary task. Based on the natural language understanding and processing capabilities of the large model, text summarization is performed on the call content between the agent and the customer to obtain the work order summary.

[0102] Finally, the customer service system optimizes the large model based on the processing results of the large model's output of the training text.

[0103] By building a closed loop using real business data and fine-tuning the large model using task-oriented, refined prompting engineering, a highly efficient, accurate, standardized, and continuously evolving intelligent work order processing system was ultimately achieved.

[0104] Optionally, the large model can be optimized based on its processing results from the training text output, including:

[0105] For the pre-defined complex business work order classification, the large model is optimized based on the historical work order classification data recorded by the agents.

[0106] For work order summaries, the large model is optimized based on historical work order summary data recorded by agents.

[0107] Specifically, for pre-defined complex business order classifications, the customer service system can fine-tune the large model based on historical order classification data recorded by agents to improve the accuracy of order classification. A sample Prompt for fine-tuning the large order classification model is as follows:

[0108] {

[0109] "input": "Based on the following work order categories for banking services: [Follow-up, Inquiry, Suggestion, Complaint Withdrawal, Praise, Complaint], please use your understanding of banking services, combined with the following dialogue between customer service and user: [|zx: Hello, what's your name? How can I help you? |kh: Hello, this is Ms. Li. I've been going to Branch A for over six months. Every time I go to withdraw money on Sundays, I have to go to the counter; I can't withdraw cash there. At other times, I have to use the ATM. Because this is my retirement pension from some year, there are odd amounts, so every time..." They won't let me withdraw it. It's been about seven months, every single month. |zx: Uh, which branch are you referring to? |kh: Branch A. |zx: Which city is Yinhua in? |kh: It's next to the Chengdu TV Tower... |kh: Okay, thank you! |zx: You're welcome. Do you have any other questions? Okay, Ms. Li, thank you for your call. [Based on the provided list of work order types, output the work order type to which this conversation belongs. Requirement: The output work order type name must be within the provided list.]

[0110] "target": "complaint"

[0111] }

[0112] For work order summaries, the text summary generation paradigm learned by the large model in the unsupervised task differs from the summary paradigm in the customer service system. This task requires fine-tuning the large model using historical work order summary data from agent records to ensure that the summary generation paradigm of the large model is the same as the summary paradigm of agent records in the customer service system. A sample Prompt for fine-tuning the large model for work order summaries is as follows:

[0113] {

[0114] "input": "You are a bank customer service representative with extensive banking experience. Based on your professional knowledge, summarize the user's needs based on the dialogue between the customer and the representative. Requirements: 1. 50-200 words; 2. The summary must be relevant to the dialogue and cannot be fabricated; 3. The sentences must be fluent and logical. The dialogue between the customer service representative and the user is as follows: [|zx: Hello, how can I help you? |kh: Hello. |zx: Hello sir. |kh: Uh, why did I receive a text message from ICBC? |zx: Um, this is Bank of Communications, sir. |kh: Bank of Communications? |zx: Yes, this is Bank of Communications. |kh: Bank of Communications. Bank of Communications counter. Why did Bank of Communications send me this? |zx: Do I have a Bank of Communications account? Oh, you are not a Bank of Communications customer, and you received a text message from Bank of Communications, sir. |kh: Yes! |zx: Um, what is the content of the text message you received? |kh: Uh, one. He said..." In the Bank of Communications reminder, Mr. Xiong is you. Yes, some documents will expire in seven days. Please check your valid ID card. The suggestion is to go to the bank or mobile banking. In the app, go to your personal information, maintain it, click below, edit and update in time. Your account will be suspended. Please provide account transaction services for you. |zx: Yes, so you are this. |kh: And this is? |zx: Yes, is this you, sir? |kh: No, I am now. |zx: Yes, you received a text message from someone else, right? Yes, it should be our ***'s text message. And yes, have you changed your phone number recently, sir? |kh: No! |zx: Yes, in that case, it should be you. You received a text message from someone else. It's possible that someone else left the number with the bank, and the process was wrong, and it was left with your number, so you received the message. That's probably the situation. |kh: I thought it was a problem with your bank's payment, so you sent me a text message... |zx: Okay, thank you, thank you.

[0115] "target": "Verification of mistakenly received SMS. On [Date], my mobile number ************ received an SMS from *****, which roughly stated: 'Mr. / Ms. **, your ID document registered with our bank is about to expire. Please verify your identity information as soon as possible.' I have never conducted any business with Bank of Communications, and I have not recently changed my mobile number. I do not know anyone named **, and no one has lent my mobile number to me. I request that Bank of Communications verify the reason for sending this type of SMS and cancel the sending of such SMS messages. Please verify, process, and reply."

[0116] }

[0117] In this way, training the model with the high-quality judgment results left by the agents when dealing with these difficult problems is equivalent to injecting the implicit experience and decision-making logic of the agents into the large model, which can greatly improve the classification accuracy and reliability of the model in complex scenarios.

[0118] Optionally, the large model can be tuned, including:

[0119] Based on the pre-trained large model, the low-rank adaptive fine-tuning algorithm is used to fine-tune the large model according to the defined loss function.

[0120] Among them, the low-rank adaptive fine-tuning algorithm is the LoRa algorithm.

[0121] Specifically, based on a pre-trained large model foundation, fine-tuning and updating the large model is performed to suit the target task of the current scenario. Model fine-tuning is conducted on the target task dataset, defining loss functions for the target task such as cross-entropy loss, and using gradient descent to adjust network parameters to adapt to the feature representation of the target task. This stage aims to transfer the knowledge of the pre-trained model, enabling it to better adapt to the context or features of the target task.

[0122] In traditional full-parameter fine-tuning, the model parameters are randomly initialized to Φ0. Training is repeated for each batch, and the model parameters are updated Φ0 + ΔΦ using a gradient update algorithm, as shown in the following formula:

[0123]

[0124] The main drawback of this fine-tuning method is that for language models with a huge number of parameters, each fine-tuning training requires adjusting the weights by the same order of magnitude as the original parameter set, which requires massive training data and consumes a lot of computing resources.

[0125] The LoRa training method used in this application, compared to traditional fine-tuning methods, not only helps reduce storage and computational costs, but also facilitates model capability transfer on small datasets by refining model parameters, ensuring that the output of large models can follow prompt instructions and match human expectations. For each different downstream task, the corresponding low-rank matrix can be learned and updated, achieving scenario-specific model performance updates through two smaller-sized low-rank matrices. A low learning rate is used during fine-tuning training, and smaller step sizes are applied to these parameters to minimize the impact on the capabilities of the base model.

[0126] Optionally, to ensure the timeliness of large model updates, the system will divide training into full training and incremental training. The system supports setting up daily incremental training, such as triggering a training task every evening, using the day's data for model training. It automatically pushes the day's processed work order data to the model and uses automated scripts and scheduled tasks to fine-tune the existing model multiple times in small batches. Daily updates require ensuring that the daily incremental training data is at least several thousand records; the specific frequency of incremental training is set based on the amount of data the system can acquire daily.

[0127] Full training involves acquiring all data from a data warehouse over a specific period and retraining the pre-trained model using LoRa fine-tuning. This method addresses the performance drift issue caused by incremental training by typically conducting full-data fine-tuning training monthly or quarterly to ensure that the training data maintains the same distribution as the real production data, thereby achieving optimal results.

[0128] In this way, by choosing a parameter-efficient and computationally lightweight approach, high-performance specialization of large models on specific tasks is achieved, while perfectly solving the core pain points of high cost, catastrophic forgetting, and cumbersome deployment caused by full parameter fine-tuning.

[0129] Optionally, model monitoring can be performed, including:

[0130] The performance of the large model is evaluated based on its performance parameters, including accuracy, recall, and ROUGE score.

[0131] Based on monitoring tools and preset thresholds, the system operation status of the large model is monitored, and if an anomaly is detected, an alarm is triggered.

[0132] Establish a log system to record historical operations and events.

[0133] Specifically, for the feature extraction and work order classification functions of large models, the model performance can be evaluated using accuracy, recall, and F1 score.

[0134] For the work order summary function of the large model, the model performance can be evaluated using ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores. LoRa models obtained from the daily incremental training at the model artifact repository are used to evaluate the large model using the above evaluation method, recording their scores across various dimensions. These scores are then compared with the performance of the production model to determine whether an update and deployment of the production model is necessary.

[0135] The monitoring tools are primarily used to monitor the overall system status, including hardware devices, software operation, and network connectivity. Furthermore, real-time monitoring can be achieved by setting thresholds, such as CPU utilization exceeding 90% or GPU utilization exceeding 95%. Once a problem is detected, such as GPU utilization exceeding 95%, alerts can be immediately sent via email, SMS, or other means.

[0136] You can also create logs to record all historical operations and events, which will facilitate subsequent troubleshooting and analysis.

[0137] This provides comprehensive protection from three dimensions: business effectiveness, system stability, and traceability, greatly reducing application risks in the production environment and ensuring service quality.

[0138] Corresponding to the above-described work order processing method based on a large model, this application also provides a work order processing device based on a large model. Figure 5 A schematic diagram of a work order processing device based on a large model is provided for this application, as shown below. Figure 5 As shown in the figure, this embodiment provides a work order processing device based on a large model, including:

[0139] The conversion module 501 is used to convert the voice call corresponding to the work order to be processed into text to be processed.

[0140] Module 502 is used to construct prompt words. The prompt words are used to instruct the large model to perform work order processing on the text to be processed. Work order processing includes at least one of the following: feature extraction, work order classification, and work order summary.

[0141] Module 503 is used to call the large model. The prompt words are input into the large model so that the large model can perform corresponding processing on the text to be processed based on the prompt words and obtain the work order processing result.

[0142] Optionally, module 503 can also be used to:

[0143] In the experimental environment, the large model is pre-trained and tuned based on the training data, the tuned large model is evaluated, and the model parameters of the large model that has passed the evaluation are imported into the test environment.

[0144] In the test environment, inference images are built through continuous integration and continuous delivery processes to evaluate model performance and review the content generated by the model, and the large model that passes the review is pushed to the production environment.

[0145] In a production environment, model inference images are packaged through continuous integration and continuous delivery processes for model release, inference, and monitoring.

[0146] Optionally, when calling module 503 in the experimental environment to perform model pre-training and tuning on a large model based on training data, it is specifically used for:

[0147] The training data is obtained based on the work order data generated by the production system and the work order processing results output by the large model in the past. The training data includes work order types and work order summaries.

[0148] The call samples in the training data are converted into training text, and the large model is invoked and instructed to perform work order processing through prompt words. Specifically, for work order processing of the type of feature extraction, the corresponding prompt words are used to indicate the type of feature that the large model needs to focus on; for work order processing of the type of work order classification, the corresponding prompt words are used to indicate the category definition and description of each category in the large model.

[0149] The large model is then optimized based on the processing results of the training text output.

[0150] Optionally, when calling module 503 to optimize the large model based on the processing results of the large model's output of the training text, it is specifically used for:

[0151] For the pre-defined complex business work order classification, the large model is optimized based on the historical work order classification data recorded by the agents.

[0152] For work order summaries, the large model is optimized based on historical work order summary data recorded by agents.

[0153] Optionally, when calling module 503 to optimize a large model, it is specifically used for:

[0154] Based on the pre-trained large model, the low-rank adaptive fine-tuning algorithm is used to fine-tune the large model according to the defined loss function.

[0155] Optionally, when calling module 503 to monitor the model, it is specifically used for:

[0156] The performance of the large model is evaluated based on its performance parameters, including accuracy, recall, and ROUGE score.

[0157] Based on monitoring tools and preset thresholds, the system operation status of the large model is monitored, and if an anomaly is detected, an alarm is triggered.

[0158] Establish a log system to record historical operations and events.

[0159] The work order processing device based on a large model provided in this embodiment can execute the methods provided in the above-described method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0160] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0161] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0162] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0163] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0164] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0165] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0166] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0167] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0168] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0169] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0170] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0173] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0175] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A large model-based work order processing method, characterized in that, The method comprises the following steps: convert the call voice corresponding to the to-be-processed work order into to-be-processed text; construct a prompt word, the prompt word being used to instruct a large model to perform work order processing on the to-be-processed text, the work order processing comprising at least one of the following: element extraction, work order classification, work order summary; call the large model, input the prompt word into the large model, so that the large model performs corresponding processing on the to-be-processed text based on the prompt word, and obtains a work order processing result.

2. The method of claim 1, wherein, The method further comprises the following steps: in an experimental environment, pre-training and optimizing the large model according to training data, evaluating the optimized large model, and importing the model parameters of the large model that passes the evaluation into a test environment; in the test environment, constructing an inference image through a continuous integration and continuous delivery process, evaluating the model effect and generating content review of the model, and pushing the large model that passes the review to a production environment; in the production environment, packaging the model inference image through the continuous integration and continuous delivery process, and performing release, inference and monitoring of the model.

3. The method of claim 2, wherein, The pre-training and optimizing the large model according to the training data in the experimental environment specifically comprises the following steps: obtain training data, the training data being obtained based on work order data generated by a production system and work order processing results output by a large model in the past, the training data comprising work order types and work order summaries; convert call samples in the training data into training text, call the large model and instruct the large model to perform work order processing through a prompt word; wherein for work order processing of the type of element extraction, the corresponding prompt word is used to instruct the large model to pay attention to the element type; for work order processing of the type of work order classification, the corresponding prompt word is used to instruct the large model to define and describe the categories of each classification; optimize the large model according to the processing result output by the large model based on the training text.

4. The method of claim 3, wherein, The optimizing the large model according to the processing result output by the large model based on the training text comprises the following steps: for work order classification of a preset complex business, optimize the large model based on historical work order classification data recorded by an agent; for work order summary, optimize the large model based on historical work order summary data recorded by an agent.

5. The method of claim 3, wherein, The optimizing the large model comprises the following steps: based on the pre-trained large model, use a low-rank adaptive fine-tuning algorithm according to a defined loss function to optimize the large model.

6. The method according to any one of claims 2-5, characterized in that, The monitoring the model comprises the following steps: evaluate the performance of the large model according to performance parameters of the large model, the performance parameters comprising accuracy, recall rate and ROUGE score; monitor the system running state of the large model based on a monitoring tool and a preset threshold value, and perform alarm if an abnormality is found in the monitoring; establish a log record, the log record being used to record historical operations and events.

7. A large model-based work order processing apparatus, characterized by, The device comprises: a conversion module, configured to convert call voice corresponding to a to-be-processed work order into to-be-processed text; a construction module, configured to construct a prompt word, the prompt word being used to instruct a large model to perform work order processing on the to-be-processed text, the work order processing comprising at least one of the following: element extraction, work order classification, work order summary; The calling module is configured to call the large model, input the prompt word into the large model, and enable the large model to perform corresponding processing on the to-be-processed text based on the prompt word to obtain a work order processing result.

8. An electronic device, comprising: The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-6. The memory stores computer execution instructions. The processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1-6. The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1-6.

10. A computer program product, characterised in that, ​

Images