Response statement generation method and related device

Abstract

The invention provides a response statement generation method and a related device. The method comprises the steps of performing field identification on a question statement to obtain a service field of the question statement; determining a first service type to which the question statement belongs from a plurality of service types in the service field; and based on the response knowledge corresponding to the first service type, performing response prediction on the question statement to obtain a response statement of the question statement. Through the application, the accuracy of the response statement can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating a response statement and a related device. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0003] In the related art, the generation of response sentences is obtained by directly predicting the response to the question sentence as a whole. Since the business fields and business types of the question sentence are complex and diverse, the accuracy of the response prediction will decrease, resulting in the low accuracy of the response sentence obtained by the response prediction. Summary of the invention

[0004] The embodiments of the present application provide a method for generating a response statement and a related device, which can effectively improve the accuracy of the response statement.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for generating a response statement, including:

[0007] Performing domain identification on the question statement to obtain the business domain of the question statement;

[0008] Determine a first business type to which the question statement belongs from a plurality of business types in the business field;

[0009] Based on the response knowledge corresponding to the first business type, a response prediction is performed on the question statement to obtain a response statement for the question statement.

[0010] The present application provides a device for generating a response statement, including:

[0011] An identification module, used to perform domain identification on the question statement to obtain the business domain of the question statement;

[0012] The identification module is further used to determine the first business type to which the question statement belongs from the multiple business types in the business field;

[0013] A prediction module is used to predict the answer to the question statement based on the answer knowledge corresponding to the first business type, so as to obtain an answer statement for the question statement.

[0014] An embodiment of the present application provides an electronic device, including:

[0015] A memory for storing computer executable instructions or computer programs;

[0016] The processor is used to implement the method for generating a response statement provided in an embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the method for generating a response statement provided in the embodiment of the present application.

[0018] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the method for generating a response statement described in the embodiment of the present application.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] Even for the same question statement in different business fields, the answers generated are different. Different business fields usually include multiple business types. For the same question statement, the answer statements under different business types are also different. In view of this, when generating an answer statement for a question statement, the present application first obtains the business domain of the question statement by performing domain identification on the question statement, and then determines the first business type to which the question statement belongs. Since question statements of different business types are based on different answer knowledge when predicting answers, the answer prediction of the question statement is performed based on the answer knowledge corresponding to the first business type to obtain the answer statement of the question statement. In this way, by performing domain identification on the question statement and business category identification under the corresponding business field, an answer statement is generated for the question statement based on the identification result, which overcomes the influence of different business fields and business categories on the generated answer statement and improves the accuracy of the generated answer statement. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the architecture of a system for generating a response statement provided in an embodiment of the present application;

[0022] Figure 2 It is a structural diagram of an electronic device for generating a response statement provided in an embodiment of the present application;

[0023] Figure 3 It is a flowchart of a method for generating a response statement provided in an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the principle of the method for generating a response statement provided in an embodiment of the present application;

[0025] Figure 5 It is a structural diagram of the business model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0027] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0031] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed may be executed.

[0032] 2) Artificial Intelligence (AI): It is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] 3) Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by induction. The pre-trained model is the latest development in deep learning, which integrates the above technologies.

[0034] 4) Natural Language Processing (NLP): It is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not a general study of natural language, but the development of computer systems that can effectively realize natural language communication, especially the software systems therein. Therefore, it is a part of computer science. Natural language processing is mainly used in machine translation, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition and other aspects.

[0035] 5) Large Language Model (LLM): It is a machine learning model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data. They are usually based on deep learning architectures. Large language models are also called large language models or large models. Large language models refer to deep learning models trained with large amounts of text data. They contain billions or even larger parameters and can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities. Large language models have the following characteristics:

[0036] Learning ability: Through training on massive amounts of text data, large language models can learn a wealth of language knowledge and expressions, including grammar, semantics, and common expression habits.

[0037] Pattern recognition: Large language models can recognize common text patterns and semantic associations, such as the co-occurrence relationship between words, the logical structure and semantic roles of sentences, etc.

[0038] Contextual understanding: Large language models can capture contextual information in text, understand the impact of previous text on subsequent text, and generate appropriate responses based on the context.

[0039] Generation capability: The large language model can generate relevant natural language text based on the input information, including answering questions, generating articles, and conducting conversations.

[0040] Resolving ambiguity: Despite the polysemy and ambiguity of language, large language models resolve ambiguity through contextual information and language regularities, providing more accurate and appropriate text generation or understanding.

[0041] The application scenarios of large language models are very broad. They can be applied to intelligent customer service, intelligent question and answer, natural language generation, advertising recommendation, games and other fields. They can improve the efficiency and accuracy of human-computer interaction and enhance user experience.

[0042] In the related art, the generation of response sentences is obtained by directly predicting the response to the question sentence as a whole. Since the business fields and business types of the question sentence are complex and diverse, the accuracy of the response prediction will be reduced, resulting in the low accuracy of the response sentence obtained by the response prediction.

[0043] The embodiments of the present application provide a method for generating a response statement and a related device, which can effectively improve the accuracy of the response statement. The following describes an exemplary application of the system for generating a response statement provided in the embodiments of the present application.

[0044] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the response statement generation system 100 provided in an embodiment of the present application. The terminal (terminal 400 is shown as an example) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0045] The terminal 400 is used for the user to use the client 410 and display the test script on the graphical interface 410-1 (the graphical interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0046] In some embodiments, the server 200 may be an independent physical server, or a server cluster or business system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto. The electronic device provided in the embodiment of the present application may be implemented as a terminal or as a server. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0047] In some embodiments, the server 200 performs domain identification on the question statement to obtain a business domain, determines a first business type from multiple business types in the business domain, and sends the first business type to the terminal 400. The terminal 400 predicts an answer to the question statement based on the answer knowledge corresponding to the first business type to obtain an answer statement.

[0048] In other embodiments, the terminal 400 performs domain identification on the question statement to obtain a business domain, determines a first business type from multiple business types in the business domain, and sends the first business type to the server 200. The server 200 predicts an answer to the question statement based on the answer knowledge corresponding to the first business type to obtain an answer statement.

[0049] In other embodiments, the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0050] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0051] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 for generating a response statement provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0052] The processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0053] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 430.

[0054] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0055] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0056] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0057] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0058] In some embodiments, the apparatus for generating a response statement provided in the embodiments of the present application may be implemented in software. Figure 2 The generation device 455 of the response statement stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, including the following software modules: an identification module 4551, a prediction module 4552, and these modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0059] In other embodiments, the response statement generation device provided in the embodiment of the present application can be implemented in hardware. As an example, the response statement generation device provided in the embodiment of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the response statement generation method provided in the embodiment of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASIC, Application Specific Integrated Ci rcuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field Programmable Gate Array) or other electronic components.

[0060] In some embodiments, the terminal or server can implement the method for generating a response statement provided in an embodiment of the present application by running a computer program or a computer executable instruction. For example, the computer program can be a native program (e.g., a dedicated response statement generation program) or a software module in an operating system, for example, a response statement generation module that can be embedded in any program (such as an instant messaging client, an album program, an electronic map client, a navigation client); for example, it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0061] The method for generating a response statement provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0062] See also Figure 3 , Figure 3 is a flow chart of the method for generating a response statement provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 103 are shown for illustration. The method for generating a response statement provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will be described using the server alone as an example.

[0063] In step 101, domain identification is performed on the question statement to obtain the business domain of the question statement.

[0064] In some embodiments, the above-mentioned domain identification refers to using natural language processing (NLP) techniques to analyze and identify the question statement to be answered in order to determine the specific domain it belongs to. It usually involves extracting and analyzing keywords, phrases, or context information in the question to identify the specific business domain of the question. Specifically, preprocess the question text, including removing irrelevant punctuation marks, special characters, converting the text to a unified format (such as converting all to lowercase), and removing stop words (such as meaningless words like "de", "he", etc.). Segment the preprocessed text to extract keywords and phrases. Chinese word segmentation usually needs to借助 natural language processing tools such as jieba, HanLP, etc. Perform word frequency statistics on the segmented keywords and phrases to find the words with higher frequencies, which may be keywords related to the business domain. Perform part-of-speech tagging on the segmentation results to distinguish different parts of speech such as nouns, verbs, adjectives, etc. This helps to further analyze the semantics of the keywords. According to the word frequency and part of speech, filter out the keywords with higher relevance to the business domain. A certain threshold can be set to only retain the words with frequencies higher than the threshold. Construct a co-occurrence matrix from the filtered keywords to analyze the correlation between the keywords. The co-occurrence matrix can help us understand the internal relationship between the keywords and then infer the business domain. In addition to keywords and phrases, context information is also an important basis for identifying the business domain. The sentences, paragraphs, or the entire document in the question can be analyzed to understand the topics and backgrounds involved. Perform comprehensive analysis on keywords, phrases, context information, etc., and combine domain knowledge to judge the business domain of the question. Verify the analysis results, and the identified business domain can be confirmed by consulting relevant materials or professionals. Thus, domain identification helps to provide more accurate and relevant answers for users.

[0065] In some embodiments, the business domain refers to a series of related activities, processes, and functions involved in the operation of an organization or enterprise. These activities usually revolve around a specific industry or market, such as finance, healthcare, education, tourism, etc. The business domain includes various business processes, resource allocations, organizational structures, etc. necessary for an enterprise to achieve its business goals.

[0066] As an example, the business domain of the above-mentioned question statement can be the financial domain, and the business domain of the above-mentioned question statement can also be a non-financial domain.

[0067] As an example, domain identification of a question statement usually means determining the professional or industry category to which the question belongs. For example, if the question is "How to increase crop yields?" then domain identification may determine that this question belongs to the field of "agriculture" or "agricultural technology." Specifically for this example, it can be further analyzed that this question may involve multiple sub-fields such as botany, soil science, meteorology, etc. For example, for the question "How to increase crop yields?", through the above steps, the following fields may be identified: Agricultural technology: involving new planting technologies, agricultural machinery, smart agriculture, etc. Plant nutrition: involving soil fertility, fertilizer use, nutrient regulation, etc. Pest and disease control: involving the identification of pests and diseases, prevention and control methods, biological pesticides, etc.

[0068] In some embodiments, the above-mentioned domain identification of the question statement to obtain the business domain of the question statement can be achieved in the following way: domain identification of the question statement is performed through a large language model to obtain candidate business domains of the question statement; the candidate business domains are verified, and when the verification passes, the candidate business domain is determined as the business domain of the question statement.

[0069] In some embodiments, a large language model (LLM) is a machine learning model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters, which help them learn complex patterns in language data and are usually based on deep learning architectures. Large language models are also called large language models or large models. Large language models refer to deep learning models trained with large amounts of text data, containing billions or even larger parameters, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities.

[0070] The application scenarios of large language models are very broad. They can be applied to intelligent customer service, intelligent question and answer, natural language generation, advertising recommendation, games and other fields. They can improve the efficiency and accuracy of human-computer interaction and enhance user experience.

[0071] In different application scenarios, the large language model is usually trained with a large amount of data from the corresponding application scenarios. In this application, the large language model can more accurately identify the areas involved in the problem after training, thereby improving the accuracy and professionalism of subsequent processing.

[0072] In some embodiments, the training process of a large language model is described below. First, data collection and preprocessing are performed. Input data: The amount of input data, usually tens of GB to TB of text data. Features: Text data includes texts in multiple languages ​​and different fields (such as news, novels, conversations, etc.), as well as language patterns of different styles and structures. The output data can be cleaned text data, which usually includes the removal of irrelevant characters, HTML tags, stop words, etc. Word segmentation or tokenization of text data, decomposing sentences into words or subword units. Relevance: The preprocessing step converts the original text into a format suitable for language model training, improving the model's ability to learn and generate quality text. For model architecture design, the input data does not directly use data in the design phase, but is designed based on data features and model objectives. Output data: The designed model architecture, such as the number of layers of the Transformer, the number of hidden units, the type of attention mechanism, etc. Relevance: The architectural design of the model determines how to process the input text data and affects the performance of the model and the quality of the output. Then pre-training is performed, and the input data can be unlabeled large-scale text data. Quantity Depending on the model size and pre-training goals, a vocabulary of billions to trillions may be required. Features Diverse language usage and patterns, including contextual information. For output data, model parameters, which represent the language patterns and statistical regularities of text data. Relevance: The model updates parameters by learning from input data in order to capture the deep structure of the language. Then perform model optimization and parameter adjustment. Input data: gradient information, calculated based on the difference between model predictions and actual outputs. Output data: updated model parameters. Relevance: Adjust parameters through optimization algorithms to reduce the error between model predictions and actual data. Then perform validation and fine-tuning. Input data: validation set, which contains labeled data or data with reference answers. Quantity: Usually much smaller than training data, but sufficient to represent the data distribution. Features: Should reflect the real-world scenarios where the model will be applied. Output data: performance indicators of the model on the validation set, such as loss, accuracy, etc. Fine-tuned model parameters. Relevance: Evaluate model performance through the validation set, and adjust model parameters based on the evaluation results to improve the effectiveness of the model. Then the model is evaluated. Input data: test set, which is independent of the training and validation data sets. Output data: performance evaluation results of the model on the test set. Relevance: the test set is used to evaluate the generalization ability of the model and ensure the performance of the model in practical applications. Then the model is stored and deployed, and the model parameters are trained. Output data: stored model files and model instances deployed in the production environment. Relevance: the purpose of storing and deploying the model is to use the model for language tasks in practical applications. Then subsequent iterations are carried out. Input data: user feedback and new text data. Output data: continuously updated and optimized model.Relevance: Improve the model based on user feedback and new data to ensure that the model can adapt to the changing language environment and usage scenarios.

[0073] In some embodiments, the training process of a large language model is described below. First, the purpose and goal of the model are determined, such as text generation, text understanding, dialogue system, etc. A suitable model architecture is selected, such as Transformer, BERT, etc. The input data can be a large amount of text data collected, and the source can be books, websites, news articles, social media, etc. Then the data is preprocessed, the data is cleaned, and noise is removed, such as HTML tags, non-text content, etc.; word segmentation, encoding, standardization, etc. The text data is word segmented and encoded, and converted into a format that the model can understand. If the model requires supervised learning, the data is annotated, such as sentiment analysis, entity recognition, etc. The model is trained using the preprocessed text data. Large-scale unsupervised pre-training is usually used, followed by fine-tuning on specific tasks. During the iterative training process, the model parameters are adjusted through the back propagation algorithm. The model is evaluated using a validation set to check the performance of the model. Model parameters such as learning rate, batch size, etc. are adjusted according to the evaluation results, and multiple iterations may be required to optimize the model.

[0074] In some embodiments, the data type of the input data is usually text data, including books, articles, web page content, etc. The amount of data is very large, usually calculated in billions or more characters. The characteristics of the data can be the characteristics of text data including words, sentences, paragraphs, etc., as well as possible grammatical and semantic information. The type of output data can be based on the design of the model, and the output can be generated text, classification labels, probability distributions, etc. The characteristics of the output data can be that the output text may have a specific language style, theme or emotional tendency. The text content in the input data is the basis for the model to generate the output text. The model predicts or generates the corresponding output by learning the pattern, structure, semantics and other information of the input data. For example, a model that generates text predicts the next word or sentence that may appear based on the context of the input text. Large language models usually require a lot of data to train because they need to learn the extensive patterns and structures of the language. The number is usually measured in GB or TB, involving tens to hundreds of billions of characters. The characteristics of the input data include the length of the text, vocabulary richness, grammatical structure, emotional tendency, etc. It may also include metadata such as author, publication date, source, etc., which helps the model understand the context and background.

[0075] In some embodiments, after training, the large language model can more accurately identify the field involved in the problem. The following continues to explain the training process of the large language model in combination with the application scenario of field identification. First, determine the specific application scenario such as the model used to identify the problem field. Select a suitable model architecture, such as Transformer, and design the number of layers and hidden units of the model. Collect text data from different fields, such as science and technology, medicine, law, economy, etc. Clean the data and remove irrelevant information, such as HTML tags, advertisements, etc. Word segmentation, decomposing the text into words or subword units. Standardize the text, such as unifying uppercase and lowercase, removing special characters, etc. If supervised learning is used, annotate the data set and mark the field label to which the text belongs. Use unlabeled data for unsupervised pre-training to learn the general representation of language. Use a predefined loss function, such as language modeling loss, for optimization. Based on pre-training, use the labeled data for fine-tuning to optimize the model's ability to identify the problem field. Use the validation set and test set to evaluate the performance of the model. Adjust the model parameters or training process to improve the accuracy and professionalism of the model. The type of input data can be text data, including books, articles, web page content, etc. The amount of data can range from billions to hundreds of billions of characters, depending on the scale of the model and the application scenario. The characteristics of the data include, but are not limited to, vocabulary, grammatical structure, semantic information, text length, domain-specific terms, etc. The type of output data can be a text classification label or a domain label, indicating which specific domain the input text belongs to. The characteristics of the output data are usually discrete labels, such as "technology," "medicine," "law," etc. The correlation between the input data and the output data, the text content and structure in the input data provide the information needed for the model to identify the problem domain. The large language model predicts the domain label in the output data by learning the domain-specific terms, phrases, and context in the input data. After training, the large language model can identify the problem domain. By analyzing the input problem text, the model can more accurately identify the domain involved. Improve processing accuracy. After identifying the domain, the large language model can adjust its response strategy for the specific domain and provide more accurate and professional follow-up processing, such as answers to professional terms, retrieval of related documents, etc.

[0076] As an example, suppose there is a question statement to be answered: "How do I apply for a credit card?" Domain identification through the large language model: The large language model analyzes the keywords in the question, such as "application", "credit card", etc. Based on these keywords, the large language model identifies possible candidate business domains, such as: financial services, banking services, personal finance, etc. Verification of candidate business domains: The candidate business domains identified above are verified, which may be done in the following ways: Check whether there are relevant business rules or processes related to "applying for a credit card". Query whether there are records of the keyword "credit card application" in the database. Compare with business experts or knowledge bases to confirm whether it matches with existing business domains. If the verification result shows that the "financial services" domain is highly related to the question statement and contains the business process and rules for credit card application, then the verification result will indicate that the candidate business domain has passed the verification. Determine the business domain: The system determines "financial services" as the business domain and continues to process the question on this basis, such as providing specific steps and guidelines for credit card application. Through this process, a question statement to be answered "How do I apply for a credit card?" is successfully associated with the "financial services" domain, and the question is further processed and answered.

[0077] In this way, by using a large language model to automatically identify domains, the time and labor costs of manual judgment can be greatly reduced, and the efficiency of problem handling can be improved. The verification step ensures that the business domain determined is correct, allowing the model to flexibly respond to various complex scenarios. It helps to avoid assigning problems to the wrong business domain, reducing the error rate and the complexity of subsequent processing. Accurate business domain identification can provide more professional and relevant answers, thereby improving the user's interactive experience.

[0078] In some embodiments, the above-mentioned verification of the candidate business domain can be achieved in the following manner: determine the reference business domain of the question statement from preset business domains; if the reference business domain is the same as the candidate business domain, determine that the verification is passed; if the reference business domain is different from the candidate business domain, determine that the verification is not passed.

[0079] In some embodiments, the above-mentioned determination of the reference business domain of the question statement from the preset business domain can be achieved in the following manner: for each preset business domain, determining the similarity between the statement set of the preset business domain and the question statement; and determining the preset business domain that meets the similarity condition as the reference business domain of the question statement.

[0080] In some embodiments, the preset business field whose similarity satisfies the condition may refer to the preset business field with the greatest similarity, or may refer to the preset business field with the similarity greater than a similarity threshold.

[0081] As an example, suppose there are several preset business domains and their corresponding statement sets: Business domain: e-commerce: statement set: {"how to shop online", "how to pay for an order", "how to cancel an order", etc.}; Business domain: travel reservation: statement set: {"how to book a hotel", "how to cancel a hotel reservation", "how to book an air ticket", etc.}, and there is a question statement to be answered: "I want to cancel my hotel reservation. Follow the steps below: Determine similarity: Compare the similarity between the question statement "I want to cancel my hotel reservation" and the statement sets of each business domain. Similarity with the e-commerce domain: low, because the question involves hotel reservation, not shopping. Similarity with the travel reservation domain: high, because the question directly mentions "hotel reservation." Determine the reference business domain: Since the question statement has the greatest similarity with the statement set of the travel reservation domain, the travel reservation domain is determined as the reference business domain. Perform domain verification: Assuming that the candidate business domain is "e-commerce", since the reference business domain (travel reservation) is different from the candidate business domain (e-commerce), it is determined that "the candidate business domain has not passed verification."

[0082] As an example, suppose there is a question sentence: "I want to know the latest stock market situation," which can be compared with a set of sentences in preset business fields, such as "stock market," "finance," "investment," etc. By calculating the similarity, it is found that the "stock market" business field has the highest similarity, so "stock market" is used as the reference business field for the question sentence. If we have a candidate business field: "finance," we compare it with the reference business field "stock market." Since they are the same, it can be determined that the candidate business field "finance" has passed the verification. This means that our system can accurately identify and verify business fields, improving the overall verification effect.

[0083] In this way, for each set of statements in a preset business field, the similarity between the statement set and the question statement is determined; the preset business field with the greatest similarity is determined as the reference business field of the question statement; if the reference business field is the same as the candidate business field, the candidate business field is determined to have passed the verification; if the reference business field is different from the candidate business field, the candidate business field is determined to have failed the verification, thereby helping to improve the accuracy and efficiency of business field verification. By calculating the similarity between the question statement and the set of statements in the preset business field, the business field to which the question belongs can be accurately determined, which can not only reduce the error of manual judgment, but also save time and resources.

[0084] In some embodiments, the statement set includes multiple sub-sentences, and the determining the similarity between the statement set and the question statement can be achieved in the following manner: performing feature extraction on the question statement to obtain the feature of the question statement; performing feature extraction on each sub-sentence to obtain the feature of the sub-sentence, and determining the similarity of the sub-sentences based on the feature distance between the feature of the sub-sentence and the feature of the question statement; performing a preset operation on the similarity of each sub-sentence to obtain the similarity.

[0085] In some embodiments, the above-mentioned performing a preset operation on the similarity of each sub-sentence to obtain the similarity between the sentence set of the preset business field and the question sentence can be achieved as follows: summing the similarities of each sub-sentence to obtain the similarity between the sentence set of the preset business field and the question sentence.

[0086] In some embodiments, the above-mentioned determination of the similarity of the sub-sentences based on the feature distance between the features of the sub-sentences and the features of the question sentences can be achieved in the following way: the inverse of the feature distance between the features of the sub-sentences and the features of the question sentences is determined as the similarity of the sub-sentences.

[0087] In some embodiments, the above-mentioned feature distance generally refers to a metric used to describe the difference or similarity between objects in the fields of data analysis, pattern recognition or image processing, and can be calculated by various algorithms to reflect the degree of separation of two objects in a certain feature space. The feature distance is not a single value, but is based on the selected features and distance metrics, and can be Euclidean distance, Manhattan distance, cosine similarity, etc.

[0088] As an example, the expression of the similarity above can be:

[0089]

[0090] Among them, T is used to indicate the similarity, t i It is used to indicate the similarity of the ith sub-sentence in the sentence set, t i is any one of the M sub-statements, M is used to indicate the number of sub-statements in the statement set, 1≤i≤M, M is a positive integer greater than 1, 0≤t i ≤1.

[0091] In this way, through feature extraction, the core content of the question and the user's true intention can be understood more accurately, thereby providing more accurate answers. Calculation based on feature similarity can help determine which sub-sentences are most relevant to the question, so that these sub-sentences can be given priority when generating answers, improving the relevance and quality of the answers. By quantifying the similarity between sub-sentences and questions, information directly related to the question can be more systematically generated to avoid providing irrelevant or incorrect information. Feature extraction of question statements can identify specific details of the question, such as keywords and syntactic structure, which helps to deeply understand the question. Comparison between the features of sub-sentences and the features of question statements allows the system to make accurate matches, thereby providing more accurate answers. Similarity calculation can help quickly locate the sub-sentences most relevant to the question and improve the efficiency of information retrieval.

[0092] In some embodiments, after the above-mentioned verification of the candidate business domain, the following processing can also be performed: if the verification fails, the recognition accuracy of the large language model in the candidate business domain is obtained; if the recognition accuracy is greater than a threshold, the candidate business domain is determined as the business domain of the question statement; if the recognition accuracy is less than or equal to the threshold, the reference business domain is determined as the business domain of the question statement.

[0093] In some embodiments, the recognition accuracy of the large language model refers to the percentage of information correctly recognized and classified by the model in a specific task or field. When the candidate business field is not verified, the recognition accuracy of the large language model in the candidate business field can be achieved by comparing the model's predicted results with the actual results (usually the labels of the test data set).

[0094] In some embodiments, if the recognition accuracy is greater than a threshold, it means that the large language model can accurately recognize the candidate business domain, and the candidate business domain recognized by the large language model can be determined as the business domain.

[0095] In some embodiments, if the recognition accuracy is less than or equal to the threshold, it means that the large language model cannot accurately recognize the candidate business domain, and then the reference business domain that is not recognized by the large language model can be determined as the business domain.

[0096] In this way, the effectiveness and reliability of the large language model in actual business applications are ensured through the verification and evaluation mechanism of the system. First, through verification to determine whether the candidate business field meets the requirements, it is helpful to promptly discover the performance problems that may exist in the model in a specific field. Then, when the candidate business field fails to pass the verification, the recognition accuracy of the model is calculated. This practice can quantify the performance of the model in this field and provide an objective basis for subsequent decision-making. Most importantly, by comparing the recognition accuracy with the pre-set threshold, it can be decided whether to identify the candidate business field as a business field, or to select a reference business field, which ensures that the model will only be put into use when the performance meets the established standards, thereby avoiding business risks that may be caused by insufficient model performance. This helps to improve the accuracy of large language model deployment, optimize business processes, improve customer satisfaction, and provide scientific support for corporate decision-making.

[0097] In some embodiments, the above-mentioned verification of the candidate business field can be achieved in the following manner: performing preset information identification on the name of the candidate business field to obtain an identification result of the candidate business field; if the identification result indicates that the name of the candidate business field includes the preset information, it is determined that the candidate business field has not passed the verification; if the identification result indicates that the name of the candidate business field does not include the preset information, it is determined that the candidate business field has passed the verification.

[0098] In some embodiments, the above-mentioned identification of preset information on the name of the candidate business field to obtain the identification result of the candidate business field can be achieved in the following way: compare the preset information with the name of the candidate business field, if the name of the candidate business field includes the preset information, then determine the identification result as the name of the candidate business field does not include the preset information.

[0099] In this way, by identifying the preset information in the names of candidate business areas, potential sensitive data, such as personal information, confidential information, etc., can be effectively discovered, which is crucial for protecting user privacy and complying with data security regulations. When the recognition result indicates that the content contains preset information, it is immediately judged as failed, which helps prevent the leakage of preset information and maintain the compliance and reputation of the enterprise. On the contrary, if the recognition result shows that there is no preset information in the content, the verification result is passed, which indicates that the content in this field meets the security standards and can proceed with subsequent business processes. It not only improves the security of information processing, but also ensures that the enterprise can efficiently expand and operate its business under the premise of complying with laws and regulations.

[0100] In some embodiments, there are multiple business types under the business field, and different question statements of different business types are based on different answer knowledge when making answer predictions.

[0101] In step 102, a first business type to which the question statement belongs is determined from a plurality of business types in the business field.

[0102] In some embodiments, there are multiple business types under the business field, and different question statements of different business types are based on different answer knowledge when making answer predictions.

[0103] In some embodiments, the above step 102 can be implemented as follows: predict the question statement through a large language model to obtain the probability that the question statement belongs to each of the business types; determine the business type with the highest probability as the candidate business type to which the question statement belongs; verify the candidate business type, and if the verification passes, determine the candidate business type as the first business type to which the question statement belongs.

[0104] As an example, the following is a specific example of how to use a large language model to predict a question statement and determine the business type to which it belongs: Assume that there is a question statement to be answered: "How to debug memory leaks in Java programs?"; Predict through a large language model, input the question statement: Pass the question statement "How to debug memory leaks in Java programs?" as input to the large language model. The large language model will use its trained algorithm to predict the business type to which the question statement may belong. For example, the large language model may output the following probabilities: technical support: 80%; programming problems: 60%; software usage: 30%; other: 10%. Determine the candidate business type and select the type with the highest probability. According to the probability output by the model, the probability of technical support is the highest, which is 80%. Therefore, technical support is determined as the candidate business type of the question statement. Verify the candidate business type. Verify the candidate business type through a series of verification steps, such as using a set of predefined verification questions or standards to test the performance of the model in the field of technical support. Get the verification result. Assume that the verification result shows that the performance of the model in the field of technical support meets the expected standards. The first business type is determined and verified. Since the candidate business type technical support has passed the verification, it is determined as the first business type to which the example question statement belongs.

[0105] In this way, the possible business types and probabilities of the question statements are automatically determined, and the efficiency and accuracy of classification enable the questions to be quickly routed to the correct business areas. Secondly, by selecting the business type with the highest probability as the candidate type, the process optimizes resource allocation, making subsequent verification and answering more focused and efficient. Finally, the step of verifying the candidate business type ensures the reliability of the problem classification. Only the verified business type will be determined as the final first business type, which greatly reduces the risk of misclassification and improves the accuracy of overall business processing and user satisfaction. Not only does it optimize the problem handling process, but it also improves the user experience and the quality of enterprise business processing.

[0106] In some embodiments, the above-mentioned verification of the candidate business type to obtain the type verification result can be achieved in the following manner: performing preset information identification on the name of the candidate business type to obtain a type identification result of the candidate business type; if the type identification result indicates that the name of the candidate business type includes preset information, it is determined that the candidate business field has not passed the verification; if the type identification result indicates that the name of the candidate business type does not include the preset information, it is determined that the candidate business field has passed the verification.

[0107] In some embodiments, the above-mentioned identification of preset information on the name of the candidate business type to obtain the type identification result of the candidate business type can be achieved in the following way: comparing the preset information with the name of the candidate business type, if the name of the candidate business type includes the preset information, then determining the type identification result as the name of the candidate business type does not include the preset information.

[0108] In this way, by identifying the preset information in the names of candidate business types, it is ensured that potential sensitive data, such as personal information, financial data, etc., can be discovered and processed in a timely manner, thereby effectively preventing the risk of data leakage and privacy infringement. Secondly, when the identification result indicates the presence of preset information, the type verification result is determined to be failed, which helps prevent business types that do not meet security standards from being adopted and maintains compliance and reputation. Conversely, if the identification result shows that there is no preset information in the content, the verification result is passed, which not only confirms the applicability of the business type, but also promotes the smooth progress of business processes. It enhances the ability of information security management, reduces operational risks, and improves business processing efficiency and user trust.

[0109] In step 103, based on the answer knowledge corresponding to the first business type, an answer prediction is performed on the question sentence to obtain an answer sentence for the question sentence.

[0110] As an example, assume that the first business type is "Technical Support," question statement: "How to debug memory leaks in Java programs?"; Answer knowledge base: For the "Technical Support" business type, there is a pre-prepared knowledge base that contains a series of common questions and answers about Java programming and debugging. For example: "How to locate memory leaks in Java programs?"; Answer: "You can use Java's VisualVM tool to monitor memory usage and identify possible memory leaks." Answer prediction process: Understand the question: The large language model first understands the question statement and determines that it belongs to the "Technical Support" business type. Retrieve answer knowledge: Retrieve answer knowledge related to the question in the "Technical Support" knowledge base, which can be achieved through keyword search, semantic similarity matching, etc. Answer prediction: Based on the similarity between the retrieved answer knowledge and the question statement, the large language model predicts the most appropriate answer statement. For example, the model may identify the following answer statement: "You can use Java's VisualVM tool to monitor memory usage and identify possible memory leaks." Generate answer: The large language model generates an answer statement as an answer to the question statement. Answer statement: You can use Java's VisualVM tool to monitor memory usage and identify possible memory leaks. This tool can help you view the memory distribution of your program and find out which objects are taking up too much memory. Once you identify a memory leak, you can check the related code and make corrections.

[0111] In some embodiments, the first business type mentioned above includes multiple sub-business types, and question statements of different sub-business types are based on different answer knowledge when making answer predictions; the above step 103 can be implemented in the following way: from the multiple sub-business types of the first business type, determine the target sub-business type to which the question statement belongs; based on the answer knowledge corresponding to the target sub-business type, predict the answer to the question statement to obtain the answer statement.

[0112] As an example, assume that the first business type is "technical support," which contains the following sub-business types: Sub-business type A: programming problem; Sub-business type B: software usage problem; Sub-business type C: hardware failure; Question statement: "My Java program runs slowly, what should I do?" Determine the target sub-business type: Understand the problem: The large language model first understands the question statement and analyzes its content. Sub-business type attribution: After the model analyzes the question statement, it compares it with the key features of each sub-business type to determine the most appropriate sub-business type. For example, the model may derive the following attribution probabilities: Sub-business type A: programming problem 70%; Sub-business type B: software usage problem 20%; Sub-business type C: hardware failure 10%; Determine the target sub-business type: According to the attribution result with the highest probability, sub-business type A (programming problem) is determined as the target sub-business type of the question statement. Response prediction process: Retrieve response knowledge: Retrieve relevant response knowledge in the knowledge base of the "programming problem" sub-business type under "technical support". The knowledge base may contain the following entries: Question: What is the reason why the program runs slowly? ;Answer: The slow running of Java program may be caused by memory leaks, inefficient algorithms, or resource competition. Answer prediction: The large language model predicts the most appropriate answer statement based on the similarity between the retrieved answer knowledge and the question statement. The model may generate the following prediction: Your Java program may be running slowly due to memory leaks or inefficient algorithms. You can first use Java's VisualVM tool to analyze memory usage; Generate answer: The large language model generates the following answer statement as an answer to the question statement: Your Java program may be running slowly due to memory leaks or inefficient algorithms. It is recommended that you use Java's VisualVM tool to analyze memory usage, check for memory leaks, and optimize the algorithm logic in the code.

[0113] In some embodiments, the above-mentioned determination of the target sub-business type to which the question statement belongs from the multiple sub-business types of the first business type can be achieved in the following manner: sub-prediction is performed on the question statement through a category recognition model to obtain the probability that the question statement belongs to each sub-business type; the sub-business type with the largest probability is determined as the candidate sub-business type to which the question statement belongs; the candidate sub-business type is verified to obtain a sub-type verification result, and if the sub-type verification result indicates that the candidate sub-business type has passed the verification, the candidate sub-business type is determined as the target sub-business type to which the question statement belongs.

[0114] As an example, the question statement: A blue screen appears when my computer starts, what should I do? Sub-prediction is performed through the large language model; Input the question statement: Pass the question statement "A blue screen appears when my computer starts, what should I do?" to the large language model. Model prediction: The model predicts the probability of each sub-business type to which the question statement may belong based on the trained algorithm. For example, the model may output the following prediction results: hardware failure: 60%; software conflict: 30%; system settings: 10%. Determine the candidate sub-business type and select the type with the highest probability: According to the output of the model, the probability of hardware failure is the highest, which is 60%. Therefore, hardware failure is determined as the candidate sub-business type of the question statement. Verify the candidate sub-business type and verify the candidate sub-business type through a series of verification steps, such as using a set of verification questions related to hardware failure to test the performance of the model. Obtain the verification result, assuming that the verification result shows that the performance of the model in the hardware failure sub-business field meets the expected standards. Determine the target sub-business type, and through verification, since the candidate sub-business type hardware failure passes the verification, it is determined as the target sub-business type to which the question statement belongs. Response process (additional steps): Retrieve response knowledge: Retrieve relevant response knowledge in the knowledge base of the "hardware" sub-business type. Generate response: Generate response statements based on the retrieved knowledge, for example: A blue screen when the computer starts may be caused by hardware failure. You can first check the computer's hardware connections, such as memory sticks, hard disks, etc., and try to restart the computer.

[0115] In some embodiments, the above-mentioned predicting the answer to the question statement based on the answer knowledge corresponding to the target sub-business type to obtain the answer statement can be achieved in the following way: training a language model based on the answer knowledge corresponding to the target sub-business type to obtain a business model corresponding to the target sub-business type; predicting the answer to the question statement through the business model to obtain the answer statement.

[0116] In some embodiments, the above-mentioned language model can be a large language model. Before predicting the answer to the question statement through the business model to obtain the answer statement, the language model can be pre-trained based on the answer knowledge corresponding to the target sub-business type to obtain the business model corresponding to the target sub-business type. The language model can be trained before predicting the answer to each question statement, or the language model can be trained separately before predicting the answer to each question statement.

[0117] In some embodiments, the above-mentioned training of the large language model based on the answer knowledge corresponding to the target sub-business type to obtain the business model corresponding to the target sub-business type can be achieved in the following manner: collect data related to the target sub-business type from an existing knowledge base, historical question answering records or professional literature. These data should include question statements and corresponding correct answers. Clean the collected data to remove noise and irrelevant information to ensure the quality of the data. Add labels to the training set data, which indicate the sub-business type to which each question statement belongs and the corresponding correct answer. Divide the data set into training sets, validation sets and test sets to facilitate model evaluation and adjustment during training. Select a pre-trained large language model as a basis, such as BERT, GPT3, etc., or use a custom model architecture. Configure the model according to the needs of the target sub-business type, including adjusting parameters such as the model size, learning rate, batch size, etc. Use the training set data to train the large language model. During the training process, the model will learn how to extract features from the question statement and associate these features with the annotated answer knowledge. Evaluate the performance of the model on the validation set, and adjust the model parameters or structure based on the evaluation results to improve the accuracy and generalization ability of the model. Repeat the training and validation process until the performance of the model on the validation set reaches a satisfactory level.

[0118] As an example, see Figure 5 , Figure 5 It is a structural diagram of the business model provided in the embodiment of the present application. The business model provided in the embodiment of the present application includes a feature extraction layer 1 and a response prediction layer 2. The above-mentioned business model is used to predict the response to the question sentence to obtain the response sentence. It can be achieved in the following way: through the feature extraction layer 1, feature extraction is performed on the question sentence to obtain sentence features, and through the response prediction layer 2, based on the sentence features, the response prediction is performed on the question sentence to obtain the response sentence.

[0119] In this way, even for the same question statement, the answers generated in different business fields are different. Different business fields usually include multiple business types. For the same question statement, the answer statements in different business types are also different. In view of this, when generating an answer statement for a question statement, the present application first obtains the business domain of the question statement by performing domain identification on the question statement, and then determines the first business type to which the question statement belongs. Since question statements of different business types are based on different answer knowledge when predicting answers, the answer prediction of the question statement is performed based on the answer knowledge corresponding to the first business type to obtain the answer statement of the question statement. In this way, by performing domain identification on the question statement and business category identification under the corresponding business field, an answer statement is generated for the question statement based on the identification result, which overcomes the influence of different business fields and business categories on the generated answer statement and improves the accuracy of the generated answer statement.

[0120] The following is an explanation of an exemplary application of the embodiments of the present application in an actual financial business application scenario.

[0121] Through the method for generating the response sentence provided in the embodiment of the present application, there is no need for a large amount of manpower to annotate data. Just add a few shot examples in the prompt (for examples, refer to step 222), and you can use the powerful semantic understanding and content generation capabilities of LLM to quickly complete the intent classification judgment. If you need to fine-tune the SFT of LLM, there is no need for a large amount of manpower to do the annotation. You can let LLM generate a large amount of text in batches. After a small amount of manual review, you can quickly fine-tune the training model, reducing a lot of data preprocessing and feature engineering work. The specific step is to give instructions to the big model, so that it can automatically annotate the fine-tuned corpus according to a few shots. This reduces the labor cost of manual annotation, and we only need to quickly review the corpus annotated by the big model. Traditional NLP models can only complete the intent classification tasks in vertical fields, but LLM has been pre-trained with massive general knowledge content, has a strong emergence ability, and the generalization performance of the model is stronger. For example, when you need to identify the intent, you only need to add a few shots in the prompt to increase the scope of application of the model. LLM itself has a strong semantic understanding ability and can understand various problems of customers without regular matching.

[0122] In some embodiments, see Figure 4 , Figure 4It is a schematic diagram of the principle of the method for generating a response statement provided in the embodiment of the present application. The business needs are analyzed and the available LLM is selected as the base large model. The prompt for the first-level intent is designed. The background is a financial investment advisory assistant. The blue font part below clearly states that two types of intent are designed: financial intent and non-financial intent. First, use the excellent language summarization ability of the large model to divide the user's intentions into two categories: financial and non-financial. In the subsequent process, each category has different processing branches according to business rules. Take the prompt of a certain investment advisory assistant as an example: You are an expert in the financial field and are very professional in investment consulting and financial advice. Please make an intent classification answer to the current input content based on your professional knowledge. There are two types of intent in total, namely financial intent and non-financial intent. Keyword matching: Check whether the input content contains financial-related keywords. The keyword matching result determines whether the current input content is financial or non-financial intent.

[0123] The result is required to be returned in json format.

[0124]

[0125] In some embodiments, see Figure 4 , use the above prompt to drive the selected big model, and use the test data to test the effect of the first-level intent classification. In previous project experience, accuracy, recall, and F1SCORE were used for evaluation. Because this innovation is to use a big model for multi-classification prediction, this week is a multi-classification algorithm task, so the effect can be evaluated using traditional supervised machine learning indicators. Each piece of data in the test set has an actual classification label. By comparing the intent predicted by the big model with the actual intent, the classification effect can be evaluated.

[0126] In some embodiments, see Figure 4 If the result is satisfactory, it will jump to the compliance detection and relief. If the result is not satisfactory, it will adjust the prompt, such as adding more examples, or fine-tuning the large model with a small amount of corpus. Use the large model for compliance detection to see whether the content output by the large model contains sensitive content (such as pornography, politics, violation of moral order, etc.). If the detection fails, it will be sent to the large model to re-judge the intent.

[0127] In some embodiments, see Figure 4, set up a secondary / tertiary intention judgment process. After judging the final intention, it will be handed over to the corresponding business system and AI robot to handle the user's problem and handle the business. The above process of the secondary and tertiary intention judgment process is exactly the same, but the detailed intent type is different. Because it involves project details and some preset information, it cannot be fully given. But some examples can be given, such as the first-level intention is divided into two categories: financial / non-financial. In the first-level intention of finance, according to the actual problems of users, the second-level intention is divided into banking / insurance / securities; in the first-level intention of non-finance, according to the actual problems of users, the second-level intention is divided into social / humanities / history / military / sports / chatting; in the second-level intention of banks, according to the actual problems of users, the third-level intention is divided into: deposits / loans / credit cards / interbank business.

[0128] In some embodiments, the input of data annotation and professional algorithm personnel is reduced, and the labor cost is reduced. The specific step is to give instructions to the big model and let it automatically annotate the fine-tuned corpus according to a few shots. This reduces the labor cost of manual annotation, and only requires a quick check of the corpus annotated by the big model. Choosing the SaaS version of the base big model can reduce the computing cost investment. You only need to fine-tune the prompt template and replace the big model to quickly develop a new intent classification model application, reducing the development and iteration cycle.

[0129] The specific example is as follows: For example, the original template only supports two types of intents: You are an expert in the financial field, and you are very professional in investment consulting and financial advice. Please answer the current input content according to your professional knowledge. There are two types of intents in total, namely banking intent and non-banking intent.

[0130] Step 1: Keyword matching: Check whether the input content contains bank-related keywords.

[0131] Step 2: Keyword matching results to determine whether the current input content is banking or non-banking intent.

[0132] The result is required to be returned in json format.

[0133]

[0134] In some embodiments, more intents can be supported by modifying the template. For example, the following modification can support more intents. You are an expert in the financial field and are very professional in investment consulting and financial advice. Please answer the current input content according to your professional knowledge. There are three types of intents in total, namely banking intent, insurance intent and non-bank insurance intent.

[0135] Step 1: Keyword matching: Check whether the input content contains keywords related to banking and insurance.

[0136] Step 2: Keyword matching results to determine whether the current input content is banking, insurance or non-banking insurance intent.

[0137] The result is required to be returned in json format.

[0138]

[0139] It is understandable that in the embodiments of the present application, when data related to question statements is involved and when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0140] The following further describes the exemplary structure of the response statement generating device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules in the response statement generation device 455 stored in the memory 450 may include: an identification module 4551, used to perform domain identification on the question statement to obtain the business domain of the question statement; the identification module 4551, also used to determine the first business type to which the question statement belongs from multiple business types in the business domain; the prediction module 4552, also used to predict the answer to the question statement based on the response knowledge corresponding to the first business type, to obtain the response statement of the question statement.

[0141] In some embodiments, the above-mentioned recognition module is also used to perform domain recognition on the question statement through a large language model to obtain candidate business domains of the question statement; verify the candidate business domains, and if the verification passes, determine the candidate business domain as the business domain of the question statement.

[0142] In some embodiments, the above-mentioned identification module is also used to determine the reference business field of the question statement from the preset business fields; if the reference business field is the same as the candidate business field, it is determined that the verification is passed; if the reference business field is different from the candidate business field, it is determined that the verification is not passed.

[0143] In some embodiments, the above-mentioned identification module is also used to determine, for each preset business field, the similarity between the sentence set of the preset spy field and the question sentence; and determine the preset business field that meets the similarity condition as the reference business field of the question sentence.

[0144] In some embodiments, the above-mentioned recognition module is also used to perform feature extraction on the question sentence to obtain the features of the question sentence; for each sub-sentence, perform feature extraction on the sub-sentence to obtain the features of the sub-sentence, and determine the similarity of the sub-sentences based on the feature distance between the features of the sub-sentences and the features of the question sentence; perform a preset operation on the similarity of each of the sub-sentences to obtain the similarity.

[0145] In some embodiments, the above-mentioned identification module is also used to obtain the recognition accuracy of the domain identification model in the candidate business domain when the verification fails; if the recognition accuracy is greater than a threshold, the candidate business domain is determined as the business domain of the question statement; if the recognition accuracy is less than or equal to the threshold, the reference business domain is determined as the business domain of the question statement.

[0146] In some embodiments, the above-mentioned identification module is also used to perform preset information identification on the name of the candidate business field to obtain an identification result of the candidate business field; if the identification result indicates that the name of the candidate business field includes preset information, it is determined that the candidate business field has not passed the verification; if the identification result indicates that the name of the candidate business field does not include the preset information, it is determined that the candidate business field has passed the verification.

[0147] In some embodiments, the above-mentioned identification module is also used to predict the question statement through a prediction model to obtain the probability that the question statement belongs to each business type; determine the business type with the highest probability as the candidate business type to which the question statement belongs; verify the candidate business type, and if the verification passes, determine the candidate business type as the first business type to which the question statement belongs.

[0148] In some embodiments, the above-mentioned identification module is also used to perform preset information identification on the name of the candidate business type to obtain a type identification result of the candidate business type; if the type identification result indicates that the name of the candidate business type includes preset information, it is determined that the candidate business field has not passed the verification; if the type identification result indicates that the name of the candidate business type does not include the preset information, it is determined that the candidate business field has passed the verification.

[0149] In some embodiments, the first business type includes multiple sub-business types, and question statements of different sub-business types are based on different answer knowledge when making answer predictions; the above-mentioned identification module is also used to determine the target sub-business type to which the question statement belongs from the multiple sub-business types of the first business type; based on the answer knowledge corresponding to the target sub-business type, the answer to the question statement is predicted to obtain the answer statement.

[0150] In some embodiments, the above-mentioned identification module is also used to perform sub-prediction on the question statement through a type identification model to obtain the probability that the question statement belongs to each sub-business type; determine the sub-business type with the largest probability as the candidate sub-business type to which the question statement belongs; verify the candidate sub-business type to obtain a sub-type verification result, and if the sub-type verification result indicates that the candidate sub-business type has passed the verification, then determine the candidate sub-business type as the first business type to which the question statement belongs.

[0151] In some embodiments, the above-mentioned recognition module is also used to train a large language model based on the response knowledge corresponding to the target sub-business type to obtain a business model corresponding to the target sub-business type; through the business model, the answer to the question statement is predicted to obtain the answer statement.

[0152] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the method for generating a response statement described in the embodiment of the present application.

[0153] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the method for generating a response statement provided in the embodiment of the present application, for example, Figure 3 A method for generating a response statement is shown.

[0154] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEP ROM, flash memory, magnetic surface storage, optical disk, or CDROM; or it may be various electronic devices including one or any combination of the above memories.

[0155] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0156] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0157] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0158] In summary, the embodiments of the present application have the following beneficial effects:

[0159] (1) Even for the same question statement, the answers generated are different in different business fields. Different business fields usually include multiple business types. For the same question statement, the answer statements under different business types are also different. In view of this, when generating an answer statement for a question statement, the present application first obtains the business domain of the question statement by performing domain identification on the question statement, and then determines the first business type to which the question statement belongs. Since question statements of different business types are based on different answer knowledge when predicting answers, the answer prediction of the question statement is performed based on the answer knowledge corresponding to the first business type to obtain the answer statement of the question statement. In this way, by performing domain identification on the question statement and business category identification under the corresponding business field, an answer statement is generated for the question statement based on the identification result, thereby overcoming the influence of different business fields and business categories on the generated answer statement and improving the accuracy of the generated answer statement.

[0160] (2) By using a large language model to automatically identify domains, the time and labor costs of manual judgment can be greatly reduced, and the efficiency of problem handling can be improved. The verification step ensures that the business domain identified is correct, allowing the model to flexibly respond to various complex scenarios. This helps avoid assigning problems to the wrong business domain, reducing the error rate and complexity of subsequent processing. Accurate business domain identification can provide more professional and relevant answers, thereby improving the user's interactive experience.

[0161] (3) By using a large language model to automatically identify the domain, the time and labor cost of manual judgment can be greatly reduced, and the efficiency of problem handling can be improved. Large language models are usually trained with a large amount of data and can more accurately identify the domain involved in the problem, thereby improving the accuracy and professionalism of subsequent processing. It can adapt to a variety of different problems and ensure that the determined business domain is correct through verification steps, so that the model can flexibly respond to various complex scenarios. It helps to avoid assigning problems to the wrong business domain, reducing the error rate and the complexity of subsequent processing. Accurate business domain identification can provide more professional and relevant answers, thereby improving the user's interactive experience.

[0162] (4) For each set of statements in a preset business domain, determine the similarity between the statement set and the question statement; determine the preset business domain with the greatest similarity as the reference business domain of the question statement; if the reference business domain is the same as the candidate business domain, determine that the candidate business domain has passed the verification; if the reference business domain is different from the candidate business domain, determine that the candidate business domain has not passed the verification, thereby helping to improve the accuracy and efficiency of business domain verification. By calculating the similarity between the question statement and the set of statements in the preset business domain, the business domain to which the question belongs can be accurately determined, which can not only reduce the error of manual judgment, but also save time and resources.

[0163] (5) Through feature extraction, the core content of the question and the user's true intention can be understood more accurately, thereby providing more accurate answers. Calculation based on feature similarity can help determine which sub-sentences are most relevant to the question, so that these sub-sentences can be given priority when generating answers, improving the relevance and quality of the answers. By quantifying the similarity between sub-sentences and questions, information directly related to the question can be more systematically generated, avoiding providing irrelevant or erroneous information. Feature extraction of question statements can identify specific details of the question, such as keywords and syntactic structure, which helps to deeply understand the question. Comparison between the features of sub-sentences and the features of question statements allows the system to make accurate matches, thereby providing more accurate answers. Similarity calculation can help quickly locate the sub-sentences most relevant to the question, improving the efficiency of information retrieval.

[0164] (6) The effectiveness and reliability of the large language model in actual business applications are ensured through a systematic verification and evaluation mechanism. First, by judging whether the candidate business domain meets the requirements, it helps to promptly discover performance problems that may exist in the model in a specific domain. Then, when the candidate business domain fails to pass the verification, the recognition accuracy of the model is calculated. This practice can quantify the performance of the model in this domain and provide an objective basis for subsequent decision-making. Most importantly, by comparing the recognition accuracy with the pre-set threshold, it can be decided whether to identify the candidate business domain as a business domain or to select a reference business domain. This ensures that the model will only be put into use when its performance meets the established standards, thereby avoiding business risks that may be caused by insufficient model performance. This helps to improve the accuracy of large language model deployment, optimize business processes, improve customer satisfaction, and provide scientific support for corporate decision-making.

[0165] (7) By identifying the preset information in the names of candidate business areas, potential sensitive data, such as personal information and confidential information, can be effectively discovered, which is crucial for protecting user privacy and complying with data security regulations. When the recognition result indicates that the content contains preset information, it is immediately judged as failed, which helps prevent the leakage of preset information and maintain the compliance and reputation of the enterprise. On the contrary, if the recognition result shows that there is no preset information in the content, the verification result is passed, which indicates that the content in this area meets the security standards and can proceed with subsequent business processes. This not only improves the security of information processing, but also ensures that the enterprise can efficiently expand and operate its business under the premise of complying with laws and regulations.

[0166] (8) The possible business types and probabilities of the question statements are automatically determined, and the efficiency and accuracy of classification enable the questions to be quickly routed to the correct business areas. Secondly, by selecting the business type with the highest probability as the candidate type, the process optimizes resource allocation, making subsequent verification and answering more focused and efficient. Finally, the step of verifying the candidate business type ensures the reliability of the question classification. Only the verified business type will be determined as the final first business type, which greatly reduces the risk of misclassification and improves the accuracy of overall business processing and user satisfaction. Not only does it optimize the problem handling process, but it also improves the user experience and the quality of enterprise business processing.

[0167] (9) By identifying the preset information in the names of candidate business types, it is ensured that potential sensitive data, such as personal information, financial data, etc., can be discovered and processed in a timely manner, thereby effectively preventing the risk of data leakage and privacy infringement. Secondly, when the identification result indicates the presence of preset information, the type verification result is determined to be failed, which helps prevent business types that do not meet security standards from being adopted and maintains compliance and reputation. Conversely, if the identification result shows that there is no preset information in the content, the verification result is passed, which not only confirms the applicability of the business type, but also promotes the smooth progress of business processes. It enhances the ability of information security management, reduces operational risks, and improves business processing efficiency and user trust.

[0168] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for generating a response statement, characterized in that: The method comprises: Performing domain identification on the question statement to obtain the business domain of the question statement; Determine a first business type to which the question statement belongs from a plurality of business types in the business field; Based on the response knowledge corresponding to the first business type, a response prediction is performed on the question statement to obtain a response statement for the question statement.

2. The method according to claim 1, characterized in that The performing of domain identification on the question statement to obtain the business domain of the question statement includes: Performing domain recognition on the question statement through a large language model to obtain candidate business domains of the question statement; The candidate business domain is verified, and if the verification is passed, the candidate business domain is determined as the business domain of the question statement.

3. The method according to claim 2, characterized in that The verifying of the candidate business field includes: Determine a reference business domain of the question statement from preset business domains; If the reference business field is the same as the candidate business field, then it is determined that the verification is passed; If the reference business domain is different from the candidate business domain, it is determined that the verification fails.

4. The method according to claim 3, characterized in that Determining the reference business domain of the question statement from the preset business domain includes: For each preset business field, determining the similarity between the sentence set of the preset business field and the question sentence; The preset business domain whose similarity meets the condition is determined as the reference business domain of the question statement.

5. The method according to claim 4, characterized in that The statement set includes a plurality of sub-statements, and the determining the similarity between the statement set in the preset business domain and the question statement includes: Performing feature extraction on the question sentence to obtain features of the question sentence; For each sub-sentence, extract features of the sub-sentence to obtain features of the sub-sentence, and determine the similarity of the sub-sentence based on a feature distance between the features of the sub-sentence and the features of the question sentence; The similarity of each sub-sentence is subjected to a preset operation to obtain the similarity between the sentence set of the preset business field and the question sentence.

6. The method according to claim 3, characterized in that After verifying the candidate business domain, the method further includes: If the verification fails, obtaining the recognition accuracy of the large language model in the candidate business field; If the recognition accuracy is greater than a threshold, determining the candidate business domain as the business domain of the question statement; If the recognition accuracy is less than or equal to the threshold, the reference business domain is determined as the business domain of the question sentence.

7. The method according to claim 2, characterized in that The verifying of the candidate business field includes: Performing preset information recognition on the name of the candidate business field to obtain a recognition result of the candidate business field; If the recognition result indicates that the name of the candidate business field includes preset information, determining that the candidate business field has not passed the verification; If the recognition result indicates that the name of the candidate business field does not include the preset information, it is determined that the candidate business field passes the verification.

8. A device for generating a response statement, characterized in that: The device comprises: An identification module, used to perform domain identification on the question statement to obtain the business domain of the question statement; The identification module is further used to determine the first business type to which the question statement belongs from the multiple business types in the business field; A prediction module is used to predict the answer to the question statement based on the answer knowledge corresponding to the first business type, so as to obtain an answer statement for the question statement.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the method for generating a response statement as described in any one of claims 1 to 7 when executing computer executable instructions or computer programs stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the method for generating a response statement as described in any one of claims 1 to 7 is implemented.

Images