Problem solving method, computing device and computer readable storage medium

Abstract

The embodiments of this specification provide a problem processing method, a computing device and a computer-readable storage medium, wherein the problem processing method includes: using a problem processing model to perform data analysis on a problem to be processed, and obtaining reference text information corresponding to the problem to be processed; determining a target data acquisition unit based on the reference text information, and determining a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection requirement; inputting the problem to be processed and the reference text into an answer prediction model, and obtaining an initial answer corresponding to the problem to be processed; using the problem processing model, detecting the correlation between the problem to be processed and the initial answer, and determining a target answer corresponding to the problem to be processed based on the correlation detection result; avoiding the problem that a trained neural network may not be able to predict an accurate answer based on limited knowledge, and improving the accuracy of the answer.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and more particularly to a problem solving method. One or more embodiments of this specification also relate to another problem solving method, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of artificial intelligence technology, neural networks also perform corresponding tasks in various types of application scenarios according to the needs of actual applications. For example, in problem-solving scenarios, neural networks can be used to predict answers to questions.

[0003] In the current process of using neural networks to predict answers to questions, since the knowledge learned by the trained neural network is limited, it may not be possible to predict the exact answer based on the limited knowledge. Therefore, how to improve the accuracy of the answer has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a method for problem solving. One or more embodiments of this specification also relate to another method for problem solving, a problem solving device, another problem solving device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defect that a neural network in the prior art may not be able to predict an accurate answer based on limited knowledge.

[0005] According to a first aspect of an embodiment of this specification, a problem handling method is provided, including:

[0006] Using the problem processing model to perform data analysis on the problem to be processed, and obtaining reference text information corresponding to the problem to be processed;

[0007] Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0008] Inputting the problem to be processed and the reference text into an answer prediction model to obtain an initial answer corresponding to the problem to be processed;

[0009] The problem processing model is used to detect the relevance between the problem to be processed and the initial answer, and the target answer corresponding to the problem to be processed is determined according to the relevance detection result.

[0010] According to a second aspect of an embodiment of this specification, there is provided a problem handling device, including:

[0011] A data analysis module is configured to perform data analysis on the problem to be processed using the problem processing model to obtain reference text information corresponding to the problem to be processed;

[0012] A text acquisition module is configured to determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0013] An answer prediction module is configured to input the question to be processed and the reference text into an answer prediction model to obtain an initial answer corresponding to the question to be processed;

[0014] The answer detection module is configured to use the question processing model to detect the correlation between the question to be processed and the initial answer, and determine the target answer corresponding to the question to be processed according to the correlation detection result.

[0015] According to a third aspect of an embodiment of this specification, another problem handling method is provided, which is applied to a cloud-side device, including:

[0016] The problem to be processed sent by the receiving terminal side device is analyzed by using the problem processing model to obtain reference text information corresponding to the problem to be processed;

[0017] Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0018] Inputting the problem to be processed and the reference text into a large language model to obtain an initial answer corresponding to the problem to be processed;

[0019] Using the problem processing model, the correlation between the problem to be processed and the initial answer is detected, and the target answer corresponding to the problem to be processed is determined according to the correlation detection result;

[0020] The target answer is sent to the end-side device.

[0021] According to a fourth aspect of an embodiment of this specification, another problem handling device is provided, which is applied to a cloud-side device, including:

[0022] A data analysis module is configured to receive the pending issues sent by the terminal side device, and perform data analysis on the pending issues using the issue processing model to obtain reference text information corresponding to the pending issues;

[0023] A text acquisition module is configured to determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0024] An answer prediction module is configured to input the question to be processed and the reference text into a large language model to obtain an initial answer corresponding to the question to be processed;

[0025] An answer detection module is configured to detect the relevance between the question to be processed and the initial answer using the question processing model, and determine the target answer corresponding to the question to be processed according to the relevance detection result;

[0026] The answer sending module is configured to send the target answer to the terminal device.

[0027] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0028] Memory and processor;

[0029] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of any one of the above methods are implemented.

[0030] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of any one of the above methods when executed by a processor.

[0031] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of any one of the above methods when executed by a processor.

[0032] One or more embodiments of the present specification provide a method for processing a question. Before using an answer prediction model to predict an answer, the question processing model can be used to perform data analysis on the question to be processed, and based on the reference text information obtained from the data analysis, an associated reference text can be determined for the question to be processed; then, by inputting the question to be processed and the reference text into the answer prediction model, the answer prediction model can predict the answer for the question to be processed based on the additional supplementary knowledge of the reference text, thereby avoiding the problem that the knowledge learned by the trained neural network is limited and an accurate answer may not be predicted based on the limited knowledge.

[0033] Moreover, after obtaining the initial answer corresponding to the problem to be processed, in order to further improve the accuracy of the answer, the problem processing model can also be used to detect the correlation between the problem to be processed and the initial answer, and determine the accurate target answer for the problem to be processed based on the correlation detection result, thereby avoiding the problem that the trained neural network may not be able to predict the accurate answer based on limited knowledge, and improving the accuracy of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is an application diagram of a problem solving method provided by an embodiment of this specification;

[0035] Figure 2 is a flow chart of a problem handling method provided by an embodiment of this specification;

[0036] Figure 3 is a schematic diagram of a search and determination prompt in a problem handling method provided by an embodiment of the present specification;

[0037] Figure 4 is a schematic diagram of a document quality detection prompt in a problem handling method provided by an embodiment of the present specification;

[0038] Figure 5 is a schematic diagram of a reply quality detection prompt in a problem handling method provided by an embodiment of the present specification;

[0039] Figure 6 It is a schematic diagram of a processing process of a problem processing method provided by an embodiment of this specification;

[0040] Figure 7 is a flow chart of another problem handling method provided by an embodiment of this specification;

[0041] Figure 8 It is a structural schematic diagram of a problem handling device provided by an embodiment of this specification;

[0042] Fig. 9 is a schematic diagram of the structure of another problem handling device provided by an embodiment of this specification;

[0043] Fig.10 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0044] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0045] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0046] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0047] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0048] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0049] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0050] First, the terms involved in one or more embodiments of this specification are explained.

[0051] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can be used to understand and analyze natural language input by users, thereby providing services and assistance to users in different fields and tasks.

[0052] Question answering system (QA system): a system that automatically answers questions asked by users.

[0053] Text Generation: The model generates coherent and meaningful paragraphs of text.

[0054] Instruction fine-tuning: Further tune the pre-trained model on a specific task to improve its performance on the specific task.

[0055] Search determination: Determines whether to call a search and the search source based on the user query.

[0056] Rewrite: Transform the input question to improve retrieval or answering performance.

[0057] RAG (Retrieval Augmented Generation): Supplement and improve the model's answering ability by retrieving external resources.

[0058] Retriever: A tool or algorithm used to filter and extract relevant information from large data sets.

[0059] Retrieval base: A data set or document library from which the retriever searches and extracts relevant information.

[0060] Contextual understanding: The model has a deep understanding of the input information, including semantics, context, and implicit meaning.

[0061] Retrieval-augmented agent: An autonomous system that perceives, makes decisions, and executes RAG actions in a specific environment. The retrieval-augmented agent can be understood as a large model.

[0062] With the continuous development of artificial intelligence technology, neural networks also perform corresponding tasks in various types of application scenarios according to the needs of actual applications. For example, in problem-solving scenarios, neural networks can be used to predict answers to questions.

[0063] In recent years, the rapid development of large language models (LLMs) has greatly promoted the research and application of natural language processing and attracted widespread attention from academia and industry. The large language models can demonstrate strong natural language understanding and generation capabilities by pre-training on massive data. These models have performed well in many tasks, including text generation, translation, sentiment analysis, and question-answering systems. For example, question-answering systems (QA systems) use the natural language processing capabilities of LLMs to automatically understand and answer users' questions. These systems are widely used in search engines, virtual assistants, and online customer service. These applications not only improve user experience, but also play an important role in business and academic research. However, although these models perform well in many tasks, when dealing with queries that require external knowledge support or are complex and changeable, the intrinsic knowledge of the model alone is often not enough. For example, the intrinsic knowledge of the model alone is still insufficient when dealing with complex queries that require external knowledge support. It is in this context that it is particularly important to further improve the accuracy and reliability of the question-answering system.

[0064] In response to the above problems, this specification provides three solutions. The first solution is a document library retrieval solution. Since information retrieval systems and question-answering systems usually rely on keyword matching and simple rule sets, in this solution, when a user enters a query, the system retrieves documents containing the corresponding keywords from a preset document library. However, this solution has several significant disadvantages:

[0065] 1. Lack of deep understanding of keyword matching: This method relies on simple keyword matching and lacks deep semantic understanding of user queries. It cannot handle synonyms, morphological changes, and complex queries input by users, which can easily lead to inaccurate retrieval results.

[0066] 2. Limitations of the rule system: The rule system needs to be manually written and maintained. When faced with new problems or queries in different fields, its scalability and adaptability are very limited; manual labeling is costly and cannot be updated in a timely manner.

[0067] 3. Lack of context: Most traditional systems cannot consider context and multiple rounds of user interactions and can only handle static, single questions; this causes the system to perform poorly in continuous conversations and complex problem handling.

[0068] The second solution is a hybrid search and question-answering system, which combines traditional search algorithms with modern machine learning techniques. These systems use keyword matching to perform preliminary searches, while using deep learning models to rank and process the results. The implementation of this method usually includes the following steps: 1. Keyword search: Use search algorithms to retrieve documents related to user queries from local document libraries or network resources; 2. Document ranking: Use pre-trained deep learning models (such as BERT) to rank retrieved documents to improve relevance; 3. Answer extraction: Extract the paragraphs or sentences that best match the user query from the top-ranked documents, and use machine learning models to further refine the answers.

[0069] However, this solution has major technical flaws: First, the keyword matching method in this solution may ignore semantic relevance, resulting in inaccurate retrieval results. Second, document ranking and answer extraction rely on the performance of pre-trained models, and the computational complexity is high. Finally, a large amount of labeled data is required to train and fine-tune deep learning models, which increases costs and development cycles.

[0070] The third solution is a question-answering system based on knowledge graphs. Knowledge graph technology stores and manages information in a structured way, enabling the system to understand complex queries semantically. The main processes of this type of system include: 1. Knowledge graph construction: a large amount of structured and unstructured data is integrated to build a knowledge graph, where nodes represent entities and edges represent relationships between entities; 2. Semantic query parsing: through natural language processing technology, user queries are converted into relational queries in the graph; 3. Graph search and reasoning: search and reason along the nodes and edges in the knowledge graph to find the most relevant answers.

[0071] However, this solution has major technical flaws: First, building and maintaining a knowledge graph requires a large amount of high-quality structured data and manual annotation, which is costly. Second, updating and expanding the knowledge graph is relatively complex and may not reflect the latest information in a timely manner. Finally, for very complex or open-domain problems, the coverage and depth of detail of the knowledge graph may be insufficient.

[0072] Based on this, in this specification, a problem handling method is provided. This specification also involves another problem handling method, a problem handling device, another problem handling device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0073] See also Figure 1 , Figure 1 A schematic diagram of an application of a problem solving method provided according to an embodiment of this specification is shown. Figure 1 It can be seen that the user can send a question to the server 104 through the client 102, and the server 104 can splice the question with the search judgment prompt, and input the spliced ​​input data to the intelligent agent for search judgment; the intelligent agent first uses the Rewrite technology to semantically rewrite the question in the input data, and then performs search judgment analysis on the semantically rewritten question, and outputs the search type according to the instructions of the search judgment prompt; the server 104 selects the most suitable search strategy for document search according to the search type, and obtains the reference document associated with the question; then, the question and the reference document associated with the question are input into the large language model for answer prediction, and the predicted answer corresponding to the question is obtained; then, the server 104 uses the intelligent agent to perform quality inspection on the predicted answer output by the large language model, and sends the answer that passes the quality inspection to the client 102 for display to the user.

[0074] See also Figure 2 , Figure 2 A flowchart of a problem handling method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0075] Step 202: Use the problem processing model to perform data analysis on the problem to be processed, and obtain reference text information corresponding to the problem to be processed.

[0076] Among them, the problem to be processed can be understood as a problem that requires answer prediction. The problem to be processed can be text data or text and image data combined. That is to say, the problem to be processed can be a text problem to be processed, for example, "What is the weather like today?", "What is the date of the Dragon Boat Festival?" and other text data; or, the problem to be processed can be a picture and text problem to be processed, for example, the image of a table and the text data "What is on the table" can constitute a picture and text problem to be processed.

[0077] The problem processing model can be understood as a model that can perform at least two data processing operations on the problem to be processed. The problem processing model can be an agent, a large model, etc., and no specific limitation is made here. In one or more embodiments of this specification, the problem processing model can be a search enhancement agent.

[0078] Reference text information can be understood as attribute information corresponding to the reference text. For example, the reference text information can be a reference text identifier corresponding to the reference text, source information of the reference text and / or search information of the reference text (such as search keywords, search formulas), and no specific restrictions are made here.

[0079] The reference text can be understood as a knowledge document associated with the problem to be processed, which is used to assist the answer prediction model in understanding and / or answering the problem to be processed. For example, when the problem to be processed is "What is 3 to the power of 2?", the reference text can be a junior high school mathematics knowledge document; when the problem to be processed is "What is the effect of angelica sinensis?", the reference text can be a paper related to traditional Chinese medicine, a doctor's explanation of angelica sinensis on the Internet, and other documents;

[0080] Specifically, this specification provides a problem processing method. Considering that the knowledge learned by the trained neural network is limited, it may not be possible to predict an accurate answer based on the limited knowledge. Therefore, in the process of processing the problem to be processed, the problem processing model can be used to perform data analysis on the problem to be processed, so as to obtain reference text information of the reference text corresponding to the problem to be processed.

[0081] In one or more embodiments provided in this specification, the problem handling method provided in this specification uses a retrieval-augmented agent to implement the retrieval-augmented generation (RAG) technology in order to solve the limitations of large language models in processing complex queries. This technology makes up for the lack of inherent knowledge of the model by combining an external knowledge base. Once a user asks a question, the system will automatically search based on the question, extract relevant information from external resources, and then combine it with the inherent knowledge of the model to provide users with more accurate and detailed answers; RAG technology not only improves the accuracy of the model's answers, but also expands the applicability of the model in specific application scenarios.

[0082] At the same time, the question handling method provided in this specification also implements the Rewrite technology through the retrieval enhancement agent, which can rewrite the user's original question, which can further enhance the performance of retrieval and answering; the rewritten question is clearer and more concise, which helps to improve the retrieval efficiency and enables the system to find relevant information more quickly and accurately. This process takes into account the context of the question and the user's intention, ensuring that the rewriting does not deviate from the user's real needs. Combining these technologies lays the foundation for the efficient operation of the agent in various application scenarios; the specific implementation method is as follows.

[0083] The method of using the problem processing model to perform data analysis on the problem to be processed to obtain reference text information corresponding to the problem to be processed includes:

[0084] Inputting the problem to be processed into the problem processing model for semantic conversion to obtain the problem to be processed after semantic conversion;

[0085] Performing search and determination on the pending problem after the semantic conversion to obtain a search and determination result corresponding to the pending problem;

[0086] In the case where it is determined according to the search determination result that the reference text needs to be obtained, the reference text information corresponding to the problem to be processed is determined from the search determination result.

[0087] The semantic conversion can be understood as semantic rewriting of the problem to be processed, that is, semantic rewriting of the text content in the problem to be processed.

[0088] The search judgment can be understood as a judgment operation to determine whether it is necessary to search for reference texts for the problem to be processed. The search judgment result can be understood as the judgment result of whether the reference text is searched for the problem to be processed. The judgment result can be referred to Figure 3 There are four SEARCH TYPEs in the search results. Figure 3 It is a schematic diagram of a search and determination prompt in a problem handling method provided in an embodiment of the present specification.

[0089] Specifically, the problem processing method provided in this specification firstly combines the problem to be processed with the search judgment prompt information to obtain the search judgment input data corresponding to the problem to be processed, and inputs the search judgment input data into the problem processing model;

[0090] Secondly, the problem to be processed is semantically transformed using the problem processing model, thereby obtaining the problem to be processed after the semantic transformation; and based on the search judgment prompt information, the problem to be processed after the semantic transformation is searched and judged to obtain the search judgment result corresponding to the problem to be processed; the search judgment prompt information can be understood as Figure 3 The search decision prompt displayed; based on Figure 3 It can be seen that the search decision prompt contains multiple search types. The search decision prompt is used to guide the problem processing model to perform search decisions on the problems to be processed and give corresponding answers; the search types are Continue using Evidence, Closed retrieval, OpenRetrieval, and Non-retrieval.

[0091] Among them, Continue using Evidence means that historical retrieved evidence can be used (for example, using the evidence from the previous round). For example, the corresponding answer format output by the question processing model is: "Continue using evidence: "Relevant information is included in the evidence ({{"Continue using Evidence":"Relevant information is included in theevidence"}}). ".

[0092] Among them, Closed retrieval means using a closed search method to obtain the reference text (such as searching a local library, searching a specific server), etc. For example, the corresponding answer format output by the question processing model is: "Closed retrieval": "Giant panda conservation ({{"Closed retrieval": "Giant panda conservation"}}.)".

[0093] Among them, Open Retrieval means obtaining reference text by open search (such as full network search). For example, the corresponding answer format output by the question processing model is: "Open Retrieval": "weather forecast for tomorrow in Washington"}}".

[0094] Among them, Non-retrieval means not performing any search. For example, the corresponding answer format output by the question processing model is: "No verification or additional information needs to be verified or supplemented ({{"Non-Retrieval": "no additional information needs to be verified or supplemented"}})".

[0095] Finally, when it is determined that a reference text needs to be searched based on the search judgment result output by the problem processing model, the reference text information corresponding to the problem to be processed is determined from the search judgment result. For example, the reference text information can be a knowledge source identifier, an unconstrained data source, a specified search term, and other information.

[0096] In the above embodiment, for the determination search operation, the query input by the user is intelligently determined and rewritten to determine the search type, and the query is optimized to improve the search effect. In other words, this method uses the Rewrite technology to semantically optimize the user's question through intelligent search determination and question rewriting, making the search more accurate; the rewritten question can better capture the true intention of the user's query, thereby improving the relevance of the search results.

[0097] Step 204: Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection requirement.

[0098] Among them, the data acquisition unit can be understood as a unit for providing parameter text. For example, the data acquisition unit can be a database, a local disk, a server, a search engine, etc., which is not specifically limited here; there can be multiple data acquisition units, and the target data acquisition unit can be one of the multiple data acquisition units.

[0099] The text quality detection can be understood as an operation of performing a quality detection on the reference text, and the text quality detection operation can be performed by the problem processing model. For example, the text quality detection operation can be a text screening operation, a correlation detection operation between the reference text and the problem to be processed, an adaptation detection, etc.

[0100] In one or more embodiments provided in this specification, although RAG technology and Rewrite technology help improve the overall performance of the question-answering system, it is also necessary to consider the impact of the quality of the retrieved documents on the final answer. The size and parameters of large language models have a significant impact on their ability to handle noisy documents. Due to their complex architecture, large models can resist noise to a certain extent, but small models are easily disturbed, which in turn affects the quality of answers. Therefore, it is very important to design an efficient search text quality detection mechanism. In this mechanism, scoring the retrieved text (such as A / B / C / D) and filtering documents according to the scores is a key step in improving the quality of answers. For LLMs of different sizes, high-quality documents suitable for their characteristics should be used, which can significantly improve the reliability and accuracy of the system. In addition, an automated search decision mechanism is also indispensable. By automatically determining whether a local library search, an external open search, or the use of the previous round of evidence is required, resource utilization and response time can be optimized, thereby improving the efficiency of the overall system and user experience. Combined with instruction fine-tuning technology, the agent is further optimized for specific tasks to ensure that it performs well in real environments and better meet the needs of various application scenarios. These measures together build an efficient, intelligent and reliable question-answering system to provide users with high-quality services.

[0101] Based on this, this method introduces quality detection and document screening modules and adaptation modules into the intelligent agent, so that the most suitable documents can be screened out according to the characteristics of different models, reducing excessive dependence on a specific model and improving the accuracy of the answer; the specific implementation method is as follows.

[0102] The step of determining a target data acquisition unit based on the reference text information, and determining a reference text associated with the problem to be processed from the target data acquisition unit, comprises:

[0103] Determining a unit identifier and text search information corresponding to the reference text from the reference text information;

[0104] Determining the target data acquisition unit based on the unit identifier, and determining the reference text to be detected associated with the problem to be processed from the target data acquisition unit based on the text search information, wherein the target data acquisition unit is a unit corresponding to the unit identifier among the multiple data acquisition units;

[0105] The problem processing model is used to perform quality inspection on the problem to be processed and the reference text to be inspected, and the reference text associated with the problem to be processed is determined according to the quality inspection result.

[0106] The unit identifier can be understood as information that uniquely identifies the target data acquisition unit. For example, the unit identifier can be the name, website, interface name, IP address, etc. of the target data acquisition unit, and no specific limitation is made here.

[0107] The text search information can be understood as information used to search the reference text. For example, the text search information can be understood as keywords, search formulas, search conditions, reference text identifiers (such as reference text names, reference text numbers, etc.) corresponding to the reference text, etc., and no specific restrictions are made here.

[0108] The reference text to be tested can be understood as a reference text that needs to be quality tested.

[0109] Among them, using the problem processing model to perform quality inspection on the problem to be processed and the reference text to be detected can be understood as fusing the problem to be processed and the reference text to be detected with the quality inspection prompt information to obtain quality inspection input data, and inputting the quality inspection input data into the problem processing model to perform quality inspection on the problem to be processed and the reference text to be detected.

[0110] Taking the scenario in which the problem processing method provided in this specification uses a large language model to predict answers as an example, the problem processing method is explained, wherein the problem processing model can be an intelligent agent (i.e., a large language model enhanced intelligent agent that realizes dynamic retrieval and quality inspection), and the answer prediction model is a large language model; based on this, the workflow of the intelligent agent starts with the problem to be processed input by the user, and this input (i.e., the problem to be processed) first enters the search and judgment stage; according to the result of the search and judgment (i.e., reference text information), the intelligent agent has four processing methods: searching the local library, open search, using the evidence of the previous round, and directly continuing without any search.

[0111] When it is determined based on the results that a local database search (i.e., a specific database) or an open search is needed, or evidence from a previous round is used, the search keywords or query conditions are used to obtain evidence (i.e., reference text) related to the problem to be addressed from the above data sources;

[0112] Then, the retrieved documents are scored (such as A / B / C / D) and the most suitable documents are filtered based on the scoring results.

[0113] In the above embodiment, the intelligent search judgment operation of the intelligent agent can intelligently determine the most appropriate search strategy based on the user's query content, including searching the local library, open search, using the evidence of the previous round, or continuing without searching. It can select the most suitable search strategy according to the query type to improve system resource utilization and response speed; and it can perform quality detection on the searched reference text to ensure the quality of the reference text.

[0114] In one or more embodiments provided in this specification, the intelligent search determination operation can intelligently determine the most appropriate search strategy based on the user query content, including searching the local library, open search, and using the previous round of evidence. The specific implementation method is as follows.

[0115] The target data acquisition unit is a historical data storage unit, a target server, or at least two other servers;

[0116] The step of determining the target data acquisition unit based on the unit identifier, and determining the reference text to be detected associated with the problem to be processed from the target data acquisition unit based on the text search information, comprises:

[0117] Determine a historical data storage unit based on the unit identifier, determine an associated historical reference text associated with the problem to be processed from the historical reference texts stored in the historical data storage unit based on the text search information, and determine the associated historical reference text as the reference text to be detected; or

[0118] Determine a target server based on the unit identifier, search service data in the target server based on the text search information, obtain a search text associated with the text search information, and use the search text as the reference text to be detected; or

[0119] At least two other service ends are determined based on the unit identifier, and based on the text search information, service data in each other service end is searched respectively to obtain other search texts associated with the text search information, and the other search texts are used as the reference texts to be detected.

[0120] The historical data storage unit may be understood as a data storage unit for storing historical reference texts, and the historical data storage unit may be a database, a local disk, or the like.

[0121] The target server can be understood as a specific server, for example, a server that provides knowledge retrieval services.

[0122] Following the above example, the agent's workflow starts with user input, which first enters the search judgment stage. According to the results of the search judgment, the agent has four processing methods: search the local library, open search, use the evidence of the previous round, or directly continue writing without any search. If you choose to search the local library (i.e., the target server), you will enter the local library retrieval stage. The retrieved content will undergo quality inspection and document screening and adaptation, and then generate the final answer. If you choose to open search, you will search from multiple data sources on the Internet (i.e., at least two other servers) to obtain reference documents corresponding to the question, and the system will perform quality inspection on the obtained content. The documents that pass the inspection will be screened and adapted, and then the final answer will be generated. Similarly, if you choose to use the evidence of the previous round, the evidence of the previous round will be obtained from the historical database (i.e., the historical data storage unit). These evidences will also undergo the same quality inspection and screening process, and finally generate an answer.

[0123] In the above embodiment, the intelligent search judgment operation of the intelligent agent can intelligently determine the most appropriate search strategy based on the user's query content, including searching the local library, open search, using the evidence of the previous round, or continuing without searching. It can select the most suitable search strategy according to the query type, thereby improving system resource utilization and response speed.

[0124] In one or more embodiments provided in this specification, the using the problem processing model to perform quality detection on the problem to be processed and the reference text to be detected, and determining the reference text associated with the problem to be processed according to the quality detection result, includes:

[0125] Inputting the problem to be processed and the reference text to be tested into the problem processing model for quality testing to obtain a quality testing result;

[0126] In the case where it is determined that the quality inspection result is consistent with a first quality inspection result of at least two preset quality inspection results, determining the reference text to be inspected as a reference text associated with the problem to be processed;

[0127] When it is determined that the quality inspection result is consistent with the second quality inspection result of the at least two preset inspection results, based on the quality inspection result and the text search information, the reference text to be inspected associated with the problem to be processed is re-determined from the target data acquisition unit, and the problem to be processed and the reference text to be inspected are re-input into the problem processing model for quality inspection until the quality inspection result is consistent with the first quality inspection result.

[0128] The first quality detection result may be understood as a quality detection result with relatively high quality, for example, a high-quality detection result, and there may be multiple first quality detection results.

[0129] The second quality detection result may be understood as a quality detection result indicating lower quality, for example, a low quality detection result. The second quality detection result may be one or more or empty.

[0130] It should be noted that inputting the problem to be processed and the reference text to be detected into the problem processing model for quality detection can be understood as fusing the problem to be processed and the reference text to be detected with the quality detection prompt information to obtain quality detection input data, and inputting the quality detection input data into the problem processing model for quality detection of the problem to be processed and the reference text to be detected; the quality detection prompt information can be understood as Figure 4 In the document quality detection prompt, Figure 4 Schematic diagram of a document quality detection prompt in a problem handling method provided by an embodiment of this specification. Figure 4 It can be seen that the document quality detection prompt contains multiple document quality classifications. The document quality detection prompt is used to guide the problem processing model to perform quality detection on the problem to be processed and the reference text to be detected, and give corresponding answers; the document quality classification includes 4 quality classifications of A / B / C / D.

[0131] Using the above example, based on Figure 4It can be seen that quality detection, document screening and adaptation operations are completed by the intelligent agent. By sending the document quality detection prompt to the trained intelligent agent, A / B / C / D quality (i.e., quality detection result) is output; subsequently, the quality of the document to be input is determined according to the amount of parameters for generating a large model. If the ability to generate a large model is strong and has strong anti-interference ability, then documents with qualities of A / B / C / D (i.e., the first quality detection result) can be input; if the ability to generate a large model is weak and has weak anti-interference ability, then only documents with quality A (i.e., the first quality detection result) are input to avoid generating wrong answers; the remaining documents with qualities of B / C / D (the second quality detection result) are not adopted.

[0132] In the above embodiment, the document quality detection module of the intelligent agent is used to evaluate the quality of the retrieved evidence and score it to ensure that the answer returned to the user is accurate and useful.

[0133] In one or more embodiments provided in the present specification, the target data acquisition unit is determined based on the unit identifier, and the reference text to be detected associated with the problem to be processed is determined from the target data acquisition unit based on the text search information, including: utilizing a RAG link to determine the target data acquisition unit based on the unit identifier, and the reference text to be detected associated with the problem to be processed is determined from the target data acquisition unit based on the text search information.

[0134] Continuing with the above example, in order to address the limitations of large language models in processing complex queries, this method introduces retrieval-augmented generation technology; this technology uses RAG links to retrieve reference data related to the problem to be processed in the external knowledge base, making up for the lack of inherent knowledge of the model.

[0135] Step 206: Input the question to be processed and the reference text into an answer prediction model to obtain an initial answer corresponding to the question to be processed.

[0136] The answer prediction model may be a model for predicting answers to questions to be processed. For example, the answer prediction model may be a large model, a large language model, etc.

[0137] The initial answer can be understood as the answer obtained by the answer prediction model when predicting the answer to the question to be processed. For example, when the question to be processed is "What is the weather like today?", the initial answer can be "Today's weather is sunny and the temperature is 10-25 degrees."

[0138] In one or more embodiments provided in this specification, when the question processing model determines that it is not necessary to obtain reference text, the question to be processed can be directly input into the answer prediction model for processing, and the specific implementation method is as follows.

[0139] After searching and determining the pending problem after the semantic conversion and obtaining the search and determination result corresponding to the pending problem, the method further includes:

[0140] When it is determined according to the search result that the reference text does not need to be obtained, the question to be processed is input into the answer prediction model to obtain an initial answer corresponding to the question to be processed.

[0141] Using the above example, the agent has four options for handling the problem based on the result of the search: searching the local database, open search, using the evidence from the previous round, or continuing the problem without any search. In the case of no search, the agent generates the final answer directly.

[0142] Step 208: Using the question processing model, detect the correlation between the question to be processed and the initial answer, and determine the target answer corresponding to the question to be processed according to the correlation detection result.

[0143] The target answer can be understood as the answer that passes the relevance test.

[0144] In one or more embodiments provided in this specification, the using of the question processing model to detect the relevance between the question to be processed and the initial answer, and determining the target answer corresponding to the question to be processed according to the relevance detection result, includes:

[0145] Using the question processing model, detecting the relevance between the question to be processed and the initial answer to obtain the relevance detection result;

[0146] When it is determined that the relevance detection result is consistent with a first relevance detection result among at least two preset relevance detection results, the initial answer is determined as the target answer corresponding to the question to be processed.

[0147] Among them, the first correlation detection result can be understood as a quality detection result indicating that the correlation between the question to be processed and the initial answer is relatively high or relatively good, for example, a high correlation detection result, and there may be multiple first correlation detection results.

[0148] The second correlation detection result may be understood as a quality detection result indicating that the correlation between the question to be processed and the initial answer is low or poor, for example, a low correlation detection result. There may be multiple second correlation detection results.

[0149] It should be noted that, using the problem processing model to detect the correlation between the problem to be processed and the initial answer to obtain the correlation detection result can be understood as fusing the problem to be processed and the initial answer with the correlation detection prompt information to obtain correlation detection input data, and inputting the correlation detection input data into the problem processing model to perform correlation detection to obtain the corresponding correlation detection result; the correlation detection prompt information can be understood as a reply quality detection prompt; the reply quality detection prompt includes multiple types of detection scores (i.e., correlation detection results), and the reply quality detection prompt is used to guide the problem processing model to detect the correlation between the problem to be processed and the initial answer.

[0150] Continuing with the above example, the final answer generated needs to go through a round of generation quality inspection; if the quality inspection fails, the system will reject the answer and output the response and quality score; the quality score can be found in Figure 5 , Figure 5 It is a schematic diagram of a reply quality detection prompt in a problem handling method provided by an embodiment of the present specification; wherein, score 1, score 2, and score 3 may be first correlation detection results; score 4 and score 5 may be second correlation detection results.

[0151] If the quality check passes, the answer is accepted and the response and quality score are output. The entire process is designed to ensure the quality of the output answers is reliable, and to optimize the results through testing and screening at different steps to achieve the high-standard answers expected by users.

[0152] In the above embodiment, the model response quality detection module of the agent is used to evaluate the matching score between the final answer generated by the large model and the user request, and provide the user with a quantifiable confidence score. The prompt used for response quality detection can participate in Figure 5 , Figure 5 It is a schematic diagram of a reply quality detection prompt in a problem handling method provided in an embodiment of the present specification.

[0153] In one or more embodiments provided in this specification, the using the question processing model to detect the relevance between the question to be processed and the initial answer, and after obtaining the relevance detection result, further includes:

[0154] When it is determined that the relevance detection result is consistent with the second relevance detection result of the at least two preset relevance detection results, using the problem processing model to perform data analysis on the relevance detection result and the problem to be processed, obtain updated text information corresponding to the problem to be processed, and use the updated text information as reference text information corresponding to the problem to be processed;

[0155] Continue to determine the target data acquisition unit based on the reference text information, and determine the reference text associated with the problem to be processed from the target data acquisition unit, until the relevance detection result is consistent with the first relevance detection result, and determine the target answer corresponding to the problem to be processed.

[0156] Continuing with the above example, the final answer generated needs to go through a round of generation quality inspection; if the quality inspection fails, the system will reject the answer and output a response and quality score; then, the operations corresponding to the above steps 202 to 206 are performed again, that is, the reference text corresponding to the question to be processed is retrieved, and the answer is predicted to obtain a new answer; then, the relevance inspection of the answer is continued until the quality inspection passes and the answer is accepted.

[0157] In one or more embodiments provided in this specification, before using the problem processing model to perform data analysis on the problem to be processed and obtaining reference text information corresponding to the problem to be processed, the method further includes:

[0158] Receiving the pending question sent by the client, wherein the pending question is sent by the client when the user performs a question uploading operation on the client;

[0159] After determining the target answer corresponding to the problem to be processed according to the relevance detection result, the method further includes:

[0160] The target answer is sent to the client for display.

[0161] Specifically, the problem processing method provided in this specification can be applied to a server, which can receive the problem to be processed sent by the client, and after obtaining the target answer using the answer prediction model, the target answer can be sent to the client for display, thereby meeting the user's problem processing needs.

[0162] In one or more embodiments provided in this specification, the problem processing model needs to be trained, and the specific training method is as follows.

[0163] After the problem processing model is used to perform data analysis on the problem to be processed and reference text information corresponding to the problem to be processed is obtained, the method further includes:

[0164] Determine a sample question for the problem processing model to be trained, process the sample question using the answer prediction model, and obtain an initial sample reference text and an initial sample answer corresponding to the sample question;

[0165] Performing data screening on the initial sample answers to obtain target sample answers, and performing data screening on the initial sample reference text to obtain target sample reference text;

[0166] The problem processing model to be trained is trained using the sample question, the target sample reference text and the target sample answer to obtain the problem processing model.

[0167] Continuing with the above example, this manual can use the method of instruction fine-tuning (SFT) to train the agent. Before implementing instruction fine-tuning, all data sets from all sources are converted into a unified structured format to ensure that the model can interpret each task consistently.

[0168] The data set can be obtained through a large language model. Specifically, the training data comes from the original data in the database. After cleaning, the corresponding training data (i.e., data set) is generated by the large language model through different prompts.

[0169] Then, in the process of instruction fine-tuning (SFT), the model is exposed to a large number of training examples with explicit instruction labels, each of which contains the task instruction, input data, and expected output. The model learns how to extract key information from the input data and produce appropriate outputs based on the given instructions. To ensure that the model can generalize to unseen instructions, this method uses multiple levels of instruction abstraction, from specific task descriptions to more general functional categories, so that the model can flexibly apply what it has learned in new scenarios.

[0170] After that, the DeepSpeed ​​interface is used to load this data and train the agent, and in each training batch, it receives samples containing instructions and expected outputs. The model tries to predict the output, calculates the loss value, and backpropagates this information to update the model's weights so that the model is closer to the correct answer in subsequent predictions.

[0171] As training progresses, the model gradually learns how to produce high-quality output corresponding to different instructions, whether it is improving the formulation of queries, evaluating the relevance of documents, or improving the accuracy of responses.

[0172] It should be noted that through the acceleration of DeepSpeed, this process can be completed in a shorter time while maintaining high precision. The final model has the ability to perform a variety of tasks, greatly expanding its application scenarios in the real world.

[0173] In the above embodiment, the modular architecture between the agent and the large language model enables training and fine-tuning to be performed at a finer granularity, reducing the training cost of the overall system. At the same time, intelligent search judgment and quality detection help reduce the introduction of irrelevant documents and improve the efficiency of the system. In addition, the use of a large language model to generate corresponding training data greatly reduces the annotation cost compared to manual work.

[0174] One or more embodiments of the present specification provide a method for processing a question. Before using an answer prediction model to predict an answer, the question processing model can be used to perform data analysis on the question to be processed, and based on the reference text information obtained from the data analysis, an associated reference text can be determined for the question to be processed; then, by inputting the question to be processed and the reference text into the answer prediction model, the answer prediction model can predict the answer for the question to be processed based on the additional supplementary knowledge of the reference text, thereby avoiding the problem that the knowledge learned by the trained neural network is limited and an accurate answer may not be predicted based on the limited knowledge.

[0175] Moreover, after obtaining the initial answer corresponding to the problem to be processed, in order to further improve the accuracy of the answer, the problem processing model can also be used to detect the correlation between the problem to be processed and the initial answer, and determine the accurate target answer for the problem to be processed based on the correlation detection result, thereby avoiding the problem that the trained neural network may not be able to predict the accurate answer based on limited knowledge, and improving the accuracy of the answer.

[0176] The following combination Figure 6 , taking the application of the problem processing method provided in this specification in the scenario of implementing answer prediction by a large language model-enhanced intelligent agent based on dynamic retrieval and quality inspection as an example, the problem processing method is further explained. Figure 6 A schematic diagram of the processing process of a problem processing method provided by an embodiment of the present specification is shown, which specifically includes three parts: determining training data, training an intelligent agent, and answer prediction.

[0177] Determining the training samples refers to determining the training data used to train the agent, which specifically includes the following steps:

[0178] 1. Determine the original data from the database as training samples.

[0179] Among them, based on Figure 6 It can be seen that the training samples include but are not limited to:

[0180] Training sample 1 can be: question "What is the weather like in Hangzhou?" Figure 3 The search decision prompt shown indicates the prompt information for the large language model to determine which search method to select;

[0181] Training sample 2 can be: question "What is the weather like in Hangzhou?", reference document corresponding to the question, Figure 4 The document quality detection prompt shown, prompt information indicating the large language model to determine which document quality detection result to select;

[0182] Training sample 3 can be: question "What is the weather like in Hangzhou?", the answer to the question, Figure 5 The response quality detection prompt shown indicates prompt information for the large language model to determine which response quality detection result to select;

[0183] 2. According to the prompt in each training sample, the question, reference document and / or answer are concatenated to obtain the input data of the large language model, and then input into the large language model for reasoning.

[0184] 3. Use the large language model to perform semantic reasoning on the input data and obtain the corresponding reasoning results;

[0185] For example, for training data 1, the inference result output by the large language model may be: "{Open search: Hangzhou today's weather}";

[0186] For training data 2, the inference result output by the large language model can be: "Quality A: The retrieved evidence usually focuses on providing the most relevant facts, has high information consistency, and focuses on directly answering user queries. It is suitable for queries on basic facts and definitions."

[0187] For training data 3, the inference result output by the large language model may be: "5-completely relevant and accurate, a good response".

[0188] 4. Use the inference result as the sample label, and then use the training samples and corresponding sample labels to train the model of the intelligent agent.

[0189] Among them, training the intelligent agent refers to using training data to train the model of the intelligent agent, which specifically includes the following steps:

[0190] 1. Before training the agent using instruction fine-tuning, all data sets from all sources are converted into a unified structured format to ensure that the model can interpret each task consistently.

[0191] 2. Load training data in batches through the DeepSpeed ​​interface and train the agent.

[0192] Among them, the SFT process involves exposing the model to a large number of training instances with clear instruction labels, each of which contains task instructions (i.e. the prompt mentioned above), input data (i.e. questions, documents, etc. in the training samples) and expected output (i.e. sample labels).

[0193] 3. The agent learns how to extract key information from the input data and produce appropriate predicted outputs based on the given task instructions;

[0194] 4. Based on the predicted output and sample labels, the loss value is calculated and this information is back-propagated to update the weights of the model, so that the agent is closer to the correct answer in subsequent predictions, thereby obtaining a trained agent.

[0195] 5. Repeat the training steps of steps 2-3 above. As the training progresses, the model gradually masters how to produce corresponding high-quality outputs based on different instructions, whether it is improving the formulation of queries, evaluating the relevance of documents, or improving the accuracy of replies, thereby obtaining a trained intelligent agent.

[0196] Among them, answer prediction refers to the large language model-enhanced intelligent agent based on dynamic retrieval and quality inspection to achieve answer prediction of the large language model, which specifically includes the following steps:

[0197] 1. Receive the question input by the user, concatenate the question with the search decision prompt, and input the concatenated input data into the intelligent agent for search decision.

[0198] 2. The agent first uses the Rewrite technology to semantically rewrite the questions in the input data, then performs search decision analysis on the semantically rewritten questions, and outputs the search type according to the instructions of the search decision prompt.

[0199] 3. The system uses the RAG (Retrieval Enhancement) link to select the most suitable search strategy for document search based on the search type and obtain reference documents related to the question.

[0200] Specifically, depending on the search type result determined by the search, the agent has four processing options: searching the local repository (e.g., searching for documents from Wikipedia’s server), open search, using evidence from the previous round, or continuing directly without any search.

[0201] The implementation of various search processing methods is as follows:

[0202] (1) If you choose to search the local database, you will enter the local database search phase; search the specific data source based on the search keywords corresponding to the question to obtain content documents;

[0203] For the searched content document, the content document and the question will be spliced ​​with the document quality detection prompt, and the spliced ​​data will be input into the intelligent agent for document quality detection;

[0204] If the test is passed, the document screening and adaptation module of the intelligent agent will be used to screen the documents and obtain the evaluation results (see Figure 4 Quality assessment results in

[0205] If the document is determined to have passed the screening based on the evaluation results, it can be used as input data for the subsequent large language model.

[0206] (2) If open search is selected, the system searches for specific information or answers from a wide range of unconstrained data sources (such as the Internet) based on the search keywords corresponding to the question;

[0207] For the searched content document, the content document and the question will also be spliced ​​with the document quality detection prompt, and the spliced ​​data will be input into the intelligent agent for document quality detection;

[0208] If the test is passed, the document screening and adaptation module of the intelligent agent will be used to screen the documents and obtain the evaluation results (see Figure 4 Quality assessment results in

[0209] If the document is determined to have passed the screening based on the evaluation results, it can be used as input data for the subsequent large language model.

[0210] (3) If you choose to use the evidence from the previous round (i.e., the historical content documents obtained by searching in the process of answering the questions in the previous round), then obtain the evidence from the previous round from the historical database.

[0211] Since the previous round of evidence also goes through the same quality inspection and screening process, it can be directly used as a reference document for the current round of questions.

[0212] (4) Without performing any search, the final answer is generated directly without searching for reference documents.

[0213] 4. Input the question and the reference documents associated with the question into the large language model for answer prediction to obtain the predicted answer corresponding to the question.

[0214] It should be noted that when a search strategy without any search is adopted, the question is directly input into the large language model for answer prediction to obtain the predicted answer corresponding to the question.

[0215] 5. Use intelligent agents to perform quality checks on the predicted answers output by the large language model.

[0216] Specifically, first, the user's question and the predicted answer output by the large language model are spliced ​​with the response quality detection prompt, and the spliced ​​input data is input into the intelligent agent for response quality detection.

[0217] Secondly, the agent analyzes the questions and predicted answers in the input data, and outputs the corresponding quality scores according to the instructions of the response quality detection prompt (see Figure 5 );

[0218] If the quality score is less than the preset threshold (i.e., 4 or 5 points), it is determined that the quality check has failed, and the system will reject the answer and output the response and quality score;

[0219] If the quality score is greater than or equal to the preset threshold (i.e., 4 or 5 points), it is determined that the quality test has passed, and the answer is accepted. The response and quality score are also output.

[0220] It should be noted that, if the answer quality test fails, the answer prediction operations of steps 2 to 5 above may be continued until the answer quality test passes.

[0221] 6. Send the answers that pass the test to the user.

[0222] Based on the above steps, it can be seen that the problem handling method in one or more embodiments of the present specification provides a large language model enhanced intelligent agent based on dynamic retrieval and quality inspection, which can realize intelligent search judgment, Rewrite technology, and RAG technology, significantly improving the accuracy and reliability of the question-answering system; it can not only intelligently optimize the search strategy and enhance the retrieval effect, but also improve the system response efficiency and answer quality through automatic data generation and quality detection modules; and the modular design of the separation of the intelligent agent and the large language ensures flexibility and adaptability, so that the system can quickly adapt to different fields and task requirements, and has broad application prospects and high value.

[0223] Based on the above content, it can be seen that the intelligent search judgment module of the agent can intelligently determine the most appropriate search strategy based on the user's query content, including searching the local library, open search, using the evidence of the previous round, or continuing without searching. It can select the most suitable search strategy according to the query type to improve system resource utilization and response speed.

[0224] The intelligent agent can use Rewrite technology to perform semantic optimization on user queries, making search queries clearer and more concise, and enhancing retrieval effects. Through semantic optimization, the rewritten queries can better capture user intent and improve the relevance and accuracy of retrieval results.

[0225] The agent introduces a quality detection module to score the retrieved documents (such as A / B / C / D) and selects the most suitable documents based on the scoring results.

[0226] Moreover, during the training process of the intelligent agent, data for supervised fine-tuning can be automatically generated based on a high-performance large language model. Automatically generating SFT data makes the model training and fine-tuning process more efficient and can quickly respond to the needs of different fields and new tasks. This method replaces traditional manual labeling and greatly reduces costs and time investment.

[0227] At the same time, the modularly designed agent makes this method independent of the specific large language model and can be seamlessly integrated with any LLM. This design achieves true pluggability and enhances the adaptability and flexibility of the system. Users can flexibly select and replace the large language model according to specific task requirements and application scenarios, and the agent can automatically adapt and optimize the workflow.

[0228] See also Figure 7 , Figure 7 A flowchart of another problem handling method provided according to an embodiment of the present specification is shown. The another problem handling method is applied to a cloud-side device and specifically includes the following steps.

[0229] Step 702: receiving the pending issues sent by the terminal side device, and performing data analysis on the pending issues using the issue processing model to obtain reference text information corresponding to the pending issues.

[0230] Step 704: Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection requirement.

[0231] Step 706: Input the question to be processed and the reference text into a large language model to obtain an initial answer corresponding to the question to be processed.

[0232] Step 708: Using the question processing model, detect the correlation between the question to be processed and the initial answer, and determine the target answer corresponding to the question to be processed according to the correlation detection result.

[0233] Step 710: Send the target answer to the terminal device.

[0234] In one or more embodiments provided in this specification, the cloud-side device can be a central cloud device of a distributed architecture or an edge cloud device of a distributed architecture. The cloud-side device can be a cloud-side device with a cloud desktop system or cloud desktop software installed and deployed, such as a cloud server, cloud host, etc.; the terminal-side device can be understood as any terminal that interacts with the cloud-side device for data; the terminal can be a laptop computer, desktop computer, tablet computer, smart device, server, etc.

[0235] One or more embodiments of the present specification provide a question processing method applied to a cloud-side device. Before using a large language model to predict an answer, the question processing model can be used to perform data analysis on the question to be processed, and based on the reference text information obtained from the data analysis, an associated reference text is determined for the question to be processed; then, by inputting the question to be processed and the reference text into the large language model, the large language model can predict the answer to the question to be processed based on the additional supplementary knowledge of the reference text, thereby avoiding the problem that the knowledge learned by the trained neural network is limited and an accurate answer may not be predicted based on the limited knowledge.

[0236] Moreover, after obtaining the initial answer corresponding to the problem to be processed, in order to further improve the accuracy of the answer, the problem processing model can also be used to detect the correlation between the problem to be processed and the initial answer, and determine the accurate target answer for the problem to be processed based on the correlation detection result, thereby avoiding the problem that the trained neural network may not be able to predict the accurate answer based on limited knowledge, and improving the accuracy of the answer.

[0237] The above is an illustrative solution of another problem-solving method of this embodiment. It should be noted that the technical solution of the other problem-solving method and the technical solution of the above-mentioned one problem-solving method belong to the same concept, and the details of the technical solution of the other problem-solving method that are not described in detail can all be referred to the description of the technical solution of the above-mentioned one problem-solving method.

[0238] Corresponding to the above method embodiment, this specification also provides a problem handling device embodiment, Figure 8 FIG. 1 is a schematic diagram showing a structure of a problem handling device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0239] The data analysis module 802 is configured to perform data analysis on the problem to be processed using the problem processing model to obtain reference text information corresponding to the problem to be processed;

[0240] The text acquisition module 804 is configured to determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0241] The answer prediction module 806 is configured to input the question to be processed and the reference text into an answer prediction model to obtain an initial answer corresponding to the question to be processed;

[0242] The answer detection module 808 is configured to use the question processing model to detect the relevance between the question to be processed and the initial answer, and determine the target answer corresponding to the question to be processed according to the relevance detection result.

[0243] Optionally, the data analysis module 802 is further configured to:

[0244] Inputting the problem to be processed into the problem processing model for semantic conversion to obtain the problem to be processed after semantic conversion;

[0245] Performing search and determination on the pending problem after the semantic conversion to obtain a search and determination result corresponding to the pending problem;

[0246] In the case where it is determined according to the search determination result that the reference text needs to be obtained, the reference text information corresponding to the problem to be processed is determined from the search determination result.

[0247] Optionally, the problem handling device further includes a problem input module, which is configured to:

[0248] When it is determined according to the search result that it is not necessary to obtain the reference text, the question to be processed is input into the answer prediction model to obtain an initial answer corresponding to the question to be processed.

[0249] Optionally, the text acquisition module 804 is further configured to:

[0250] Determining a unit identifier and text search information corresponding to the reference text from the reference text information;

[0251] Determining the target data acquisition unit based on the unit identifier, and determining the reference text to be detected associated with the problem to be processed from the target data acquisition unit based on the text search information, wherein the target data acquisition unit is a unit corresponding to the unit identifier among the multiple data acquisition units;

[0252] The problem processing model is used to perform quality inspection on the problem to be processed and the reference text to be inspected, and the reference text associated with the problem to be processed is determined according to the quality inspection result.

[0253] Optionally, the target data acquisition unit is a historical data storage unit, a target server, or at least two other servers;

[0254] The text acquisition module 804 is further configured to:

[0255] Determine a historical data storage unit based on the unit identifier, determine an associated historical reference text associated with the problem to be processed from the historical reference texts stored in the historical data storage unit based on the text search information, and determine the associated historical reference text as the reference text to be detected; or

[0256] Determine a target server based on the unit identifier, search service data in the target server based on the text search information, obtain a search text associated with the text search information, and use the search text as the reference text to be detected; or

[0257] At least two other service ends are determined based on the unit identifier, and based on the text search information, service data in each other service end is searched respectively to obtain other search texts associated with the text search information, and the other search texts are used as the reference texts to be detected.

[0258] Optionally, the text acquisition module 804 is further configured to:

[0259] Inputting the problem to be processed and the reference text to be tested into the problem processing model for quality testing to obtain a quality testing result;

[0260] In the case where it is determined that the quality inspection result is consistent with a first quality inspection result of at least two preset quality inspection results, determining the reference text to be inspected as a reference text associated with the problem to be processed;

[0261] When it is determined that the quality inspection result is consistent with the second quality inspection result of the at least two preset inspection results, based on the quality inspection result and the text search information, the reference text to be inspected associated with the problem to be processed is re-determined from the target data acquisition unit, and the problem to be processed and the reference text to be inspected are re-input into the problem processing model for quality inspection until the quality inspection result is consistent with the first quality inspection result.

[0262] Optionally, the text acquisition module 804 is further configured to:

[0263] The reference text acquisition model is used to determine the target data acquisition unit based on the unit identifier, and based on the text search information, the reference text to be detected associated with the problem to be processed is determined from the target data acquisition unit.

[0264] Optionally, the answer detection module 808 is further configured to:

[0265] Using the question processing model, detecting the relevance between the question to be processed and the initial answer to obtain the relevance detection result;

[0266] When it is determined that the relevance detection result is consistent with a first relevance detection result among at least two preset relevance detection results, the initial answer is determined as the target answer corresponding to the question to be processed.

[0267] Optionally, the data analysis module 802 is further configured to:

[0268] When it is determined that the relevance detection result is consistent with the second relevance detection result of the at least two preset relevance detection results, using the problem processing model to perform data analysis on the relevance detection result and the problem to be processed, obtain updated text information corresponding to the problem to be processed, and use the updated text information as reference text information corresponding to the problem to be processed;

[0269] The text acquisition module 804 is further configured to:

[0270] Continue to determine the target data acquisition unit based on the reference text information, and determine the reference text associated with the problem to be processed from the target data acquisition unit, until the relevance detection result is consistent with the first relevance detection result, and determine the target answer corresponding to the problem to be processed.

[0271] Optionally, the problem processing device further includes a model training module configured to:

[0272] Determine a sample question for the problem processing model to be trained, process the sample question using the answer prediction model, and obtain an initial sample reference text and an initial sample answer corresponding to the sample question;

[0273] Performing data screening on the initial sample answers to obtain target sample answers, and performing data screening on the initial sample reference text to obtain target sample reference text;

[0274] The problem processing model to be trained is trained using the sample question, the target sample reference text and the target sample answer to obtain the problem processing model.

[0275] Optionally, the problem handling device further includes a problem receiving module, which is configured to:

[0276] Receiving the pending question sent by the client, wherein the pending question is sent by the client when the user performs a question uploading operation on the client;

[0277] The question processing device further includes an answer sending module, which is configured to:

[0278] The target answer is sent to the client for display.

[0279] One or more embodiments of the present specification provide a question processing device. Before using an answer prediction model to predict an answer, the question processing model can be used to perform data analysis on the question to be processed, and based on the reference text information obtained from the data analysis, an associated reference text is determined for the question to be processed; then, by inputting the question to be processed and the reference text into the answer prediction model, the answer prediction model can predict the answer for the question to be processed based on the additional supplementary knowledge of the reference text, thereby avoiding the problem that the knowledge learned by the trained neural network is limited and an accurate answer may not be predicted based on the limited knowledge.

[0280] Moreover, after obtaining the initial answer corresponding to the problem to be processed, in order to further improve the accuracy of the answer, the problem processing model can also be used to detect the correlation between the problem to be processed and the initial answer, and determine the accurate target answer for the problem to be processed based on the correlation detection result, thereby avoiding the problem that the trained neural network may not be able to predict the accurate answer based on limited knowledge, and improving the accuracy of the answer.

[0281] The above is a schematic scheme of a problem solving device of this embodiment. It should be noted that the technical scheme of the problem solving device and the technical scheme of the problem solving method described above are of the same concept, and the details not described in detail in the technical scheme of the problem solving device can be found in the description of the technical scheme of the problem solving method described above.

[0282] Corresponding to the above method embodiment, this specification also provides another problem handling device embodiment, Fig. 9 FIG. 2 shows a schematic diagram of another problem handling device provided by an embodiment of the present specification. Fig. 9 As shown, the device is applied to a cloud-side device, including:

[0283] The data analysis module 902 is configured to receive the pending issues sent by the terminal side device, and perform data analysis on the pending issues using the issue processing model to obtain reference text information corresponding to the pending issues;

[0284] The text acquisition module 904 is configured to determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection;

[0285] The answer prediction module 906 is configured to input the question to be processed and the reference text into a large language model to obtain an initial answer corresponding to the question to be processed;

[0286] The answer detection module 908 is configured to detect the relevance between the question to be processed and the initial answer using the question processing model, and determine the target answer corresponding to the question to be processed according to the relevance detection result;

[0287] The answer sending module 910 is configured to send the target answer to the terminal device.

[0288] One or more embodiments of the present specification provide a question processing device applied to a cloud-side device. Before using a large language model to predict an answer, the question processing model can be used to perform data analysis on the question to be processed, and based on the reference text information obtained from the data analysis, an associated reference text is determined for the question to be processed; then, by inputting the question to be processed and the reference text into the large language model, the large language model can predict the answer to the question to be processed based on the additional supplementary knowledge of the reference text, thereby avoiding the problem that the knowledge learned by the trained neural network is limited and an accurate answer may not be predicted based on the limited knowledge.

[0289] Moreover, after obtaining the initial answer corresponding to the problem to be processed, in order to further improve the accuracy of the answer, the problem processing model can also be used to detect the correlation between the problem to be processed and the initial answer, and determine the accurate target answer for the problem to be processed based on the correlation detection result, thereby avoiding the problem that the trained neural network may not be able to predict the accurate answer based on limited knowledge, and improving the accuracy of the answer.

[0290] The above is a schematic scheme of another problem-solving device of this embodiment. It should be noted that the technical scheme of the another problem-solving device and the technical scheme of the another problem-solving method described above belong to the same concept, and the details of the technical scheme of the another problem-solving device that are not described in detail can all be referred to the description of the technical scheme of the another problem-solving method described above.

[0291] Fig.10 The block diagram of a computing device 1000 according to an embodiment of the present specification is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.

[0292] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0293] In one embodiment of the present specification, the above components of the computing device 1000 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0294] The computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1000 may also be a mobile or stationary server.

[0295] The processor 1020 is used to execute the following computer program / instruction, which implements the steps of any one of the above methods when executed by the processor.

[0296] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to any of the above method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of any of the above method embodiments.

[0297] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0298] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to any of the above method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of any of the above method embodiments.

[0299] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of any one of the above methods when executed by a processor.

[0300] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of any of the above methods belong to the same concept, and the details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of any of the above methods.

[0301] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0302] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0303] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0304] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0305] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A problem solving method, comprising: Using a question processing model to perform data analysis on the question to be processed to obtain reference text information corresponding to the question to be processed, wherein the question processing model is obtained by model training using training data generated by an answer prediction model, and the question processing model is used to implement answer prediction of the answer prediction model; Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection, and the target data acquisition unit is a historical data storage unit, a target server, or at least two other servers; Inputting the problem to be processed and the reference text into the answer prediction model to obtain an initial answer corresponding to the problem to be processed; The problem processing model is used to detect the relevance between the problem to be processed and the initial answer, and the target answer corresponding to the problem to be processed is determined according to the relevance detection result.

2. According to the problem processing method of claim 1, the step of using the problem processing model to perform data analysis on the problem to be processed to obtain reference text information corresponding to the problem to be processed comprises: Inputting the problem to be processed into the problem processing model for semantic conversion to obtain the problem to be processed after semantic conversion; Performing search and determination on the pending problem after the semantic conversion to obtain a search and determination result corresponding to the pending problem; In the case where it is determined according to the search determination result that the reference text needs to be obtained, the reference text information corresponding to the problem to be processed is determined from the search determination result.

3. According to the problem processing method of claim 2, after searching and determining the problem to be processed after the semantic conversion and obtaining the search and determination result corresponding to the problem to be processed, the method further comprises: When it is determined according to the search result that it is not necessary to obtain the reference text, the question to be processed is input into the answer prediction model to obtain an initial answer corresponding to the question to be processed.

4. The problem processing method according to any one of claims 1 to 3, wherein the step of determining a target data acquisition unit based on the reference text information and determining a reference text associated with the problem to be processed from the target data acquisition unit comprises: Determining a unit identifier and text search information corresponding to the reference text from the reference text information; Determining the target data acquisition unit based on the unit identifier, and determining the reference text to be detected associated with the problem to be processed from the target data acquisition unit based on the text search information, wherein the target data acquisition unit is a unit corresponding to the unit identifier among the multiple data acquisition units; The problem processing model is used to perform quality inspection on the problem to be processed and the reference text to be inspected, and the reference text associated with the problem to be processed is determined according to the quality inspection result.

5. The problem processing method according to claim 4, wherein the step of determining the target data acquisition unit based on the unit identifier and determining the reference text to be detected associated with the problem to be processed from the target data acquisition unit based on the text search information comprises: Determine a historical data storage unit based on the unit identifier, determine an associated historical reference text associated with the problem to be processed from the historical reference texts stored in the historical data storage unit based on the text search information, and determine the associated historical reference text as the reference text to be detected; or Determine a target server based on the unit identifier, search service data in the target server based on the text search information, obtain a search text associated with the text search information, and use the search text as the reference text to be detected; or At least two other service ends are determined based on the unit identifier, and based on the text search information, service data in each other service end is searched respectively to obtain other search texts associated with the text search information, and the other search texts are used as the reference texts to be detected.

6. The problem processing method according to claim 4, wherein the problem processing model is used to perform quality detection on the problem to be processed and the reference text to be detected, and the reference text associated with the problem to be processed is determined according to the quality detection result, comprising: Inputting the problem to be processed and the reference text to be tested into the problem processing model for quality testing to obtain a quality testing result; In the case where it is determined that the quality inspection result is consistent with a first quality inspection result of at least two preset quality inspection results, determining the reference text to be inspected as a reference text associated with the problem to be processed; When it is determined that the quality inspection result is consistent with the second quality inspection result of the at least two preset inspection results, based on the quality inspection result and the text search information, the reference text to be inspected associated with the problem to be processed is re-determined from the target data acquisition unit, and the problem to be processed and the reference text to be inspected are re-input into the problem processing model for quality inspection until the quality inspection result is consistent with the first quality inspection result.

7. The problem processing method according to any one of claims 1 to 3, wherein the problem processing model is used to detect the relevance between the problem to be processed and the initial answer, and a target answer corresponding to the problem to be processed is determined according to the relevance detection result, comprising: Using the question processing model, detecting the relevance between the question to be processed and the initial answer to obtain the relevance detection result; When it is determined that the relevance detection result is consistent with a first relevance detection result among at least two preset relevance detection results, the initial answer is determined as the target answer corresponding to the question to be processed.

8. The problem processing method according to claim 7, wherein the problem processing model is used to detect the relevance between the problem to be processed and the initial answer, and after obtaining the relevance detection result, the method further comprises: When it is determined that the relevance detection result is consistent with the second relevance detection result of the at least two preset relevance detection results, using the problem processing model to perform data analysis on the relevance detection result and the problem to be processed, obtain updated text information corresponding to the problem to be processed, and use the updated text information as reference text information corresponding to the problem to be processed; Continue to determine the target data acquisition unit based on the reference text information, and determine the reference text associated with the problem to be processed from the target data acquisition unit, until the relevance detection result is consistent with the first relevance detection result, and determine the target answer corresponding to the problem to be processed.

9. The problem processing method according to any one of claims 1 to 3, before using the problem processing model to perform data analysis on the problem to be processed and obtaining reference text information corresponding to the problem to be processed, further comprising: Determine a sample question for the problem processing model to be trained, process the sample question using the answer prediction model, and obtain an initial sample reference text and an initial sample answer corresponding to the sample question; Performing data screening on the initial sample answers to obtain target sample answers, and performing data screening on the initial sample reference text to obtain target sample reference text; The problem processing model to be trained is trained using the sample question, the target sample reference text and the target sample answer to obtain the problem processing model.

10. The problem processing method according to any one of claims 1 to 3, before using the problem processing model to perform data analysis on the problem to be processed and obtaining reference text information corresponding to the problem to be processed, further comprising: Receiving the pending question sent by the client, wherein the pending question is sent by the client when the user performs a question uploading operation on the client; After determining the target answer corresponding to the problem to be processed according to the relevance detection result, the method further includes: The target answer is sent to the client.

11. A problem handling method, applied to a cloud-side device, comprising: The receiving end-side device sends a question to be processed, and uses a question processing model to perform data analysis on the question to be processed to obtain reference text information corresponding to the question to be processed, wherein the question processing model is obtained by model training using training data generated by a large language model, and the question processing model is used to implement answer prediction of the large language model; Determine a target data acquisition unit based on the reference text information, and determine a reference text associated with the problem to be processed from the target data acquisition unit, wherein the reference text is a text that meets the text quality detection, and the target data acquisition unit is a historical data storage unit, a target server, or at least two other servers; Inputting the problem to be processed and the reference text into the large language model to obtain an initial answer corresponding to the problem to be processed; Using the problem processing model, the correlation between the problem to be processed and the initial answer is detected, and the target answer corresponding to the problem to be processed is determined according to the correlation detection result; The target answer is sent to the end-side device.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images