Intelligent agent language ability automatic testing method based on multi-language model self-evolution

Abstract

The invention provides an intelligent agent language ability automatic testing method based on multi-language model self-evolution. The intelligent agent language ability automatic testing method based on multi-language model self-evolution comprises the steps that associated information is acquired; performing similarity retrieval in a pre-constructed sample library to obtain a retrieval sample meeting a set requirement; constructing cue words; the prompt words are input into a voter, the voter comprises a plurality of voting language models, and each voting language model outputs a corresponding voting result; and based on the plurality of voting results, evaluating an output result of the intelligent agent to obtain a judgment result. The purpose of improving efficiency and accuracy is achieved. And when the confidence coefficient is high enough, the result is automatically incorporated into the sample library, and high-quality samples are continuously increased in a dynamic expansion manner, so that the generalization ability of the voting language model is improved, and the self-evolution of the model is realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an automated testing method for the language ability of an intelligent agent based on self-evolution of a multi-language model. Background Art

[0002] General artificial intelligence is to make intelligent agents like humans, not only able to independently perceive the environment, think, make decisions, learn new skills, and perform tasks, but also able to effectively collaborate with humans or other intelligent agents. At the same time, intelligent agents can understand human emotions and follow social ethics and moral norms.

[0003] When interacting with an intelligent agent through voice and the intelligent agent gives feedback (answer / action), how to judge whether the feedback is correct is an important technology for evaluating the intelligent agent.

[0004] At present, the following three methods are generally used to test the language ability of general artificial intelligence: Manual judgment results (manual testing): Testers judge whether the answers of the agent are accurate, reasonable, and natural based on a predefined set of questions through interaction with the agent. This method is usually suitable for complex dialogue scenarios and questions that cannot be verified by simple rules. It can simulate the usage scenarios of real users and give the most intuitive feedback.

[0005] Automated testing based on keyword rule verification of a specific corpus case library: This method builds a specific corpus or case library, predefines the templates of questions and answers, and verifies whether the agent's answers meet expectations through string matching. This method is often used to check some simple common sense questions and small talk questions. By setting a set of rules or keywords, when the agent's answer contains these keywords, it is judged to be correct.

[0006] Automated testing based on traditional natural language processing technology: Automated testing based on natural language processing (NLP) technology mainly understands the agent's answer through word segmentation, part-of-speech tagging, semantic analysis, syntactic analysis, intent recognition, action mapping, etc. Through these technologies, the test system can not only check whether the answer is accurate, but also determine whether the semantics match, whether the context is coherent, and even handle complex multi-round dialogues and ambiguous expressions. NLP technology uses semantic analysis to make agent testing no longer limited to simple string matching.

[0007] The inventors found that the prior art had the following disadvantages during the implementation of this embodiment: Manual judgment results: a) Low efficiency, especially in large-scale testing, and slow execution; b) Test results may be affected by the subjectivity of testers and lack consistency and objectivity.

[0008] Automated testing based on specific corpus keyword rule verification: a) Unable to handle the diversity of language expressions, relying on predefined templates, and unable to adapt to new scenarios and changes in new sentences. b) String matching cannot understand semantics, leading to potential misjudgments and insufficient test coverage.

[0009] Automated testing based on traditional natural language processing technology: a) It is complex to implement, requires a lot of computing resources, and has low efficiency for large-scale testing. b) The effectiveness and accuracy of NLP technology depends on the training quality of the model. If the model is not perfect, it may lead to deviations in understanding and affect the test results. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides an automated testing method for the language ability of an intelligent agent based on self-evolution of a multi-language model, which at least partially solves the problems of low efficiency and inaccurate test results existing in the prior art.

[0011] In a first aspect, the present disclosure provides an automated testing method for the language ability of an intelligent agent based on self-evolution of a multi-language model, comprising: Obtaining relevant information based on the output results obtained by the agent in response to user input; Based on the output results and the associated information, similarity search is performed in a pre-built sample library to obtain search samples that meet the set requirements; Construct prompt words based on output results, related information and retrieval samples; Input the prompt word into a voting machine, wherein the voting machine includes a plurality of voting language models, each voting language model votes on the input separately, and each voting language model outputs a corresponding voting result; Based on multiple voting results, the output results of the intelligent agent are evaluated to obtain the judgment result.

[0012] Optionally, the obtaining of associated information based on an output result obtained by the agent in response to a user input includes: Convert the user's input voice into corresponding text; Intent recognition is performed based on the text and user input is classified according to the scenario of the recognition result.

[0013] Optionally, the obtaining of associated information based on an output result obtained by the agent in response to a user input includes: Convert the voice response in the agent's output into the corresponding output text; Obtain scene information and the agent's action sequence based on the output text.

[0014] Optionally, in the similarity search in the pre-built sample library based on the output result and the associated information, the sample similarity selection includes: A length-based sample selector, a maximum marginal correlation-based sample selector, and / or a semantic similarity-based sample selector.

[0015] Optionally, the voter includes at least three voting language models.

[0016] Optionally, in the evaluation of the output result of the intelligent agent based on multiple voting results, when the voting results of the voting language models are different, the voting result with the same majority of voting results is selected.

[0017] Optionally, after the step of evaluating the output result of the intelligent agent based on multiple voting results to obtain the judgment result, the method further includes expanding the pre-constructed sample library based on the confidence score of the judgment result output by the voter.

[0018] Optionally, the pre-built sample library is expanded based on the confidence score of the decision result output by the voter, including: When the confidence is greater than the set value, the data related to the voting results will be stored in the sample library; when the confidence is not greater than the set value, the voting results will be placed in the annotation pool, and after manual evaluation and annotation of the voting results in the annotation pool, the data related to the evaluated and labeled voting results will be stored in the sample library.

[0019] In a second aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model as described in the first aspect.

[0020] In a third aspect, an embodiment of the present disclosure further provides a computer-readable storage medium storing computer instructions for causing a computer to execute any of the methods for automated testing of intelligent agent language capabilities based on self-evolution of a multi-language model as described in the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the methods for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model as described in the first aspect.

[0022] The present invention provides an automated testing method for the language ability of an intelligent agent, wherein the automated testing method for the language ability of an intelligent agent based on the self-evolution of a multi-language model obtains related information, then performs a similarity search in a pre-constructed sample library, constructs prompt words based on information such as the retrieved similar samples, and inputs the prompt words into a voter for voting. The language ability of the intelligent agent is automatically tested based on the voting results of the voter, and voting is performed through multiple voting language models to ensure higher reasoning accuracy and consistency, and the test is automatically completed, thereby achieving the purpose of improving efficiency and accuracy.

[0023] The output results are accompanied by a confidence score. When the confidence is high enough, the results are automatically included in the sample library, and high-quality samples are continuously added in a dynamic expansion manner, thereby improving the generalization ability of the voting language model and realizing the self-evolution of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0025] Figure 1 A flowchart of an automated testing method for the language ability of an intelligent agent based on self-evolution of a multi-language model provided in an embodiment of the present disclosure; Figure 2 A principle block diagram of an automated testing device for the language ability of an intelligent agent based on self-evolution of a multi-language model provided in an embodiment of the present disclosure; Figure 3 A functional block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0027] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0028] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0030] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0031] With the rapid development of artificial intelligence technology, the ability of intelligent agents in language understanding and generation has received increasing attention. The proposed method of automated testing of the language ability of intelligent agents based on the self-evolution of multilingual models aims to evaluate and verify the language processing ability of intelligent agents in an automated way. This method can not only save a lot of manpower and time costs, but also provide more objective and accurate test results.

[0032] The core of the automated testing method of the language ability of intelligent agents based on the self-evolution of multilingual models is to simulate the interaction process between users and intelligent agents and conduct a comprehensive evaluation of the language processing ability of the intelligent agents.

[0033] For ease of understanding, Figure 1 As shown, this embodiment discloses an automated testing method for the language ability of an intelligent agent based on self-evolution of a multilingual model, comprising: Step S101: acquiring related information based on the output result obtained by the agent in response to the user input; Obtaining relevant information based on the output results obtained by the agent in response to user input, including: Convert the user's input voice into corresponding text; Perform intent recognition based on text and classify user input based on the scenario of the recognition result; Speech-to-text technology is the process of converting user voice input into readable text information. The key components of speech-to-text technology are as follows: Speech signal preprocessing: In the process of speech-to-text conversion, the speech signal needs to be preprocessed first. This includes noise reduction, echo cancellation, gain control, etc., to improve the quality of the speech signal and ensure the accuracy of subsequent processing.

[0034] The main purpose of noise reduction is to reduce or eliminate the interference of background noise on speech signals. In daily life, the environment we live in is often accompanied by various noises, such as traffic noise, human voices, background noise of electronic devices, etc. If these noises are not processed, they will mix with the speech signal, making it difficult for the speech recognition system to accurately recognize and transcribe the speech content. Noise reduction technology analyzes audio input and identifies noise patterns, then applies filters to reduce or eliminate unwanted noise while preserving the voice signal. This allows the speech recognition system to capture the user's voice more clearly and improve recognition accuracy.

[0035] Noise reduction also helps improve the quality of speech signals, making speech-to-text results more accurate and reducing misunderstandings and errors caused by noise interference.

[0036] Echo cancellation technology is mainly used to deal with acoustic echo, which usually occurs in hands-free or conferencing applications when the sound of the speaker is fed back to the microphone multiple times. This echo will cause the speaker to hear his or her own speech for a short period of time, seriously affecting the call quality and the accuracy of speech recognition.

[0037] Echo cancellation technology ensures sound clarity and fidelity by identifying and eliminating audio echoes. It can intelligently identify and eliminate up to 99.9% of audio echoes, and multiple high-precision AI technologies separate near-end signals from echo signals to ensure accurate restoration of sound quality.

[0038] In the case of multi-person voice chat applications, turning on echo cancellation can significantly increase user activity time and engagement, because users can hear the other party's voice more clearly without being disturbed by echoes.

[0039] Feature extraction: The preprocessed speech signal needs to be converted into feature vectors, which can represent the key information of the speech signal. Common feature extraction methods include Mel frequency cepstral coefficients and Mel spectrum energy features.

[0040] Acoustic model: The acoustic model is the core of the speech-to-text system, which is responsible for mapping feature vectors to acoustic units (such as phonemes or letters). Deep learning technology, especially recurrent neural networks and convolutional neural networks, plays an important role in the construction of acoustic models.

[0041] Language model: The language model is used to predict the probability distribution of a text sequence. It helps the acoustic model select the most likely word sequence among multiple possible candidates. The language model can be based on a neural network, such as a long short-term memory network.

[0042] Post-processing: The post-processing step involves correcting the recognition results to improve accuracy. This may include word-level language model rearrangement, grammatical correction, and context-dependent adjustments.

[0043] Intent identification is usually based on natural language processing techniques, which analyze the user's text input and map it to predefined intent categories. This can be achieved through keyword matching, pattern recognition or machine learning methods, such as classification algorithms or deep learning models.

[0044] Entity recognition: Along with identifying intent, entity recognition is also an important step. Entity recognition aims to extract key information from user input, such as names, places, dates, etc., which are crucial to understanding the user's specific needs.

[0045] Scenario Classification: Based on the identified intents and entities, the user’s input can be classified into different scenarios. Scenario classification helps the agent provide more personalized and contextual responses. For example, if the user asks about the weather, the agent can classify it as a “weather query” scenario and provide the corresponding weather information.

[0046] Context management: In multi-turn conversations, context management is crucial to keep the conversation coherent and relevant. The agent needs to maintain the conversation state so that it can reference previous information in subsequent interactions.

[0047] Response Generation: Finally, based on the results of intent recognition and scene classification, the agent will generate a corresponding response. This may involve retrieving information, executing commands, or providing suggestions.

[0048] According to the final use case scenario, it is divided into common sense, chat, scenario and action command categories. For examples of different categories, see Table 1 and Table 2: Table 1. Example of language questions

[0049] Table 2: Action instruction example table

[0050] Convert the voice response in the agent's output into the corresponding output text; Obtain scene information and the agent's action sequence based on the output text.

[0051] For common sense questions, in order to verify whether the answer is correct, it is also necessary to obtain the current scene information. For action instruction questions, in order to verify whether the answers and behaviors are correct, it is also necessary to obtain information about the current scene and the action sequence of the agent.

[0052] After the agent responds to user input, it needs to obtain relevant information related to the output result. This information may include but is not limited to: The contextual information input by the user; the semantic information of the output results of the intelligent agent; the emotional tendency of the output results of the intelligent agent; and the grammatical structure of the output results of the intelligent agent.

[0053] The purpose of obtaining this information is to more accurately evaluate the output results of the agent and provide necessary data support for subsequent similarity retrieval.

[0054] Step S102: performing similarity search in a pre-built sample library based on the output results and associated information to obtain search samples that meet set requirements; The sample library records the user's questions, answers, and answers to each question, actions and corresponding contextual information, whether the result of each answer action is correct, the thinking process behind the result, and the confidence in the result.

[0055] Optionally, in the similarity search in the pre-built sample library based on the output result and the associated information, the sample similarity selection includes: A length-based sample selector, a maximum marginal correlation-based sample selector, and / or a semantic similarity-based sample selector.

[0056] Based on the output results and associated information of the agent, similarity retrieval is performed in the pre-built sample library. The samples contained in the sample library can be manually annotated or automatically generated by machine learning methods. The purpose of similarity retrieval is to find samples similar to the output results of the agent for further comparison and analysis.

[0057] The construction of the sample library needs to consider the following aspects: Diversity of samples: The sample library needs to contain a variety of language samples to cover different language scenarios; Representativeness of samples: The samples need to be representative of the various language situations that the agent may encounter; Sample annotation quality: Samples need to be accurately annotated to facilitate subsequent similarity retrieval and evaluation.

[0058] Based on the current question, answer and context information, the 10 most relevant samples can be retrieved from the sample library. For Chinese text, the samples selected by the semantic similarity sample selector are of higher quality and have better final results.

[0059] Step S103: constructing prompt words based on the output results, related information and search samples; The answer action and context information of the current question to be judged, as well as the reference answer selected from the sample library, are integrated into the prompt word template to generate the final prompt word.

[0060] After obtaining similar samples, it is necessary to construct prompt words based on the output results, related information, and retrieval samples. Prompt words are key information used to guide the voter to make decisions. They can be keywords, phrases, or entire sentences. The construction of prompt words needs to consider the following factors: Relevance of prompt words: The prompt words need to be highly relevant to the agent’s output and associated information; Prompt word coverage: The prompt word needs to cover all the key information of the agent's output as much as possible; Simplicity of prompt words: Prompt words need to be concise and clear so that the voting machine can understand and process them easily.

[0061] Step S104: inputting the prompt word into a voting device, wherein the voting device includes a plurality of voting language models, each voting language model votes on the input separately, and each voting language model outputs a corresponding voting result; The constructed prompt words are input into the voting machine, which contains multiple voting language models. Each model votes on the input separately and outputs the corresponding voting results. The voting language model can be a machine learning model based on different algorithms, such as support vector machine, random forest, neural network and deep learning model. Each voting language model has its own characteristics and advantages. By integrating multiple models, the accuracy and robustness of decision-making can be improved.

[0062] Step S105: Based on multiple voting results, the output results of the intelligent agent are evaluated to obtain a judgment result.

[0063] Before conducting automated testing, a series of evaluation criteria need to be set. These criteria can be: Accuracy criteria: how well the agent’s output matches the expected results; Consistency criterion: whether the agent's responses are consistent across contexts; Naturalness criterion: whether the agent's response conforms to human language habits; Reliability criterion: whether the agent's response is stable and reliable.

[0064] Optionally, the voter includes at least three voting language models.

[0065] In the evaluation of the output result of the intelligent agent based on multiple voting results, when the voting results of the voting language models are different, the voting results with the same majority are selected.

[0066] If the three voting language models have different judgments on the same question and answer, then the results of the two voting language models with the same judgment are selected as the final result. Even if a voting language model is wrong, you can still rely on the majority of correct versions to ensure normal operation, thereby achieving fault tolerance and improving the accuracy. It is found that after using three voting language models, the accuracy can be improved by about 10% compared to using only one voting language model.

[0067] Through a voting mechanism composed of multiple voting language models, the responses and behaviors of the intelligent agent are comprehensively evaluated, and the final judgment result is output to determine whether it is correct.

[0068] After the step of evaluating the output results of the intelligent agent based on multiple voting results to obtain the judgment result, the method further includes expanding the pre-constructed sample library based on the confidence score of the judgment result output by the voter.

[0069] The pre-built sample library is expanded based on the confidence score of the decision result output by the voter, including: When the confidence is greater than the set value, the data related to the voting results will be stored in the sample library; when the confidence is not greater than the set value, the voting results will be placed in the annotation pool. After manually evaluating and annotating the voting results in the annotation pool, the data related to the evaluated and annotated voting results will be stored in the sample library. The set value is set as needed. In a specific scenario, it can be set to 95 or 90, etc. The confidence range is between 0 and 100. The higher the value, the higher the confidence of the model in the predicted results. The data related to the voting results include questions, answers, context, reasoning logic, and judgment results.

[0070] Through the method of this embodiment, the sample library has the ability to self-expand and self-optimize without human intervention, achieving true self-evolution. According to the inventor's experiment, 20 data were used in the cold start phase. With the increase of running time, the scene and question and answer data are constantly enriched, and the sample library has accumulated about 500 data by itself, which significantly improves the accuracy of the entire test verification system. Compared with the traditional model system, it greatly reduces the dependence on the initial data volume, can quickly converge on a small amount of data, achieve significant results, and realize more efficient data utilization and model generalization capabilities.

[0071] Based on multiple voting results, the output of the agent is comprehensively evaluated. The evaluation includes: The accuracy of the agent’s output: evaluates whether the agent can accurately understand the user’s intention and give the correct response; Consistency of agent output: assess whether the agent can maintain consistent responses in different contexts; Naturalness of the agent’s output: evaluate whether the agent’s response is natural and fluent, and conforms to human language habits; Reliability of the agent’s output: Evaluate whether the agent’s response is stable and reliable, and not affected by external factors.

[0072] The voting machine can adopt models generated by deep learning algorithms, neural network algorithms, etc., which have a certain understanding and reasoning ability of problems, etc., and can refer to the existing Gpt, Tongyi Qianwen and Gemini, etc.

[0073] Deep learning and neural network algorithms have made significant progress in simulating human understanding and reasoning capabilities of language. The development of these technologies has made it possible to build intelligent systems that can handle complex language tasks.

[0074] Deep learning algorithms are a series of algorithms based on artificial neural networks. These algorithms learn complex patterns in data by imitating the way the human brain processes information. Deep learning models usually contain multiple layers, each layer consists of multiple neurons, which can learn the characteristics of different levels of data. The core of deep learning algorithms is to automatically extract features without manually designing feature extraction rules.

[0075] The development of neural network algorithms can be traced back to the 1940s, when scientists tried to simulate the neural network of the human brain. Over time, neural network algorithms have experienced many rises and falls. In the early 21st century, with the improvement of computing power and the availability of big data, deep learning algorithms began to achieve breakthrough progress in areas such as image recognition, speech recognition, and natural language processing.

[0076] The application of deep learning algorithms in language understanding is mainly concentrated in the field of natural language processing. NLP is a branch of artificial intelligence, and its key applications are in the following areas: Text classification: Deep learning models can be trained to identify the sentiment of text or to classify text into predefined categories.

[0077] Machine Translation: Deep learning models, especially those based on attention mechanisms, have significantly improved the accuracy and fluency of machine translation.

[0078] Question Answering: Deep learning models are able to understand questions and extract answers from the given text, which is very useful in question answering.

[0079] Text Summarization: Deep learning models can automatically generate summaries of text, which is very valuable for processing large amounts of information.

[0080] Conversational Systems: Deep learning models can be used to build chatbots and virtual assistants that can understand user input and provide appropriate responses.

[0081] Deep learning models learn hierarchical features of data through multi-layer structures. In text processing, this usually means that the model first learns the embedding representation of words, then the semantics of sentences, and finally the topic of the entire document. This hierarchical representation enables the model to capture the complexity and diversity of language.

[0082] Word embedding: Word embedding is a vector representation that maps words to a high-dimensional space. These vectors capture the semantic information of the words. For example, "cat" and "dog" may be closer in the vector space than "cat" and "chair".

[0083] Sentence and paragraph representations: Based on word embeddings, deep learning models can further learn sentence and paragraph representations. These representations can capture the structure and semantic information of sentences.

[0084] Document and topic representations: For longer texts such as articles or books, deep learning models can learn document-level representations that capture the theme and structure of the entire document.

[0085] Attention mechanism is an important concept in deep learning models, which allows the model to focus on the most relevant parts of the current task when processing sequence data. In machine translation, attention mechanism can help the model pay attention to related words in the source language when translating a word. In text summarization, attention mechanism can help the model determine which sentences are key to generating summaries.

[0086] Contextual understanding is a key capability of deep learning models when dealing with language tasks. By using models such as Transformer, long-range dependencies and contextual information can be better captured, which is crucial for understanding the overall meaning of a sentence or paragraph. The Transformer model achieves this through the self-attention mechanism, which allows the model to consider the entire input sequence when processing a word.

[0087] Many modern NLP models learn general language representations by pre-training on large-scale datasets and then fine-tuning them on specific tasks, which enhances the model's ability to understand problems in specific fields. The pre-training phase allows the model to learn rich language patterns, while the fine-tuning phase enables the model to adapt to specific tasks.

[0088] Models used to solve specific reasoning tasks, such as question-answering systems and dialogue systems, can generate answers by analyzing questions and contextual information, demonstrating a certain degree of reasoning ability. This reasoning ability is achieved through the language patterns and structures learned by the model, rather than by simulating the human logical reasoning process.

[0089] The automated testing process disclosed in this embodiment includes the following steps: User input simulation: simulate the interaction between the user and the agent and input test samples; Agent response: The agent processes the user input and gives a response; Related information acquisition: obtain the related information of the agent's response; Similarity search: perform similarity search in the sample library to find similar samples; Prompt word construction: construct prompt words based on response results and similar samples; Voting machine decision: input the prompt words into the voting machine to make a decision; Result evaluation: Evaluate the output of the agent based on the voting results.

[0090] Result analysis and optimization: After completing the automated test, the test results need to be analyzed. The analysis includes: Statistical analysis of test results: statistical analysis of the agent’s performance on various evaluation criteria; Comparative analysis of test results: compare the performance of the agent on different test samples; Problem diagnosis of test results: diagnose problems that the agent encountered during testing and provide suggestions for improvement; Optimization of test process: Based on the test results, optimize the test process to improve the accuracy and efficiency of the test.

[0091] The method for automated testing of the language ability of intelligent agents based on the self-evolution of multilingual models disclosed in this embodiment can comprehensively evaluate the language processing ability of intelligent agents. By simulating the interaction between users and intelligent agents, related information is automatically obtained, similarity retrieval is performed, prompt words are constructed, and a voter is used to make decisions, and finally the output results of the intelligent agent are evaluated. This method not only improves the efficiency and objectivity of the test, but also can monitor the performance of the intelligent agent in real time and make dynamic adjustments. However, this method also faces challenges such as sample library construction, voting language model selection, test result interpretation, and security and privacy protection. Future research and practice need to conduct in-depth exploration in these aspects to further improve the accuracy and reliability of the automated testing method for the language ability of intelligent agents based on the self-evolution of multilingual models.

[0092] This embodiment has the following advantages: 1. Using the voting language model to determine the test results of the intelligent agent's language ability: In the test of the intelligent agent's language ability, the voting language model is used to automatically determine the test results, which effectively improves the intelligence and adaptability of the determination, and also significantly expands the range of statements that support verification, making the test coverage wider and the verification statements richer.

[0093] 2. Voting mechanism of multiple voting language models: By mixing multiple voting language models to form a voter, the accuracy of reasoning is significantly improved. Compared with a single model, the voting method has stronger fault tolerance and consistency in complex language understanding and reasoning tasks. In the automated testing of the language ability of general artificial intelligence entities, multiple voting language models are used to form a voter to judge the test results, thereby significantly improving the accuracy and robustness of the judgment.

[0094] 3. Self-expanding sample library mechanism: A high-confidence output mechanism based on the voting language model is designed, and the confidence score is included in the output result. When the confidence is high enough, the system automatically incorporates the judgment result into the sample library, thereby realizing the self-expansion and dynamic enrichment of the sample library. Without human intervention, the generalization ability and adaptability can be gradually enhanced as the scene and question and answer content grow, and continuous evolution can be achieved. Automatic expansion mechanism of voting language model samples with high-confidence results: Based on the voting language model, the judgment result and confidence are given at the same time, and the high-confidence results are directly included in the sample library, so that the sample library can achieve self-expansion, dynamic growth and continuous optimization in actual application scenarios, and realize intelligent sample library management without human intervention, thereby effectively improving the generalization ability and evolution speed of the system.

[0095] like Figure 2 As shown, this embodiment also discloses an automatic testing device for the language ability of an intelligent agent based on self-evolution of a multi-language model, including: An information module, used to obtain relevant information based on the output results obtained by the agent in response to the user input; A retrieval module, used to perform similarity retrieval in a pre-built sample library based on the output results and associated information to obtain retrieval samples that meet set requirements; A construction module, used for constructing prompt words based on output results, related information and retrieval samples; A voting result module, used for inputting the prompt word into a voting machine, wherein the voting machine includes a plurality of voting language models, each of which votes on the input separately, and each of which outputs a corresponding voting result; The judgment module is used to evaluate the output results of the intelligent agent based on multiple voting results to obtain a judgment result.

[0096] The step of obtaining the associated information based on the output result obtained by the agent in response to the user input includes: Convert the user's input voice into corresponding text; Intent recognition is performed based on the text and user input is classified according to the scenario of the recognition result.

[0097] The step of obtaining the associated information based on the output result obtained by the agent in response to the user input includes: Convert the voice response in the agent's output into the corresponding output text; Obtain scene information and the agent's action sequence based on the output text.

[0098] In the similarity search in the pre-built sample library based on the output result and the associated information, the sample similarity selection includes: A length-based sample selector, a maximum marginal correlation-based sample selector, and / or a semantic similarity-based sample selector.

[0099] The voting device includes at least three voting language models.

[0100] In the evaluation of the output result of the intelligent agent based on multiple voting results, when the voting results of the voting language models are different, the voting results with the same majority are selected.

[0101] After the step of evaluating the output results of the intelligent agent based on multiple voting results to obtain the judgment result, the method further includes expanding the pre-constructed sample library based on the confidence score of the judgment result output by the voter.

[0102] The pre-built sample library is expanded based on the confidence score of the decision result output by the voter, including: When the confidence is greater than the set value, the data related to the voting results will be stored in the sample library; when the confidence is not greater than the set value, the voting results will be placed in the annotation pool, and after manual evaluation and annotation of the voting results in the annotation pool, the data related to the evaluated and labeled voting results will be stored in the sample library.

[0103] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0104] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the aforementioned method for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model in each embodiment of the present disclosure.

[0105] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0106] like Figure 3 The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0107] like Figure 3 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0108] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0109] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the method for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model of an embodiment of the present disclosure are executed.

[0110] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0111] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the aforementioned method for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model in each embodiment of the present disclosure are executed.

[0112] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0113] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0114] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0115] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0116] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0117] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0118] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0119] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0120] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An automated testing method for the language ability of an intelligent agent based on self-evolution of a multilingual model, characterized in that: include: Obtaining relevant information based on the output results obtained by the agent in response to the user input; Based on the output results and the associated information, similarity search is performed in a pre-built sample library to obtain search samples that meet the set requirements; Construct prompt words based on output results, related information and retrieval samples; Input the prompt word into a voting machine, wherein the voting machine includes a plurality of voting language models, each voting language model votes on the input separately, and each voting language model outputs a corresponding voting result; Based on multiple voting results, the output results of the intelligent agent are evaluated to obtain the judgment result.

2. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 1, characterized in that: The step of obtaining the associated information based on the output result obtained by the agent in response to the user input includes: Convert the user's input voice into corresponding text; Intent recognition is performed based on the text and user input is classified according to the scenario of the recognition result.

3. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 1, characterized in that: The step of obtaining the associated information based on the output result obtained by the agent in response to the user input includes: Convert the voice response in the agent's output into the corresponding output text; Obtain scene information and the agent's action sequence based on the output text.

4. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 1, characterized in that: In the similarity search in the pre-built sample library based on the output result and the associated information, the sample similarity selection includes: A length-based sample selector, a maximum marginal correlation-based sample selector, and / or a semantic similarity-based sample selector.

5. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 1, characterized in that: The voting device includes at least three voting language models.

6. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 5, characterized in that: In the evaluation of the output result of the intelligent agent based on multiple voting results, when the voting results of the voting language models are different, the voting results with the same majority are selected.

7. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 1, characterized in that: After the step of evaluating the output results of the intelligent agent based on multiple voting results to obtain the judgment result, the method further includes expanding the pre-constructed sample library based on the confidence score of the judgment result output by the voter.

8. The method for automated testing of agent language ability based on multilingual model self-evolution according to claim 7, characterized in that: The pre-built sample library is expanded based on the confidence score of the decision result output by the voter, including: When the confidence is greater than the set value, the data related to the voting results will be stored in the sample library; when the confidence is not greater than the set value, the voting results will be placed in the annotation pool, and after manual evaluation and annotation of the voting results in the annotation pool, the data related to the evaluated and labeled voting results will be stored in the sample library.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the automated testing method for the language ability of an intelligent agent based on self-evolution of a multi-language model as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the method for automated testing of the language ability of an intelligent agent based on self-evolution of a multi-language model as described in any one of claims 1-8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for automatically testing the language ability of an intelligent agent based on self-evolution of a multi-language model as described in any one of claims 1-8 is implemented.

Images