Test case generation method, test case generation device, storage medium and electronic equipment

Abstract

The embodiment of the invention relates to a test case generation method, a test case generation device, a storage medium and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a to-be-tested dialogue service; the to-be-tested dialogue service is a service that a first language model provides a dialogue for a user; obtaining first dialogue information of a first dialogue party generated through the first language model; generating first prompt information according to the first dialogue information and feature information of a second dialogue party; processing the first prompt information through a second language model to generate second dialogue information of the second dialogue party; and based on the first dialogue information and the second dialogue information, determining a test case for the to-be-tested dialogue service. According to the method, the accurate and effective test case can be conveniently generated.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a test case generation method, a test case generation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] With the rapid development of artificial intelligence technology, in order to meet the needs of various scenarios, conversational language models are widely used in the fields of chatting, shopping guides, assistants, professional analysis, etc. In order to ensure that the language model can accurately and reasonably give responses based on the user's input, it is often necessary to test and verify the language model. The method used in the prior art is often to have testers write a large amount of test data and input it into the model to be tested to test it. However, this method not only requires a large amount of manpower and time costs, but also the manually written test data is relatively single, and it cannot be flexibly adjusted according to the state of the model to be tested during the test process, and it is difficult to ensure the accuracy of the test. Summary of the invention

[0004] However, the current existing technology relies on manual generation of test cases, which has high manpower and time costs, and it is difficult to ensure the rationality and effectiveness of the generated test cases.

[0005] Therefore, there is a great need for a test case generation method that can generate accurate and effective test cases conveniently and efficiently.

[0006] In this context, embodiments of the present disclosure are intended to provide a test case generating method, a test case generating device, a computer-readable storage medium, and an electronic device.

[0007] According to a first aspect of the present disclosure, a test case generation method is provided, comprising: determining a dialogue service to be tested; the dialogue service to be tested is a service that provides dialogue for users by a first language model; obtaining first dialogue information of a first dialogue party generated by the first language model; generating first prompt information according to the first dialogue information and feature information of a second dialogue party; processing the first prompt information by a second language model to generate second dialogue information of the second dialogue party; and determining a test case for the dialogue service to be tested based on the first dialogue information and the second dialogue information.

[0008] According to a second aspect of the present disclosure, a test case generating device is provided, including: a dialogue service determining module, used to determine a dialogue service to be tested; the dialogue service to be tested is a service that provides dialogue for users by a first language model; a dialogue information acquiring module, used to acquire first dialogue information of a first dialogue party generated by the first language model; a prompt information generating module, used to generate first prompt information according to the first dialogue information and feature information of a second dialogue party; a prompt information processing module, used to process the first prompt information by a second language model to generate second dialogue information of the second dialogue party; and a test case generating module, used to determine a test case for the dialogue service to be tested based on the first dialogue information and the second dialogue information.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the test case generation method of the first aspect and its possible implementation methods are implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the test case generation method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0011] In the scheme disclosed herein, on the one hand, this exemplary embodiment proposes a new test case generation method. Compared with the prior art method of writing test data which requires a large amount of manpower and time costs, this exemplary embodiment uses a second language model to process the first prompt information, and can generate second dialogue information for dialogue testing and test cases for the dialogue service to be tested. This process requires less human participation and can generate test cases efficiently, conveniently and accurately. On the other hand, in this exemplary embodiment, the first prompt information is generated based on the first dialogue information of the first dialogue party and the feature information of the second dialogue party generated by the first language model, which integrates the relevant information of the first dialogue party and the second dialogue party, so that the generated second dialogue information can be better applied to the dialogue scenario. Compared with the test data written manually, the second dialogue information in this exemplary embodiment is more simulated and reasonable, which improves the pertinence of the test case to different dialogue services to be tested in different application scenarios, and further ensures the accuracy of the language model test. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment.

[0013] Figure 2A flow chart of a test case generating method in this exemplary embodiment is shown.

[0014] Figure 3 A schematic diagram of a visualization interface for generating first prompt information in this exemplary embodiment is shown.

[0015] Figure 4 A schematic diagram of a test case in this exemplary embodiment is shown.

[0016] Figure 5 A schematic structural diagram of a test case generating device in this exemplary embodiment is shown.

[0017] Figure 6 A schematic structural diagram of an electronic device in this exemplary embodiment is shown.

[0018] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0019] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] The embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0021] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0023] The inventors have found that the cost of generating test cases for language models is currently high, and it is difficult to ensure accuracy. Specifically, in the related art, a language model is usually tested manually, in which professionals simulate real users to have conversations with the language model to implement business testing; in other cases, test cases can also be generated by manually writing test data or collecting random corpus from data sources. However, no matter which method is used, the generation method of this test case is relatively fixed and single, which requires high manpower and time costs, and cannot generate effective and accurate test cases based on actual conditions and the characteristics of the language model.

[0024] In view of the above, the present disclosure provides a test case generation method, a test case generation device, a computer-readable storage medium and an electronic device. On the one hand, the exemplary embodiment proposes a new test case generation method. Compared with the prior art method of writing test data, which requires a large manpower cost and time cost, the exemplary embodiment uses a second language model to process the first prompt information, and can generate a second dialogue information for dialogue testing and a test case for the dialogue service to be tested. The process requires less human participation and can generate test cases efficiently, conveniently and accurately. On the other hand, in the exemplary embodiment, the first prompt information is generated according to the first dialogue information of the first dialogue party and the feature information of the second dialogue party generated by the first language model, which integrates the relevant information of the first dialogue party and the second dialogue party, so that the generated second dialogue information can be better applied to the dialogue scenario. Compared with the test data written manually, the second dialogue information in the exemplary embodiment is more simulated and reasonable, which improves the pertinence of the test case to different dialogue services to be tested in different application scenarios, and further ensures the accuracy of the language model test.

[0025] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0026] Application Scenario Overview

[0027] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0028] The embodiments of the present disclosure can be applied to relevant scenarios for testing the output data of a language model. Less human involvement is required, and accurate test cases can be generated quickly and conveniently. The application scenarios are described in detail below in conjunction with the system architecture.

[0029] Figure 1A schematic diagram of a system architecture of a test case generation method is shown. The system architecture includes a first terminal 110 and a second terminal 120, wherein the first terminal 110 and the second terminal 120 can be devices such as computers, mobile phones, and servers. The first terminal 110 is configured with a first language model, which can be used to generate a first dialogue message, and the second terminal 120 is configured with a second language model, which can be used to generate a second dialogue message in reply to the first dialogue message. The first dialogue message and the second dialogue message can determine the test case of the dialogue service to be tested. In this exemplary embodiment, the first terminal 110 and the second terminal 120 can be different terminals, for example, the first language model and the second language model are respectively configured in different devices, the first dialogue message is first generated by the first language model on the first terminal, and then the first dialogue message and the characteristic information of the second dialogue party are sent to the second terminal, so that the second language model on the second terminal generates the second dialogue message; the first terminal 110 and the second terminal 120 can also be the same terminal, for example, the first language model and the second language model can be configured on the same terminal, and the first dialogue message and the second dialogue message are respectively generated using different language models on the same terminal, etc., and the present disclosure does not make specific limitations on this.

[0030] Exemplary Methods

[0031] The exemplary embodiment of the present disclosure provides a test case generation method. Figure 2 As shown, the method may include steps S210 to S250. Figure 2 Provide detailed instructions for each step.

[0032] refer to Figure 2 In step S210, the dialogue service to be tested is determined; the dialogue service to be tested is a service in which the first language model provides dialogue for the user.

[0033] The dialogue service to be tested may be a dialogue service that needs to be tested in any scenario, such as a consulting service provided by a consulting AI (Artificial Intelligence) program, a psychological counseling service provided by a psychological counseling AI program, a dating service provided by a dating program, a chat service provided by an AI chat assistant, a business analysis service provided by an AI professional, etc. The first language model refers to an AI model used to provide dialogue services, such as a consulting AI model, a psychological counseling AI model, a dating AI model, etc. According to different service purposes, the first language model can provide different types of business services, such as "chat type", "shopping guide type", "assistant type", and "expert type". The first language model can have a dialogue with the user. For example, the user can input his confusion to the psychological counseling AI model in the form of text or voice, and the psychological counseling AI model will give a reply based on the user's input content. In this process, the service and reply content provided by the psychological counseling AI model to the user can be regarded as the dialogue service provided by the psychological counseling AI model to the user. This exemplary embodiment needs to test the service to determine whether the first language model can reasonably, accurately and effectively give a reply in the expected direction according to the user's input content in the dialogue scenario.

[0034] In step S220, first dialogue information of a first dialogue party generated by a first language model is obtained.

[0035] Among them, the first dialogue party can be regarded as the first language model party. For example, when a user has a conversation with a psychological counseling AI model, the participants in the conversation are the user and the psychological counseling AI model. The psychological counseling AI model, as a dialogue participant, can be regarded as a dialogue party. The content output or reply of the first language model is the first dialogue information.

[0036] The first dialogue information may be text or voice information. The first dialogue information may be dialogue information of one round, such as text information given by the first language model in the current round. The first dialogue information may also be dialogue information of multiple rounds, such as text information given by the first language model in the current round and the previous round.

[0037] This exemplary embodiment may store or record the first dialogue information when the first language model generates and outputs it each round, so that the first dialogue information can be directly exported in the scenario of test case generation, or a specific interface may be configured for the first language model to obtain the first dialogue information, etc.

[0038] In step S230, first prompt information is generated according to the first dialogue information and the characteristic information of the second dialogue party.

[0039] Among them, the second party refers to the party participating in the same conversation with the first party. For example, when a user has a conversation with the first language model, the user is the second party. In this exemplary embodiment, the second party can be simulated by another language model. For example, when testing the service of the psychological counseling AI model, the psychological counseling AI model is the first party and the test language model is the second party. The test language model can simulate the user's conversation with the psychological counseling AI model.

[0040] The characteristic information of the second dialogue party may be information reflecting the basic attributes or additional attributes of the second dialogue party, such as the basic attribute information of the user or the simulated user, such as name, gender, age, occupation, etc., and may also include additional attribute information of the user or the simulated user, such as preferences, preferences, character, personality, tone, temperament, personality, language style, etc. This exemplary embodiment may adjust or enrich the characteristic information of the second dialogue party according to the application scenario or actual needs.

[0041] In this exemplary embodiment, the first dialogue information may include a variety of feature information, for example, the intention, language style, emotion, etc. of the first dialogue party may be mined from the first dialogue information, and the current dialogue topic may be determined in combination with the current pending business. In this exemplary embodiment, first prompt information may be generated based on the first dialogue information and the feature information of the second dialogue party, wherein the first prompt information refers to prompt information for inputting the second language model to generate the second dialogue information, that is, how to reply to the first dialogue information generated by the first language model may be determined based on the first prompt information.

[0042] In step S240, the first prompt information is processed by a second language model to generate second dialogue information of the second dialogue party.

[0043] Among them, the second language model can be a model for generating reply content for the first language model, and the generated second dialogue information can be used to reply the first dialogue information generated by the first language model, which can be regarded as attack data for "attacking" the first language model. Since the second dialogue information is generated according to the first prompt information, it can include feature information related to the first dialogue information and feature information of the second dialogue party, so that the generated second dialogue information can include more dimensional information, for example, the second dialogue information can be information containing specific intentions, emotions or information that conforms to the characteristics of the second dialogue party, or information suitable for the current context or attitude given according to the attitude of the first dialogue party. After receiving the second dialogue information of the second dialogue party, the first language model can continue to give reply information, and then determine whether it has given accurate, effective and reasonable reply content according to the reply information, so as to test and evaluate the business processing quality of the first language model. This exemplary embodiment can generate attack information for the first language model through the second language model, and "attack" the first language model to achieve the purpose of testing the model through the model. Compared with model testing in a manual way, it is more efficient and requires less human participation.

[0044] In step S250, a test case for the dialog service to be tested is determined based on the first dialog information and the second dialog information.

[0045] Finally, based on the first dialogue information generated by the first dialogue model and the second dialogue information given by the second language model based on the first dialogue information, a test case for the dialogue service to be tested can be formed. The test case is dialogue information including the first dialogue party and the second dialogue party in the dialogue scenario, which can be one round of dialogue information or multiple rounds of dialogue information. Based on the test case, the test result of the first language model can be further analyzed and determined, such as whether it meets the expected dialogue effect, meets the basic dialogue requirements, etc.

[0046] In an exemplary embodiment, the test case generation may further include:

[0047] Obtaining the dialog environment information of the dialog service to be tested;

[0048] Generating the first prompt information according to the first dialogue information and the characteristic information of the second dialogue party may include:

[0049] First prompt information is generated according to the first dialogue information, the feature information of the second dialogue party and the dialogue environment information.

[0050] Among them, dialogue environment information refers to environmental information related to the dialogue business to be tested, such as the current dialogue scenario, such as the dialogue environment, dialogue occasion, dialogue atmosphere, or the context information of the current dialogue, such as historical dialogue information, etc., or other vertical knowledge such as related inputs and related matters that can be obtained from the external environment.

[0051] This exemplary embodiment can abstract the business process of the dialogue business to be tested into the process of "characters talking within a context", and convert such complex systems into mathematical models, wherein the first dialogue information and the characteristic information of the second dialogue party can be abstracted as i, the dialogue environment information can be abstracted as e, the second dialogue party can be f, and the first prompt information can be represented by f(i, e). Furthermore, the second dialogue information generated based on the first prompt information, that is, the reply content of the second dialogue party can be represented by y. It can be determined that after the first dialogue party gives the first dialogue information, the second dialogue model can generate the second dialogue information y according to y=f(i, e), that is, the second dialogue party can be mixed with the first dialogue party information, personal intentions, emotions, environment and other strategic information, and give a reply to the second dialogue information in a unique language style.

[0052] In an exemplary embodiment, the generating of the first prompt information according to the first dialogue information and the characteristic information of the second dialogue party may include:

[0053] Determining the intention of the first dialogue party according to the first dialogue information;

[0054] First prompt information is generated according to the intention and the characteristic information of the second dialogue party.

[0055] The first dialogue information may include the intention of the first dialogue party. For example, when the chat AI clone pretends to be a real person to communicate with the user, it may output the first dialogue information "The weather is so nice today, where do you want to go for fun?". Analysis can be performed based on the first dialogue information to determine the intention of the first dialogue party, such as the intention of "asking the user where to go for fun" or "what are your plans for today", etc., and then the first prompt information can be generated based on the intention and the characteristic information of the second dialogue party. For example, the characteristic information of the second dialogue party is the name "Xiao Su" and the occupation "student", and the generated prompt information can be a scene in which the student Xiao Su chats with his friends to discuss what to do today, etc. in a simulated real-life chat.

[0056] The intention of the first dialogue party can be determined according to the first dialogue information in a variety of ways. For example, an intention determination model can be trained using historical information as training data, or a network layer capable of intent recognition can be configured in the second language model, so as to identify the intention of the first dialogue party after obtaining the first dialogue information. In addition, keywords can also be extracted from the first dialogue information, and the intention that the first dialogue information may correspond to can be determined by matching the keywords with a preset intent list, etc. This disclosure does not make specific limitations on this.

[0057] In an exemplary embodiment, the test case generation may further include:

[0058] Obtain the target conversation topic corresponding to the conversation service to be tested;

[0059] Generate first prompt information according to the first dialogue information and the characteristic information of the second dialogue party, including:

[0060] First prompt information is generated according to the first dialogue information, the characteristic information of the second dialogue party and the target dialogue topic.

[0061] The conversation topic refers to the main theme of the conversation. Different environments or different businesses can correspond to different topics. Since the first language model can usually conduct conversation business in a specific direction, the business to be processed can have a corresponding target conversation topic. For example, the AI ​​conversation assistant for pets handles business related to pets, and its topics can be pets and animals. The conversation content can be, for example, "pet breeds and breeding", "pet companionship and growth", etc.; the AI ​​conversation assistant for psychological counseling can handle business related to psychological emotions, and its topics can be psychology and emotions, etc. Different businesses to be processed can correspond to one or more topics. For example, the topics of academic AI conversation assistants can include "history and culture", "science and technology" and other topics.

[0062] According to actual scenarios or needs, the conversation topics can be divided into multiple categories. Specifically, they can also be in the form of multi-level categories. For example, taking the topic "Pets and Animals" as an example, it can be further refined into multiple secondary categories such as "Choice of Cute Pets", "Pet Experience", "Pet Maintenance" and "Pet Knowledge". Table 1 shows 20 examples of conversation topics divided by this exemplary embodiment:

[0063] Table 1

[0064]

[0065] This exemplary embodiment can generate first prompt information based on the first dialogue information, the characteristic information of the second dialogue party and the target dialogue topic, wherein the first prompt information can include prompt information determined according to the target dialogue topic. For example, when the topic is determined to be "Pets and Animals", scene prompt information related to the topic can be further generated, such as the scene - Pet Health Consultant (online pet health consultation group). After breakfast (time), you see that everyone is discussing heatedly in the pet health group (environment). Do you have any new pet health care knowledge you want to share (topic), such as how to keep your pet's fur shiny.

[0066] In an exemplary embodiment, the test case generation may further include:

[0067] Acquire historical conversation information between the first conversation party and the second conversation party;

[0068] Determining interpersonal relationship characteristics between the first dialogue party and the second dialogue party from the historical dialogue information;

[0069] The above-mentioned acquisition of the target conversation topic corresponding to the conversation service to be tested includes:

[0070] According to the interpersonal relationship characteristics, a target dialogue topic is determined from candidate dialogue topics corresponding to the dialogue business to be tested.

[0071] Considering that in actual scenarios, multiple target conversation topics may be determined, for example, the same pending business may correspond to multiple topics. In order to further accurately determine the current conversation topic, this exemplary embodiment may also obtain historical conversation information of the first conversation party and the second conversation party, and determine the interpersonal relationship characteristics between the first conversation party and the second conversation party from the historical conversation information. The interpersonal relationship characteristics may be, for example, stranger relationship, friend relationship, family relationship, etc. The interpersonal relationship characteristics between the first conversation party and the second conversation party may be determined from the historical conversation information in a variety of ways, such as extracting keywords from the historical information and determining the interpersonal relationship characteristics by analyzing the keywords; or a mapping table of interpersonal relationship characteristics and conversation topics may be pre-configured, and the human-machine relationship characteristics between the current first conversation party and the second conversation party may be determined by performing a search in combination with the content of the mapping table through statistical analysis and detection identification of the historical conversation information.

[0072] Considering that the topics that can be discussed in different interpersonal relationships are different, for example, family and friends can discuss more private topics, while strangers can discuss fewer topics. This exemplary embodiment can determine the target conversation topic from the candidate conversation topics corresponding to the conversation service to be tested based on the interpersonal relationship characteristics, that is, this exemplary embodiment can first determine multiple candidate conversation topics based on the conversation service to be processed. For example, the candidate conversation topics can be food, travel, and family life. Then, after determining the interpersonal relationship between the first and second parties, the candidate conversation topics can be further screened. If the interpersonal relationship is a stranger relationship, the topic of family life will be screened out, and the target conversation topic is finally determined to be food, or food and travel, etc. The following Table 2 shows an example of a mapping table of interpersonal relationship characteristics and conversation topics:

[0073] Table 2

[0074]

[0075] This exemplary embodiment can determine the target conversation topic from the candidate conversation topics corresponding to the conversation service to be tested based on the mapping table of interpersonal relationship features and conversation topics. It should be noted that the above Table 2 is only an example, and in actual applications, interpersonal relationship features of various dimensions can be set according to specific needs.

[0076] In an exemplary embodiment, the generating of the first prompt information according to the first dialogue information and the characteristic information of the second dialogue party may include:

[0077] Determining characteristic information of the first dialogue party according to the first dialogue information;

[0078] First prompt information is generated according to the feature information of the first dialogue party and the feature information of the second dialogue party.

[0079] Among them, the characteristic information of the first dialogue party can be information used to reflect the characteristics of the first dialogue party, such as the character, attitude, and personality, for example, what kind of character the dialogue party simulated by the first dialogue model has, what the emotion is when speaking, what kind of personality it corresponds to, etc. These characteristics can enable the first dialogue party to have a certain degree of personalization when outputting the first dialogue information. For example, introverted people speak concisely, and talkative people speak for a long time; or lively people speak in a light tone, and sad people speak slowly; happy people are more active in conversation, and anxious people are often uneasy in conversation, etc.

[0080] The characteristic information of the first dialogue party may be obtained by processing the first dialogue information with a pre-trained learning model, or may be a pre-configured mapping table or mapping rule. After obtaining the first dialogue information, the characteristic information of the first dialogue party may be determined based on the content included therein. For example, when the first dialogue information is text information, the characteristic information of the first dialogue party may be determined based on its words, sentence segments, punctuation marks, or even context information. When the first dialogue information is voice information, the characteristic information of the first dialogue party may be determined based on its intonation, speaking speed, sentence segments, words, etc., and so on.

[0081] In an exemplary embodiment, the characteristic information of the first dialogue party includes personality characteristics and / or emotional characteristics; determining the characteristic information of the first dialogue party according to the first dialogue information may include:

[0082] Extracting language style features from the first dialogue information;

[0083] According to a preset first mapping relationship between language style and emotion, the emotion feature corresponding to the language style feature is determined, and / or according to a preset second mapping relationship between language style and personality, the personality feature corresponding to the language style feature is determined.

[0084] Among them, language style features refer to the characteristics of the dialogue between the two parties conveyed by the dialogue information. Regardless of whether the dialogue information is text information or voice information, it can show the language style of the dialogue party. For example, the language style feature can be gentle, often liking to say some caring words to others; or the language style feature is humorous, and the dialogue information often contains some pleasant or happy words or sentences; or the language style is playful, etc.

[0085] Among them, emotions can be used to reflect the mood or state of a person, such as happiness, fear, anxiety, sadness, etc. Personality can reflect the style of a person's work and life, such as extroversion, introversion, etc. This exemplary embodiment can configure a first mapping relationship between a preset language style and emotions, and a second mapping relationship between a preset language style and personality, and after extracting the language style feature from the first dialogue information, by means of the first mapping relationship and / or the second mapping relationship, determine the emotion feature and / or personality feature corresponding to the language style feature of the first dialogue party, so that this exemplary embodiment can mine the emotion feature and / or personality feature of the first dialogue party based on the first dialogue information, and generate the first prompt information with the emotion feature and / or personality feature combined with the feature information of the second dialogue party, for example, generate the first prompt information according to the emotion feature and the feature information of the second dialogue party, or generate the first prompt information according to the personality feature and the feature information of the second dialogue party, or generate the first prompt information according to the emotion feature, the personality feature and the feature information of the second dialogue party, and so on.

[0086] Emotion refers to a person's subjective experience of whether objective things meet their needs. It is a complex psychological state that includes multiple components such as physiological arousal, subjective feelings, cognitive processes and behavioral responses. In essence, emotion is a psychological tool for humans to adapt to the living environment, which can help people respond quickly to the surrounding environment and events. For example, when facing danger, fear can cause people's bodies to enter a state of stress, with faster heartbeats and higher blood pressure, and at the same time prompting people to take actions such as running away or hiding.

[0087] There are two types of emotions: basic emotions and complex emotions. Basic emotions usually include happiness, sadness, anger, fear, disgust and surprise. These emotions are universal across cultures and are the most basic emotional responses formed by humans during the evolution process. For example, happiness is a positive emotion that occurs when people's needs are met or their expectations are achieved, while sadness is often caused by losses, such as losing something important. Anger may be caused by being treated unfairly or having your goals blocked. Complex emotions are caused by a combination of basic emotions or under the influence of social and cultural factors. For example, jealousy is a complex emotion that contains multiple basic emotional components such as anger, sadness and disgust. When a person sees someone else having something or a relationship that they want, they may feel jealous. Pride is also a complex emotion that combines the components of happiness and self-affirmation. When people are satisfied with their own achievements or the achievements of their group, they will feel proud.

[0088] Table 3 below gives an example of a first mapping relationship between a language style and a sentiment:

[0089] Table 3

[0090]

[0091] The emotion classification adopted by this exemplary embodiment may include: admiration, worship, appreciation, entertainment, anxiety, awe, embarrassment, boredom, calmness, confusion, desire, disgust, pain, fascination, jealousy, excitement, fear, hatred, fun, happiness, nostalgia, romance, sadness, satisfaction, desire, sympathy and satisfaction.

[0092] In the related art, personality can be divided into 8 types, as shown in Table 4 below:

[0093] Table 4

[0094]

[0095]

[0096] The four dimensions are like four rulers. Each person's personality will fall on a certain point on the ruler. The closer this point is to the end point, the more preferences the individual has. For example, in the dimension of "attention direction", if the individual's personality is close to the extroverted end, he or she will tend to be extroverted, and the closer to the end point, the stronger the preference.

[0097] Based on the above classification, this exemplary embodiment configures a first mapping relationship between preset language styles and emotions as shown in Table 5 below:

[0098] Table 5

[0099]

[0100]

[0101] This exemplary embodiment can use the framework Self-instruct, which improves the instruction following ability of the language model by guiding it, to complete the generation of instructions, input and output samples from the language model. Among them, the instruction can be understood as a task description, guiding the language model (such as LLM (Large Language Model)) to complete the task, such as the specific requirements of text classification, question answering, text generation and other tasks; the input is the data provided to the language model, which is the information source based on the language model when executing the instruction; the output is the result generated by the language model after receiving the instruction and processing the input data. It is the feedback of the language model on the execution of the task, and its form and content depend on the nature of the instruction and input. For example, in the translation task, the output is the translated target language text; in the text generation task, the output may be new stories, poems and other text content, examples are shown in Table 6:

[0102] Table 6

[0103]

[0104] In this exemplary embodiment, the second language model can be guided to generate the second dialogue information through a generative test case, and a test case can be generated according to the first dialogue information and the second dialogue information. In addition, this exemplary embodiment can also generate the first prompt information in a display interface in a visual manner. Figure 3 An example of prompt words in a visual interface is shown, which includes information related to the first dialogue information, such as the variable part, time represents time, scenePrompt represents scene prompt words, etc., and also includes the characteristic information of the second dialogue party, such as name, age, gender and other basic information. Taking the above Xiao Su fixed personality setting as an example, the dialogue content example with the business AI chat robot is as follows: Figure 4As shown in the figure, it is a test case. It should be noted that more information can be added to the variable part as needed, and each information can be set as optional or not, and can be set or deleted as needed. In addition, in addition to setting a fixed personality, Xiao Su can also set more dimensional personality information, such as preferences, interests, language style, personality, etc.

[0105] Exemplary Devices

[0106] The exemplary embodiment of the present disclosure also provides a test case generation device. Figure 5 As shown, the test case generation device 500 may include the following program modules: a dialogue service determination module 510, used to determine the dialogue service to be tested; the dialogue service to be tested is a service that provides dialogue for users by the first language model; a dialogue information acquisition module 520, used to obtain the first dialogue information of the first dialogue party generated by the first language model; a prompt information generation module 530, used to generate first prompt information according to the first dialogue information and the feature information of the second dialogue party; a prompt information processing module 540, used to process the first prompt information by the second language model to generate the second dialogue information of the second dialogue party; a test case generation module 550, used to determine the test case for the dialogue service to be tested based on the first dialogue information and the second dialogue information.

[0107] In one embodiment, the test case generation device also includes: an environment information acquisition unit, used to obtain the dialogue environment information of the dialogue service to be tested; a prompt information generation module, including: a first information generation unit, used to generate first prompt information based on the first dialogue information, the characteristic information of the second dialogue party and the dialogue environment information.

[0108] In one embodiment, the prompt information generation module includes: an intention determination unit for determining the intention of the first dialogue party based on the first dialogue information; and a second information generation unit for generating first prompt information based on the intention and feature information of the second dialogue party.

[0109] In one embodiment, the test case generation device also includes: a topic acquisition unit, used to obtain a target dialogue topic corresponding to the dialogue service to be tested; a prompt information generation module, including: a third information generation unit, used to generate first prompt information based on the first dialogue information, the characteristic information of the second dialogue party and the target dialogue topic.

[0110] In one embodiment, the test case generation device also includes: a historical information acquisition unit, used to acquire historical conversation information of the first dialogue party and the second dialogue party; an interpersonal relationship determination unit, used to determine the interpersonal relationship characteristics between the first dialogue party and the second dialogue party from the historical conversation information; and a topic acquisition unit, used to determine the target conversation topic from the candidate conversation topics corresponding to the conversation business to be tested based on the interpersonal relationship characteristics.

[0111] In one embodiment, the prompt information generation module includes: a feature information determination unit, which is used to determine the feature information of the first dialogue party according to the first dialogue information; and a fourth information generation unit, which is used to generate the first prompt information according to the feature information of the first dialogue party and the feature information of the second dialogue party.

[0112] In one embodiment, the feature information of the first dialogue party includes personality features and / or emotional features; the feature information determination unit includes: a language style feature extraction subunit, used to extract language style features from the first dialogue information; a feature mapping subunit, used to determine the emotional features corresponding to the language style features based on a first mapping relationship between a preset language style and emotion, and / or determine the personality features corresponding to the language style features based on a second mapping relationship between a preset language style and personality.

[0113] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the above method and will not be repeated here.

[0114] Exemplary Storage Media

[0115] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method of the present disclosure when the computer program is executed by a processor. The above-mentioned method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0116] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0119] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0120] Exemplary Electronic Devices

[0121] The exemplary embodiment of the present disclosure also provides an electronic device, which may be Figure 1 Any device in the present invention. The electronic device includes a processor and a memory, and the memory is used to store executable instructions of the processor. The processor is configured to execute the above method of the present invention by executing the executable instructions.

[0122] refer to Figure 6 An electronic device according to an exemplary embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0123] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).

[0124] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can execute the following steps: Figure 2 The method steps shown, etc.

[0125] The storage unit 620 may include a volatile storage unit, such as a random access storage unit (RAM) 621 and / or a cache storage unit 622 , and may further include a read-only storage unit (ROM) 623 .

[0126] The storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0127] The bus 630 may include a data bus, an address bus, and a control bus.

[0128] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed via an input / output (I / O) interface 640. The electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 650. As shown, the network adapter 650 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0129] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0130] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0131] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A test case generation method, characterized in that: include: Determine a dialogue service to be tested; the dialogue service to be tested is a service in which the first language model provides dialogue for the user; Acquire first dialogue information of a first dialogue party generated by the first language model; generating first prompt information according to the first conversation information and the characteristic information of the second conversation party; Processing the first prompt information by using a second language model to generate second dialogue information of the second dialogue party; A test case for the to-be-tested dialogue service is determined based on the first dialogue information and the second dialogue information.

2. The method according to claim 1, characterized in that The method further comprises: Acquiring the dialog environment information of the dialog service to be tested; The generating first prompt information according to the first dialogue information and the characteristic information of the second dialogue party includes: First prompt information is generated according to the first dialogue information, the feature information of the second dialogue party and the dialogue environment information.

3. The method according to claim 1, characterized in that The generating first prompt information according to the first dialogue information and the characteristic information of the second dialogue party includes: Determining the intention of the first dialogue party according to the first dialogue information; First prompt information is generated according to the intention and the characteristic information of the second dialogue party.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a target conversation topic corresponding to the conversation service to be tested; Generating first prompt information according to the first conversation information and the characteristic information of the second conversation party includes: First prompt information is generated according to the first dialogue information, the characteristic information of the second dialogue party and the target dialogue topic.

5. The method according to claim 4, characterized in that The method further comprises: Acquire historical conversation information between the first dialogue party and the second dialogue party; Determining interpersonal relationship characteristics between the first dialogue party and the second dialogue party from the historical dialogue information; The step of obtaining a target conversation topic corresponding to the conversation service to be tested includes: According to the interpersonal relationship characteristics, a target dialogue topic is determined from candidate dialogue topics corresponding to the dialogue service to be tested.

6. The method according to claim 1, characterized in that The generating first prompt information according to the first dialogue information and the characteristic information of the second dialogue party includes: determining characteristic information of the first dialogue party according to the first dialogue information; First prompt information is generated according to the feature information of the first dialogue party and the feature information of the second dialogue party.

7. The method according to claim 6, characterized in that The characteristic information of the first dialogue party includes personality characteristics and / or emotional characteristics; and determining the characteristic information of the first dialogue party according to the first dialogue information includes: extracting language style features from the first dialogue information; The emotional feature corresponding to the language style feature is determined according to a preset first mapping relationship between the language style and the emotion, and / or the personality feature corresponding to the language style feature is determined according to a preset second mapping relationship between the language style and the personality.

8. A test case generating device, characterized in that: include: A dialogue service determination module, used to determine the dialogue service to be tested; The dialogue service to be tested is a service in which the first language model provides dialogue for the user; A dialogue information acquisition module, used to acquire first dialogue information of a first dialogue party generated by the first language model; A prompt information generating module, configured to generate first prompt information according to the first dialogue information and the characteristic information of the second dialogue party; a prompt information processing module, configured to process the first prompt information through a second language model to generate second dialogue information of the second dialogue party; A test case generation module is used to determine a test case for the to-be-tested dialogue service based on the first dialogue information and the second dialogue information.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images