Interview dialog system, evaluation grid method automation system, and interview dialog method

Abstract

To provide an interview dialogue system, an evaluation grid method automation system and an interview dialogue method capable of grasping the needs of a user efficiently and with high accuracy.SOLUTION: This system is provided with a question data generation part 11, a question data output part 12, an answer data input part 13, an evaluation item extraction part 14, a causal pair data generation part 15, a storage part 16, an output causal pair data generation part 17 and an output causal pair data output part 18. The utterance agent 6a is in charge of the question data generation unit 11 and the question data output unit 12. The listening agent 6b is in charge of the answer data inputting unit 13 and the evaluation item extracting unit 14. The recording agent 6c is in charge of the cause-and-effect pair data generation unit 15 and the storage unit 16. The integration agent 6d is in charge of the output causal pair data generation unit 17 and the output causal pair data output unit 18.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for discovering, extracting, and understanding user needs by analyzing the user's values ​​and preferences. [Background technology]

[0002] As user needs become more diverse, there is a demand for methodologies that can accurately grasp the values ​​and preferences of each individual and develop specific designs. Methods for discovering, extracting, and understanding user needs are essential core technologies for individual optimization of products and services and improving emotional value (experience value), and research is being conducted in a variety of fields, including psychological methods, marketing research methods, behavioral observation methods, emotional evaluation methods, and machine learning methods. Traditionally, methods for understanding user needs have been conducted by collecting and analyzing psychological data through interview surveys and subjective evaluation experiments. A representative interview method is the evaluation grid method (see Non-Patent Document 1).

[0003] The evaluation grid method hierarchically organizes and visualizes an evaluation structure, ranging from abstract value judgments and psychological values ​​to objective judgments and physical states. A dialogue procedure called laddering, which elicits higher-level and lower-level concepts, allows for efficient construction of an evaluation structure from interviews. As a technology using the evaluation grid method, the inventors have already proposed an emotion estimation device that includes a preparatory biometric information acquisition unit, a preparatory emotion information acquisition unit, an association construction unit, an estimated biometric information acquisition unit, and an estimation unit. The device acquires, based on the evaluation grid method, a comfort level indicating each of the subjects' comfort levels as emotion information from each of the subjects, and also acquires factor information indicating the factors behind the comfort level for each of the subjects (see Patent Document 1).

[0004] Although the evaluation grid method is widely used in research and development due to its usefulness, it has the problem of being highly burdensome in terms of both personnel and time. For this reason, efforts have been made to develop tools to support its implementation, and further improvements in efficiency are expected (see Non-Patent Documents 2 and 3). On the other hand, in the field of natural language processing, the development of Transformer-based large language models (LLMs) has dramatically expanded the possibilities of dialogue systems (see Non-Patent Document 4). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6976498 [Non-patent literature]

[0006] [Non-Patent Document 1] Sanui et al., "Extraction of the Evaluation Structure of Living Environments by the Repertoire Grid Development Method - A Study on Living Environment Evaluation Based on Cognitive Psychology (1)", Journal of Planning and Planning, Architectural Institute of Japan, No. 367, pp. 15-22, 1986. [Non-patent document 2] Tsuchida, Y. et al., "Development and Application of Evaluation Grid Method Support Tool", Architectural Institute of Japan Technical Report, Vol. 14, No. 27, pp. 205-208, 2008. [Non-patent document 3] Shunta Otani et al., “Automatic Construction of Impression Evaluation Structure Based on Word Dependency Relationships,” 17th Spring Conference of the Japan Society of Kansei Engineering, 2D2-04, 2022. [Non-patent document 4] Yuki Sakamoto et al., “Study on a dialogue system with user model estimation function using large-scale language models,” Proceedings of the 2023 Annual Conference of the Japanese Society for Artificial Intelligence, vol. 37, p.3O1OS2c01, 2023. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above circumstances, an object of the present invention is to provide an interview dialogue system, an evaluation grid method automation system, and an interview dialogue method that can grasp user needs efficiently and with high accuracy. [Means for solving the problem]

[0008] In order to solve the above problems, the interview dialogue system of the present invention is a system that automates interview dialogue using an AI agent that uses a large-scale language model. The system includes: a question data generation unit that uses an evaluation grid method to generate question data based on evaluation items related to the theme of the interview; a question data output unit that outputs the generated question data as text data or audio data; a response data input unit that accepts input of response data from interviewees to the question data; an evaluation item extraction unit that extracts evaluation items from the response data; a causal pair data generation unit that generates causal pair data combinations of the evaluation items included in the question data and the evaluation items included in the response data as causal pair data; a memory unit that stores at least the causal pair data; an output causal pair data generation unit that generates output causal pair data suitable for visualizing the evaluation structure based on the causal pair data; and an output causal pair data output unit that outputs the output causal pair data to a system that generates an evaluation structure diagram. One AI agent is responsible for the question data generation unit and the question data output unit, and another AI agent is responsible for the response data input unit, the evaluation item extraction unit, and the causal pair data generation unit. This configuration reduces the human and time burden of the evaluation grid method and prevents inconsistencies in results depending on the interviewer or analyst.

[0009] The question data generation unit preferably generates question data based on the theme of the interview, the evaluation items included in the answer data, and the causal pair data, but may also generate question data based on the answer data. For a starting question in a ladder-up or ladder-down progression, the question data generation unit generates question data based on evaluation items generated based on the theme of the interview, but for subsequent questions related to the starting question, question data is generated based on evaluation items extracted from the previous answer data. Therefore, the evaluation items that form the basis for generating question data are the evaluation items included in the question data and are used to generate causal pair data in the causal pair data generation unit. The storage unit preferably further stores question data and evaluation items included in the answer data. The storage unit may also store answer data, combinations of question data and answer data, output causal pair data, etc. The output causal pair data generation unit may further generate output causal pair data suitable for visualization of the evaluation structure based on data other than the causal pair data, such as question data and answer data. It is preferable that other AI agents can always view the question data generated by one AI agent. The answer data input unit can also accept input of answer data as text data or voice data.

[0010] In the interview dialogue system of the present invention, one AI agent is a speaking agent, and the AI ​​agent responsible for the answer data input section and the evaluation item extraction section is a listening agent, and the listening agent passes the question data and the evaluation items contained in the answer data to the speaking agent, and the speaking agent preferably determines the next question based on the interview theme or the question data received from the listening agent and the evaluation items contained in the answer data. This limits the information each AI agent processes, mitigating the lack of consistency in LLMs, known as prompt injection and hallucination problems, and speeding up problem solving.

[0011] In the interview dialogue system of the present invention, the AI ​​agent in charge of the causal pair data generation unit and the memory unit is a recording agent, and the listening agent further passes the evaluation items included in the answer data to the recording agent, which then passes the causal pair data generated in the causal pair data generation unit to the speaking agent, and the speaking agent preferably determines the next question based on the causal pair data received from the recording agent. This limits the information processed by each AI agent, allowing problem solving to proceed more quickly. Note that AI agents other than the recording agent may also be able to access the memory unit.

[0012] In the interview dialogue system of the present invention, it is preferable that the output causal pair data generation unit and the output causal pair data output unit are handled by an integrated agent, which is an AI agent. This limits the information processed by each AI agent, and allows for faster problem solving.

[0013] The evaluation grid method automation system according to a first aspect of the present invention comprises any one of the above-described interview dialogue systems and an evaluation structure visualization system that accepts input of output causal pair data and generates an evaluation structure diagram based on the input output causal pair data. This allows for the generation of an evaluation structure diagram by directly inputting the output causal pair data from the interview dialogue system into the evaluation structure visualization system. The evaluation structure visualization system, based on graph theory, represents evaluation items as nodes and connected items as edges, outputting an evaluation structure diagram that reflects the concepts of superiority and inferiority. It is possible to adjust the weighting of responses and the analytical granularity, and to aggregate responses with the same meaning into categories, enabling objective and quantitative evaluation. It is preferable that the system can extract important parts using a threshold (Katz centrality), and furthermore, it has a filtering function for interviewees and data, enabling analysis that takes individual differences into account. It is also preferable that the evaluation structure visualization system allows multiple users to share data and conduct collaborative analysis.

[0014] The evaluation grid method automation system according to a second aspect of the present invention includes any one of the interview dialogue systems described above, a category generation system that receives input of output causal pair data from the interview dialogue system, generates categories consisting of groups of evaluation items with similar content for evaluation items included in the input output causal pair data, and outputs the categories, a key phrase generation system that receives input of categories, generates representative-expression-assigned categories by assigning representative expressions appropriate to the input categories, and outputs the representative-expression-assigned categories, and an evaluation structure visualization system that receives input of output causal pair data or representative-expression-assigned categories, and generates an evaluation structure diagram based on the input output causal pair data or representative-expression-assigned categories. This makes it possible to automate the evaluation grid method and grasp user needs efficiently and with high accuracy. The evaluation structure visualization system can generate an evaluation structure diagram using either the output causal pair data input from the interview dialogue system or the categories with representative expressions input from the key phrase generation system. Note that "key phrase" and "representative expression" are synonymous, but in the following, we will mainly use the term "key phrase" when explaining the system in general terms, and the term "representative expression" when explaining expressions that are specifically assigned to categories in examples, etc.

[0015] The evaluation grid method automation system according to a third aspect of the present invention comprises a category generation system that receives input of output causal pair data generated in a format suitable for visualization of evaluation structures by aggregating causal pair data consisting of combinations of evaluation items included in question data and answer data created through interviews using the evaluation grid method, generates categories consisting of groups of evaluation items with similar content for the evaluation items included in the input output causal pair data, and outputs the categories, and a key phrase generation system that receives input of categories, generates representative-expression-assigned categories by assigning representative expressions suitable for the input categories, and outputs the representative-expression-assigned categories. This makes it possible to generate categories and key phrases simply by inputting evaluation items created manually or otherwise, and to visualize evaluation structures in interview dialogues using the evaluation grid method by outputting the generated representative-expression-assigned categories to a evaluation structure visualization system or the like.

[0016] In the evaluation grid method automation system according to the third aspect of the present invention, it is preferable that the category generation system comprises a vectorization processing unit that divides and extracts the text into word units required for morphological analysis and vectorizes the text by generating text vectors based on the word units, and a clustering processing unit that classifies the text by similar content based on the vectorized text and generates categories. In the evaluation grid method automation system according to the third aspect of the present invention, the category generation system preferably reclassifies the categories generated by the clustering processing unit into three categories: positive expressions, negative expressions, and neutral expressions.

[0017] In the evaluation grid method automation system according to the third aspect of the present invention, the category generation system and the key phrase generation system may be processed in parallel rather than sequentially. For example, this can be achieved by creating multiple "categories that group together evaluation items with similar content," inputting multiple evaluation items into a large-scale language model (LLM) that is prompt-tuned to create "short titles" for each category, and generating multiple categories and their respective representative expressions. The causal pair data is then divided into multiple batches, and each divided batch is input into the LLM. When the first batch is input into the LLM, multiple categories and their respective representative expressions are generated, and when the second and subsequent batches are input, the evaluation items are classified using one of the existing categories (and representative expressions). For evaluation items that cannot be classified into existing categories, new categories (and representative expressions) are generated and the items are classified into those categories. During the classification process, if the total number of existing categories exceeds a specified number, the categories are divided into multiple category groups, and the classification process is performed on a category group basis. Furthermore, if "Other" is output as the representative expression, or if an evaluation item is omitted or incorrectly output, the category is further divided and reclassified. This configuration makes it possible to prevent the limitations on the amount of input data that exist in existing LLMs, as well as data omissions and erroneous output that occur when many evaluation items are entered at once, and enables more efficient category and key phrase generation.

[0018] The evaluation grid method automation system according to a fourth aspect of the present invention comprises a key phrase generation system that aggregates causal pair data consisting of combinations of evaluation items contained in question data and answer data created by interviews using the evaluation grid method, generates the data in a format suitable for visualizing the evaluation structure, accepts input of categories generated for each evaluation item consisting of a group of evaluation items with similar content, generates representative-expression-assigned categories by assigning representative expressions suitable for the input categories, and outputs the representative-expression-assigned categories, and an evaluation structure visualization system that accepts input of representative-expression-assigned categories and generates an evaluation structure diagram based on the input representative-expression-assigned categories. This makes it possible to generate an evaluation structure diagram simply by inputting category data in which evaluation items have been categorized manually or otherwise.

[0019] A method for having an AI agent using a large-scale language model conduct an interview dialogue comprises the following steps, in which one AI agent is responsible for the question data generation step and the question data output step, and another AI agent is responsible for the answer data input step, the evaluation item extraction step, and the causal pair data generation step. 1) A question data generation step in which question data based on evaluation items related to the theme of the interview is generated using the evaluation grid method. 2) A question data output step of outputting the generated question data as text data or voice data. 3) A response data input step for accepting input of response data from the interviewee in response to the question data. 4) An evaluation item extraction step for extracting evaluation items from the response data. 5) A causal pair data generation step in which combinations of evaluation items included in the question data and evaluation items included in the answer data are generated as causal pair data. 6) A storage step for storing at least the causal pair data. 7) An output causal pair data generation step for generating output causal pair data suitable for visualizing the evaluation structure based on the causal pair data. 8) An output causal pair data output step for outputting the output causal pair data to a system for generating an evaluation structure diagram. [Effects of the Invention]

[0020] The interview dialogue system, evaluation grid method automation system, and interview dialogue method of the present invention have the effect of making it possible to grasp user needs efficiently and with high accuracy. [Brief explanation of the drawings]

[0021] [Figure 1] Functional block diagram of the interview dialogue system according to the first embodiment [Figure 2] An explanatory diagram of a multi-agent model according to the first embodiment. [Figure 3]Overview of the interview dialogue method of Example 1 [Figure 4] Laddering Procedures in Evaluation Grid Interviews [Figure 5] An explanatory diagram of an evaluation grid method automation system according to the second embodiment [Figure 6] Evaluation structure diagram generated by the evaluation grid method automation system of Example 2 [Figure 7] An explanatory diagram of an evaluation grid method automation system according to the third embodiment [Figure 8] Evaluation structure diagram generated by the evaluation grid method automation system of Example 3 [Figure 9] An explanatory diagram of an evaluation grid method automation system according to the fourth embodiment [Figure 10] Illustrative diagram of an evaluation grid method automation system according to the fifth embodiment [Figure 11] Evaluation structure diagram generated by the evaluation grid method automation system of Example 3 [Figure 12] Manually created evaluation structure diagram DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and variations are possible. [Example]

[0023] (Interview dialogue system) In this example, we will explain a system that automates interviews in the evaluation grid method using a large-scale language model.The GPT-4 model, a large-scale language model, is applied to the interview framework to build a causal relationship (laddering) interview dialogue system for the evaluation grid method.

[0024] (About the evaluation grid method) The evaluation grid method is a technique for expressing an individual's evaluation structure as a hierarchical structure through interviews (see Non-Patent Document 1). Figure 4 shows the laddering procedure used in evaluation grid method interviews. First, multiple stimuli to be compared are prepared and the participant is asked to rank them in order of preference. The interviewer then asks the participant why they made that judgment. The evaluation items thus obtained are then asked ladder-up questions to elicit higher-level concepts such as abstract value judgments and psychological values, and ladder-down questions to elicit lower-level concepts such as concrete, objective judgments and physical states, thereby eliciting further evaluation items. Finally, the evaluation structure is created by organizing the obtained evaluation items into a hierarchical structure.

[0025] Fig. 1 is a functional block diagram of an interview dialogue system according to Example 1. As shown in Fig. 1, the interview dialogue system 1 includes a question data generation unit 11 that uses the evaluation grid method to generate question data related to evaluation items related to the theme of the interview, a question data output unit 12 that outputs the generated question data as text data or audio data, a response data input unit 13 that accepts input of response data from interviewees to the question data, an evaluation item extraction unit 14 that extracts evaluation items from the response data, a causal pair data generation unit 15 that generates causal pair data for the evaluation items included in the question data and the evaluation items included in the response data, a storage unit 16 that stores at least the causal pair data, an output causal pair data generation unit 17 that generates output causal pair data based on the causal pair data, and an output causal pair data output unit 18 that outputs the output causal pair data to a system that generates an evaluation structure diagram.

[0026] The speaking agent 6a is responsible for the question data generation unit 11 and the question data output unit 12. The listening agent 6b is responsible for the answer data input unit 13 and the evaluation item extraction unit 14. The recording agent 6c is responsible for the causal pair data generation unit 15 and the storage unit 16. The integrating agent 6d is responsible for the output causal pair data generation unit 17 and the output causal pair data output unit 18. The hardware configuration of the interview dialogue system 1 can be a wide variety of computers, such as well-known PCs, tablet terminals, and smartphones (not shown). The number of devices used is not limited to one, and multiple computers may be combined. In such cases, for example, each computer may be provided with a separate agent, or a computer with one agent and a computer with multiple agents may be combined. In this embodiment, each agent is provided on a single PC. Therefore, question data is output by displaying the question as text data on the PC's display or by outputting audio data from the PC's speakers. Furthermore, the input of response data from the interviewee may be done by inputting text data using a keyboard or the like of the PC, or by inputting voice data using a microphone or the like of the PC. In this example, the dialogue was conducted by voice.

[0027] Here, we explain the multi-agent model that forms the core of the interview dialogue system 1. Figure 2 illustrates the multi-agent model of Example 1. A multi-agent system is a computational model in which multiple autonomous agents cooperate to solve problems. Each agent plays a different role or task, limiting the information processed by a single agent. This mitigates the lack of consistency in LLMs, known as prompt injection and hallucination, and expedites problem solving. In an interview dialogue system using the evaluation grid method, a single agent may be influenced by the participant's utterances and end the interview early or abandon its role. Furthermore, as the interview continues for a long time, the probability of the user making unexpected utterances increases. These problems can be alleviated by introducing multi-agents. Furthermore, when unexpected problems occur, recording the handover of information between agents has the advantage of making it easier to identify the problem. Furthermore, adding or modifying agents makes it possible to expand the system and adjust tasks.

[0028] The interview dialogue system 1 shown in Figure 2 is composed of four agents: a speaking agent 6a that generates questions, a listening agent 6b that extracts evaluation items, a recording agent 6c that records the conversation, and an integration agent 6d that summarizes the conversation and outputs a list of evaluation items. GPT-4 was used for the LLM. The parameters were set to Top-p=0 and Temperature=0. Note that the LLM used is not limited to GPT-4; a wide range of LLM models, such as GPT-4o and lama2, can also be applied. Also, for the sake of convenience, an interviewee 5 is illustrated in Figure 2, but the interviewee 5 is not a component of the interview dialogue system 1.

[0029] (Prompt Tuning) In the interview dialogue system 1, prompt tuning was performed for each agent so that the evaluation grid method could be appropriately carried out. In this embodiment, in order to automate the interview, related questions (Question) and answers (Answer) were exemplified in advance, and the Chain of Thought method, which directs the reasoning path, was used to describe the interview example in the prompt of each agent, thereby directing the role.

[0030] (Prompt of the speaking agent) The speaking agent 6a grasps the process of the entire experiment and generates questions. Specifically, the speaking agent 6a has a question data generation unit 11 that generates question data regarding evaluation items based on the evaluation grid method, and a question data output unit 12 that outputs the generated question data as text data or voice data. The speaking agent 6a determines the next question by referring to the theme of the interview, the question data and answer data (evaluation items) summarized by the listening agent 6b, and the laddering result (causal pair data of evaluation items) recorded by the recording agent 6c. The prompt is composed of six elements: 1) role description, 2) constraints, 3) explanation of the evaluation grid method, 4) interview example, 5) output example, and 6) interview procedure. Each element is defined using XML tags (e.g., <role description>…< / role description>). Note that since this format is for structuring data, formats such as JSON and YAML can also be used.

[0031] In the "1) role description", the role as the speaking agent 6a was explicitly specified as in the following "prompt for the explanation of the evaluation grid method". In this way, by giving attributes, it becomes possible to generate questions that match the attributes.

[0032] (Prompt for the explanation of the evaluation grid method) "You are a professional evaluation grid interviewer. I am the interview participant. Please conduct the interview with the participant according to the following instructions. -Omitted-"

[0033] In "2) Constraints," the questioner describes the topic they want to investigate as shown in the "Constraint Prompt" below. This helps to avoid irrelevant utterances and greetings.

[0034] (constraint prompt) In this example, the first thing you say is, "What is one thing you think is attractive about place A compared to place B?" Do not say anything else. There is no need to explain the evaluation grid during the interview or explain your interpretation of the participants' responses. There is no need to evaluate or comment on the participants' comments. Please conduct the interview following the explanation of the evaluation grid and the interview procedure below. ·Please proceed with the interview while referring to the value structure so far. -omission-"

[0035] In "3) Explanation of the Evaluation Grid Method," an explanation of the Evaluation Grid Method was given as shown in the "Prompt for explaining the Evaluation Grid Method" below. This was written to give an abstract understanding of what would be done next.

[0036] (Prompt for explanation of the evaluation grid method) "By comparing elements, the evaluation grid method allows us to verbalize elements that would not be possible by simply looking at a single element, and these can be obtained as answers in an interview. -Omitted-"

[0037] In "4) Interview Example," an interview example is described as shown in the "Interview Example Prompt" below. Questions and answers are provided as examples, and the line of reasoning can be directed.

[0038] (Sample interview prompt) Theme: Luxury of the car S(system),U(user),Upper-level item(upper item),Lower-level item(lower item) S: When do you feel a car is luxurious? U: When it feels sporty S: In what specific cases do you feel it has a sporty feel? U: Streamlined shape S: So what specifically do you think is streamlined? U: Thin tip, when the vehicle height is low S: In what other cases do you feel a sporty feel? -omission-"

[0039] 5) describes an "output example." At the end of the interview, the causal pair data of the evaluation items obtained from interviewee 5 must be output as output causal pair data. In this example, the data is output in a list format in CSV format, as shown in the "output example prompt" below.

[0040] (prompt in example output) 『upper-level item,lower-level item,interviewee Luxurious, sporty, ID1 Luxurious, genuine leather seats, ID1 Sporty, red color, ID1 Sporty, streamlined, ID1 -omission- Durable, good quality, ID1 Hassle-free, hard to break, ID1

[0041] In 6), the "interview procedure" was described. The interview in the evaluation grid method uses laddering to clarify the causal relationship of the evaluation of interviewee 5. Based on the laddering procedure explained in Figure 4, the prompts for the interview procedure were written as shown in the "interview procedure prompts" below.

[0042] (Interview procedure prompt) The procedure for an evaluation grid method interview involves repeatedly extracting original evaluation items and laddering them. Original evaluation items are the evaluation items that serve as the starting point for the interview. Laddering is a procedure for extracting more abstract evaluation items (higher concepts) and more specific evaluation items (lower concepts) from evaluation items. Laddering involves laddering up to extract more abstract superordinate concepts from the evaluation items, and laddering down to extract more specific subordinate concepts. The specific steps are: 1. Ladder down the original evaluation items and repeat until no further sub-concepts can be extracted. 2. Ladder up the original evaluation items and repeat until no more superordinate concepts can be extracted. 3. Move to the next original evaluation item and perform steps 1 and 2. 4. Once all original assessment items have been completed, the interview will end. -omission-"

[0043] (Listening Agent Prompt) 2 includes a response data input unit 13 that receives response data from the interviewee 5 in response to the question data. The listening agent 6b includes an evaluation item extraction unit that extracts evaluation items from the response data, and passes the question data for the interviewee 5 and data in which the responses from the interviewee 5 are aggregated into evaluation items to the speaking agent 6a, and passes the data in which the responses from the interviewee 5 are aggregated into evaluation items to the recording agent 6c. The listening agent 6b may also pass the question data and response data to the speaking agent 6a and the recording agent 6c. The prompts consisted of three elements: 1) role description, 2) constraints, and 3) example conversations, and were structured as shown in the "Listening Agent Prompt Structure" below.

[0044] (Prompt Structure of the Listening Agent) 『<Role Explanation> You are a professional evaluation grid interviewer. I am an interview participant. Conduct an interview with the participant according to the following instructions and summarize the answers. < / Role Explanation> <Constraints> · Do not say anything other than the rules. Do not output the participant's conversation in the conversation examples. · I will answer the questions. At that time, please summarize the answers according to the following rules and output only the nouns. - Omitted - · If there is a concept opposite to the answer, please infer and output the concept the participant wants to express. < / Constraints> <Conversation Examples> <Example 1> S: What needs to be the case to feel a sense of luxury in a car? U: When it has a sporty feel S: Having a sporty feel < / Example 1> - Omitted - < / Conversation Examples>』

[0045] Also, to stabilize the output, the listening agent 6b was not given the previous conversation record.

[0046] (Prompt of the Recording Agent) The recording agent 6c shown in Figure 2 records the contents of the speaking agent 6a and the listening agent 6b. Specifically, it has a causal pair data generation unit 15 that generates causal pair data with the evaluation items included in the question data and the evaluation items included in the answer data as causal pairs, and a storage unit 16 that stores the question data, the evaluation items included in the answer data, and the causal pair data. In this way, the recording agent 6c generates causal pairs composed of upper items (upper concepts) and lower items (lower concepts), and plays the role of "memory" for the speaking agent 6a and the listening agent 6b. The prompt is composed of three elements: 1) role explanation, 2) explanation of the evaluation grid method, and 3) output example.

[0047] (Integrated Agent prompt) The integration agent 6d refers to all conversation records and the summary results of the recording agent 6c, and outputs the final causal pair list of evaluation items (output causal pair data).The prompt consisted of three elements: 1) a role description, 2) an explanation of the evaluation grid method, and 3) an example of a causal pair list of evaluation items.

[0048] (Interview dialogue method) Next, a specific example of use of the interview dialogue system of the first embodiment will be described using an interview about "attractive points of place A" as an example. There are six interviewees 5. Figure 3 shows a schematic flow diagram of the interview dialogue method of Example 1. As shown in Figure 3, first, the speech agent 6a generates question data related to evaluation items based on the theme of the interview using the evaluation grid method (step S01: question data generation step). However, the first time, a question is generated to extract original evaluation items. Specifically, for example, question data such as "What is attractive about place A compared to place B?" is generated. Next, the generated question data is output as text data or voice data (step S02: question data output step). The interviewee 5 answers the given question by text or voice. Specifically, the first time, the interviewee 5 inputs a response such as "I think the town is a neat and tidy place" as text data or voice data.

[0049] The hearing agent 6b receives input of answer data from the interviewee 5 in response to the question data (step S03: answer data input step). Here, the answer data received is "I think the town is a neat and tidy place." The listening agent 6b extracts evaluation items from the response data (step S04: evaluation item extraction step). Here, the listening agent 6b extracts the evaluation item "the town is neat and tidy" from the response data "I feel that the town is neat and tidy." The listening agent 6b passes the question data and the evaluation items included in the answer data to the speaking agent 6a (step S05). Here, the question data "What is attractive about place A compared to place B?" and the evaluation item "The town is neat and tidy" are passed to the speaking agent 6a. The listening agent 6b also passes the evaluation item included in the response data to the recording agent 6c (step S06). In this example, the listening agent 6b passes the evaluation item "the town is neat and tidy" to the recording agent 6c.

[0050] The recording agent 6c generates a combination of the evaluation items included in the question data and the evaluation items included in the answer data as causal pair data (step S07: causal pair data generation step). For the first question, the recording agent 6c extracts an evaluation item such as "attractive aspects of place A" from question data such as "What are the attractive aspects of place A compared to place B?" and generates a combination of the evaluation item "attractive aspects of place A" and the evaluation item "neatness of the town" included in the answer data as causal pair data, but as will be described later, for the first causal pair data, the evaluation item included in the question data is not stored. In contrast, for questions from the second onwards, the recording agent 6c can use the evaluation items contained in the answer data that is the basis for generating the question data as the evaluation items contained in the question data, since the evaluation items contained in the answer data that is the basis for generating the question data are stored in the memory unit 16. The recording agent 6c stores the question data, the evaluation items included in the answer data, and the causal pair data (step S08: storage step). The recording agent 6c also passes the causal pair data to the speaking agent 6a (step S09). However, since the first causal pair data is a causal pair of a theme and an original evaluation item, only the original evaluation item is stored, and the theme is not stored. For the sake of convenience, only the ladder-down method, which derives more specific subordinate concepts from evaluation items, will be described as an example here, but in reality, ladder-up method, which derives more abstract superordinate concepts, is also used.

[0051] If all questions regarding the ladder-down and ladder-up based on the evaluation grid method have not been completed for the interview theme (step S10), the speech agent 6a again generates question data regarding the evaluation items (step S01). Here, since all questions regarding the ladder-down have not yet been completed, a second question regarding the ladder-down is asked. For the second and subsequent questions, the speaking agent 6a generates question data related to the evaluation items based not only on the theme of the interview but also on the question data received from the listening agent 6b, the evaluation items included in the answer data, and the causal pair data received from the recording agent 6c. Specifically, the speaking agent 6a generates question data such as, for example, "What does it take for a town to be perceived as neat and tidy?" based on the question data "What is attractive about place A compared to place B?" and the evaluation item "the town is neat and tidy," and the causal pair data consisting of a combination of the evaluation item "what is attractive about place A" included in the question data (the theme the first time) and the evaluation item "the town is neat and tidy" included in the answer data.

[0052] As in the first time, the generated question data is output as text data or voice data (step S02), and the interviewee 5 inputs a response to the question such as, "It is important that the sidewalks and buildings have good pavement." The listening agent 6b receives the input of the answer data (step S03), and extracts the evaluation items "The sidewalks are well paved" and "The buildings are well paved" from the answer data "It is necessary that the sidewalks and buildings have good pavement" (step S04). The question data "What is necessary for a town to be perceived as neat and tidy?" and the evaluation items "The sidewalks are well paved" and "The buildings are well paved" are passed to the speaking agent 6a and the recording agent 6c (step S05). The listening agent 6b also passes the evaluation items "The sidewalks are well paved" and "The buildings are well paved" to the recording agent 6c (step S06).

[0053] The recording agent 6c that has received this data generates causal pair data that is a combination of the evaluation item included in the question data and the evaluation item included in the answer data (step S07: causal pair data generation step). Here, a combination of the evaluation item "neatness of the town" included in the question data and the evaluation item "street paving is solid" included in the answer data, or a combination of the evaluation item "neatness of the town" included in the question data and the evaluation item "buildings are paved solidly" included in the answer data is generated as causal pair data. The recording agent 6c stores the evaluation items and the causal pair data included in the response data (step S08), and passes the causal pair data to the utterance agent 6a (step S09).

[0054] If all questions regarding the ladder-down and ladder-up based on the evaluation grid method have not been completed for the interview topic (step S10), the speech agent 6a again generates question data regarding the evaluation items (step S01). On the other hand, if all questions regarding the ladder-down and ladder-up based on the evaluation grid method have been completed (step S10), the integrated agent 6d generates output causal pair data suitable for visualization of the evaluation structure based on the causal pair data (step S11: causal pair data generation step), and outputs the output to a system that generates an evaluation structure diagram (step S12: causal pair data output step). Table 1 below shows an example of output causal pair data suitable for visualization of the evaluation structure. As shown in Table 1, the causal pair data consisting of a superordinate item (superordinate concept) and a subordinate item (subordinate concept) is listed. The identification number (ID) of the interviewee from whom the causal pair data was obtained is also listed. For convenience of explanation, it is not shown in Table 1, but it is also possible to filter the data using the subject's attribute data (personal identification data such as age and gender) linked to the interview subject number. In addition, the actual output causal pair data is output as data in CSV format.

[0055] [Table 1] [Example]

[0056] Fig. 5 is an explanatory diagram of the evaluation grid method automation system of Example 2. As shown in Fig. 5, the evaluation grid method automation system 10a of Example 2 is composed of an interview dialogue system 1 and an evaluation structure visualization system 4. The configuration of the interview dialogue system 1 is the same as that of Example 1.

[0057] (Evaluation Structure Visualization System) The evaluation structure visualization system 4 accepts input of output causal pair data and generates an evaluation structure diagram based on the input output causal pair data. The evaluation structure visualization system 4 can adjust the weighting of responses and the granularity of analysis based on graph theory, and can also aggregate responses with the same meaning into categories, making objective and quantitative evaluation possible. Important parts can be extracted using a threshold (Katz centrality). Furthermore, it has a filtering function for interviewees and data, making it possible to conduct analysis that takes individual differences into account. The evaluation structure visualization system 4 is also a system that allows multiple users to share data and conduct collaborative analysis.

[0058] As shown in Fig. 5, the output causal pair data output from the interview dialogue system 1 is input to the evaluation structure visualization system 4. The data input to the evaluation structure visualization system 4 is the output causal pair data shown in Table 1 above, converted into CSV format data. Fig. 6 shows a rating structure diagram generated by the rating grid method automation system of Example 2. As shown in Fig. 6, a rating structure diagram can also be generated by directly inputting output causal pair data from the interview dialogue system 1 to the rating structure visualization system 4. [Example]

[0059] Fig. 7 is an explanatory diagram of the evaluation grid method automation system of Example 3. As shown in Fig. 7, the evaluation grid method automation system 10b is composed of four systems: an interview dialogue system 1, a category generation system 2, a key phrase generation system 3, and an evaluation structure visualization system 4. The interview dialogue system 1 has the same configuration as in Example 1.

[0060] (category generation system) As shown in Figure 7, the category generation system 2 receives input of output causal pair data from the interview dialogue system 1, generates categories consisting of groups of evaluation items with similar content for the evaluation items included in the input output causal pair data, and outputs them to the key phrase generation system 3. The category generation system 2 standardizes the spelling variations of the collected evaluation items and then clusters similar items. Table 2 below shows the causal pair data input from the interview dialogue system 1, with only the evaluation items used for category generation remaining in the category generation system 2.

[0061] [Table 2]

[0062] (Pretreatment) The evaluation items contained in the causal pair data input from Interview Dialogue System 1 are first preprocessed. Preprocessing is carried out in the following order: splitting into words using morphological analysis, extracting necessary words, and unifying word conjugation forms to their root forms. MeCab's Neologd dictionary was used for the morphological analysis. In order to extract the words necessary for classifying the evaluation items, words that do not have meaning, such as particles, were excluded. The exclusion method involved extracting only nouns, verbs, adjectives, and adverbs, and then removing commonly used but unrelated words such as "good," "think," and "do."

[0063] (Vectorization) In clustering, words that characterize sentences are important, and processing is performed by vectorizing the sentences. While known vectorization methods include TF-IDF (Term Frequency-Inverse Document Frequency), in this example, vectorization was performed using Sentence-BERT (SBERT), which generates highly accurate sentence vectors taking context into account.

[0064] (Clustering) The vectorized evaluation items were classified using BERTopic. BERTopic is known to have a high coherence score for short sentences such as evaluation items. The classification accuracy was also improved by incorporating sentiment analysis BERT. In this example, UMAP was used for dimensionality reduction, HDBSCAN for clustering, and class-based TF-IDF for extracting key words. Furthermore, sentiment analysis BERT used three labels: positive, negative, and neutral. This allows for reclassification of evaluation items within the same category into separate categories of positive, negative, and neutral expressions for categories that do not use proper nouns as key words, achieving classification results close to those achieved by manual methods. Table 3 below shows an example of clustering of evaluation items by the category generation system 2. After clustering the evaluation items, the category generation system 2 outputs this category data to the key phrase generation system 3.

[0065] [Table 3]

[0066] In this example, interviewee 5 was asked "What is attractive about place A?", so there were few negative emotions, and it can be said that the effect of clustering using sentiment analysis BERT is difficult to see in Table 3 above. Therefore, as a reference example, Table 4 below shows an example of clustering when the question "What is attractive and unattractive about place B?" As shown in Table 4 below, by classifying into separate categories of positive expressions, negative expressions, and neutral expressions, it is possible to obtain classification results that are closer to manual classification.

[0067] [Table 4]

[0068] (Keyphrase generation system) 7, the key phrase generation system 3 receives the input of a category, generates a category with a representative expression assigned to the input category by assigning a representative expression appropriate to the category, and outputs the category with a representative expression to the evaluation structure visualization system. The category with a representative expression assigned is output to the evaluation structure visualization system 4 as data in CSV format. In other words, the key phrase generation system 3 extracts representative words (representative expressions) that unify the evaluation items for each category (cluster) input from the category generation system. The evaluation structure needs to be as simple as possible so that it is easy for people to interpret, and it is essential to process similar evaluation items as representatives. For this reason, representative expressions are selected based on word frequency within the cluster, concept abstraction, and dependency analysis.

[0069] Specifically, various key phrase generation methods have been proposed, including TF-IDF, TextRank (a ranking algorithm based on graph structure), and EmbedRank (an algorithm for extracting key phrases from text without a teacher). This example focuses on the fact that evaluation items generally consist of short sentences and words, and explains how to select an appropriate processing method for cases where the evaluation items constituting a category are primarily sentence-based (hereinafter referred to as "sentence category") and cases where they are primarily words (hereinafter referred to as "word category"). Here, determining whether a sentence is a given is based on whether it contains a predicate and whether it is a noun-terminated sentence. For the text category, we use EmbedRank to select highly similar evaluation items, and input them into an LLM (here, GPT-4) that has been prompt-tuned to create "short titles" to generate multiple titles. Next, we use the search function in Google Chrome (registered trademark) to find the number of searches for each title, and select the one with the most searches as the representative expression.

[0070] On the other hand, for word categories, preprocessing was performed on the evaluation items within the category, similar to the preprocessing in Category Generation System 2, and the items were divided into words. Next, synonyms were unified within the word categories by automatically determining the threshold for synonym determination. When determining the threshold for synonym determination, the EDR concept hierarchy dictionary, which groups words with similar meanings into conceptual units and describes the hierarchical relationships between concepts, and natural language processing models (BERT, word2vec) that can represent words and sentences as distributed representations were used. After that, the word with the highest importance was determined from the cluster of words after synonym unification. TF-IDF was used to calculate importance. The percentage of the word determined by TF-IDF in the entire category was calculated, and if it was below a threshold, the concept of the word in the category was replaced with one of the next level using the EDR concept system dictionary. The importance of the words in the category was calculated again, and the one with the highest value was used as the representative expression. Additionally, for categories where concept abstraction was not performed, words that have a dependency relationship with the representative expression were searched for among the evaluation items within the category and assigned as the representative expression. The Ginza morphological analyzer was used to extract the dependency relationships. Table 5 below shows an image of the categories and evaluation items to which representative expressions have been assigned.

[0071] [Table 5]

[0072] The evaluation structure visualization system 4 receives input of the output causal pair data shown in Table 1 above or the categories to which representative expressions have been assigned shown in Table 5 above, and generates an evaluation structure diagram based on the input output causal pair data or the categories to which representative expressions have been assigned. Unlike the second embodiment, the evaluation structure visualization system 4 of the third embodiment receives input of the categories to which representative expressions have been assigned, and generates an evaluation structure diagram based on the input categories to which representative expressions have been assigned, but the other configurations are the same as those of the second embodiment.

[0073] Fig. 8 shows a rating structure diagram generated by the rating grid method automation system of Example 3. As shown in Fig. 8, compared to the rating structure diagram (see Fig. 6) generated by the rating grid method automation system 10a of Example 2 in which output causal pair data is input directly from the interview dialogue system 1 to the rating structure visualization system 4, in the rating grid method automation system 10b of Example 3 in which clustering is performed by the category generation system 2 and the key phrase generation system 3, representative expressions are assigned to each category, and the categories with representative expressions assigned are then input to the rating structure visualization system 4, the amount of information is not enormous, and it can be seen that the rating structure diagram is better organized.

[0074] Next, the usefulness was verified by comparing the evaluation structure diagram (Example) generated using the evaluation grid method automation system 10b of Example 3 with the evaluation structure diagram (Comparative Example) obtained by manual evaluation grid method.

[0075] (Experiment overview) To verify the effectiveness of the evaluation grid method automation system 10b of Example 3, an evaluation structure constructed by manual interviews (comparison example) was compared with that constructed by this system (example). The comparison object was "attractive aspects of place A," and the interviewees 5 were 10 people who had visited place A.

[0076] (Experimental results) Below is an example of a dialogue in the interview dialogue system 1. It can be seen that the interview dialogue system 1 accurately identified the theme from the answers of interviewee 5 and generated the next question. It was also confirmed that the system made an accurate decision when to end the dialogue. It was confirmed that both ladder-up and ladder-down were possible in all experiments. Furthermore, for all 10 interviewees 5, the dialogue ended correctly after all laddering was completed.

[0077] (Example of a dialogue using an interview dialogue system) <Interview begins> system:We'll start the interview now. Please tell us one thing that you think is attractive about place A. user:City atmosphere system: What does it take to get a feel for the city? User: The fusion of nature and the city system: What needs to happen for nature and the city to feel integrated? user: Abundant nature system: What needs to happen to feel like you're surrounded by nature? User: Having mountains and the sea nearby system: What needs to happen for the mountains to feel closer to you? user:Not too many buildings -omission- system: Why do you find place A attractive when it combines nature and the city? User: The atmosphere is calm system: Why do you think being calm is attractive? user:To relax -omission- system: Why do you find place A attractive when you enjoy tourist attractions? user:Because I feel attached to it system:Other than that, why do you find it appealing to visit tourist attractions? user:no other system:So, is there anything else you think is attractive about Place A? user:no other system:Thank you. That's all for the interview. Thank you very much. <End>』

[0078] Figure 11 shows a portion of an evaluation structure diagram generated by the evaluation grid method automation system 10b, and Figure 12 shows a portion of an evaluation structure diagram created manually. Both show the attractive aspects of place A. In the diagram, the left side shows the superordinate concept, and the right side shows the subordinate concept. Regarding the attractiveness of place A, as shown in Figure 11 or Figure 12, the highest concepts in both figures are "increased desire to revisit" and "going there," while the lowest concepts are "a concentration of stores" and "something fun to do," which can be said to be roughly consistent. From the above, the effectiveness of the evaluation grid method automation system 10b was confirmed.

[0079] Next, to evaluate the keyphrase generation system 3, two people with experience in the evaluation grid method evaluated the "suitability" of keyphrases (representative expressions) on a four-point scale. The percentage of total points that received 6 or more out of a maximum of 8 was defined as a correct answer, and the accuracy rate was calculated. As a result, as shown in Table 6 below, the accuracy rate for the method of this embodiment was confirmed to be sufficiently high at 63.6%, compared to 40.9% for the conventional TF-IDF method, 46.4% for TextRank, and 30.9% for YAKE. These results demonstrate the effectiveness of the evaluation grid method automation system 10b.

[0080] [Table 6] [Example]

[0081] FIG. 9 is an explanatory diagram of the evaluation grid method automation system of the fourth embodiment. As shown in FIG. 9, the evaluation grid method automation system 10c is composed of two systems: a category generation system 2 and a key phrase generation system 3. Both the category generation system 2 and the key phrase generation system 3 have the same configuration as those of the third embodiment. In other words, the evaluation grid method automation system 10c does not realize the entire configuration of the evaluation grid method automation system 10b of the third embodiment, but realizes only the category generation and key phrase generation parts. As a result, categories and key phrases can be generated simply by inputting evaluation items created manually, etc., and the generated categories with representative expressions assigned can be output to an evaluation structure visualization system, etc., thereby realizing visualization of the evaluation structure in an interview dialogue using the evaluation grid method. [Example]

[0082] FIG. 10 is an explanatory diagram of the evaluation grid method automation system of Example 5. As shown in FIG. 10, the evaluation grid method automation system 10d is composed of two systems: a key phrase generation system 3 and an evaluation structure visualization system 4. Both the key phrase generation system 3 and the evaluation structure visualization system 4 have the same configuration as those of Example 3. In other words, the evaluation grid method automation system 10d does not realize all of the configuration of the evaluation grid method automation system 10b of Example 3, but only realizes key phrase generation and evaluation structure diagram generation. This structure makes it possible to generate an evaluation structure diagram simply by inputting category data in which evaluation items are categorized manually or the like. [Example]

[0083] In Example 3, an example was described in which category generation and key phrase generation were performed sequentially, but it is also possible to perform these processes simultaneously. That is, by creating multiple "categories that group together evaluation items with similar content" in the LLM and prompt-tuning the LLM to create "short titles" for each category, inputting multiple evaluation items into the LLM and generating multiple categories and their respective representative expressions, category generation and key phrase generation can be performed simultaneously. However, existing LLMs have a limit on the amount of input data, and inputting many evaluation items at once can lead to problems such as data leakage and erroneous output.

[0084] To solve this problem, the evaluation items are divided into a small number of groups (batches), which are input into the LLM in batches, with prompt tuning performed in parallel (by alternating in stages). Specifically, when the first batch (evaluation items) is input into the LLM, multiple categories and their respective representative expressions are generated. When subsequent batches are input, the evaluation items are classified using one of the existing categories (and representative expressions) (i.e., assigned to an existing category). If there are evaluation items that cannot be classified into an existing category, a new category (and representative expression) is generated and the item is classified into that new category. During the classification process, if the total number of existing categories exceeds a predetermined number, the categories are divided into multiple category groups, and classification processing is performed on a category group basis. Furthermore, if "Other" is output as the representative expression, or if an evaluation item is omitted or incorrectly output, the items are divided into smaller batches and reprocessed. By inputting the evaluation items into the LLM that has undergone the above prompt tuning, category generation and key phrase generation can be performed more efficiently. [Industrial Applicability]

[0085] The present invention is useful for a system that discovers, extracts, and understands user needs by analyzing the user's values ​​and preferences. [Explanation of symbols]

[0086] 1. Interview dialogue system 2. Category Generation System 3 Keyphrase Generation System 4 Evaluation Structure Visualization System 5. Interviewees 6a Speaking Agent 6b Listening Agent 6c Recording Agent 6d Integrated Agent 10a~10d Evaluation grid method automation system 11 Question data generation unit 12 Question data output section 13 Answer data input section 14 Evaluation item extraction section 15 Causal pair data generation unit 16 Memory section 17 Output causal pair data generation unit 18 Output causal pair data output unit

Claims

1. In a system that automates interview dialogue using an AI agent that uses a large-scale language model, The system comprises: a question data generation unit that generates question data based on evaluation items related to the theme of the interview using an evaluation grid method; a question data output unit that outputs the generated question data as text data or voice data; a response data input unit that receives response data from an interviewee in response to the question data; an evaluation item extraction unit that extracts evaluation items from the response data; a causal pair data generating unit that generates causal pair data based on a combination of an evaluation item included in the question data and an evaluation item included in the answer data; a storage unit that stores at least the causal pair data; an output causal pair data generation unit that generates output causal pair data suitable for visualization of the evaluation structure based on the causal pair data; an output causal pair data output unit that outputs the output causal pair data to a system that generates an evaluation structure diagram; Equipped with The question data generation unit and the question data output unit are handled by one AI agent, An interview dialogue system characterized in that the response data input unit, the evaluation item extraction unit, and the causal pair data generation unit are handled by other AI agents.

2. One AI agent is a speech agent, The AI ​​agent in charge of the response data input unit and the evaluation item extraction unit is a listening agent, 2. The interview dialogue system according to claim 1, wherein the listening agent passes the question data and the evaluation items included in the answer data to the speaking agent, and the speaking agent determines the next question based on the theme of the interview or the question data received from the listening agent and the evaluation items included in the answer data.

3. an AI agent in charge of the causal pair data generation unit and the storage unit is a recording agent; the listening agent further passes the evaluation items included in the answer data to the recording agent; the recording agent passes the causal pair data generated by the causal pair data generating unit to the utterance agent; 3. The interview dialogue system according to claim 2, wherein the speaking agent further determines a next question to ask based on the causal pair data received from the recording agent.

4. 4. The interview dialogue system according to claim 3, wherein the output causal pair data generation unit and the output causal pair data output unit are managed by an integrated agent that is an AI agent.

5. An interview dialogue system according to any one of claims 1 to 4; an evaluation structure visualization system that receives input of the output causal pair data and generates the evaluation structure diagram based on the input output causal pair data; An evaluation grid method automation system comprising:

6. An interview dialogue system according to any one of claims 1 to 4; a category generation system that receives an input of the output causal pair data from the interview dialogue system, generates categories made up of groups of evaluation items with similar contents for the evaluation items included in the input output causal pair data, and outputs the categories; a key phrase generation system that receives an input of the category, generates a representative expression-assigned category by assigning a representative expression suitable for the input category, and outputs the representative expression-assigned category; an evaluation structure visualization system that receives input of the output causal pair data or the representative expression-assigned categories, and generates the evaluation structure diagram based on the input output causal pair data or the representative expression-assigned categories; An evaluation grid method automation system comprising:

7. a category generation system that aggregates causal pair data consisting of combinations of evaluation items included in question data and answer data, which are created by interviews using the evaluation grid method, receives input of output causal pair data generated in a format suitable for visualization of evaluation structures, generates categories consisting of groups of evaluation items with similar contents for the evaluation items included in the input output causal pair data, and outputs the categories; a key phrase generation system that receives an input of the category, generates a representative expression-assigned category by assigning a representative expression suitable for the input category, and outputs the representative expression-assigned category; An evaluation grid method automation system comprising:

8. The category generation system includes: a vectorization processing unit that divides and extracts the text into word units required for morphological analysis, and vectorizes the text by generating text vectors based on the word units; 8. The evaluation grid method automation system according to claim 7, further comprising a clustering processing unit that classifies the vectorized sentences by similar content and generates categories.

9. In the category generation system, 9. The evaluation grid method automation system according to claim 8, wherein the categories generated by the clustering processing unit are reclassified into three categories: positive expressions, negative expressions, and neutral expressions.

10. a key phrase generation system that aggregates causal pair data consisting of combinations of evaluation items included in question data and answer data, which are created by interviews using the evaluation grid method, generates the data in a format suitable for visualizing the evaluation structure, receives input of categories generated by groups of evaluation items with similar contents for the evaluation items, generates representative expression-assigned categories by assigning representative expressions suitable for the input categories, and outputs the representative expression-assigned categories; an evaluation structure visualization system that receives an input of the representative expression-assigned categories and generates the evaluation structure diagram based on the input representative expression-assigned categories; An evaluation grid method automation system comprising:

11. A method for having an AI agent using a large-scale language model conduct an interview dialogue, a question data generation step of generating question data based on evaluation items related to the theme of the interview using an evaluation grid method; a question data output step of outputting the generated question data as text data or voice data; a response data input step of accepting input of response data from an interviewee in response to the question data; an evaluation item extraction step of extracting evaluation items from the response data; a causal pair data generating step of generating causal pair data based on a combination of an evaluation item included in the question data and an evaluation item included in the answer data; a storing step of storing at least the causal pair data; an output causal pair data generating step of generating output causal pair data suitable for visualization of an evaluation structure based on the causal pair data; an output causal pair data output step of outputting the output causal pair data to a system that generates an evaluation structure diagram; Equipped with The question data generation step and the question data output step are performed by one AI agent, An interview dialogue method characterized in that the answer data input step, the evaluation item extraction step, and the causal pair data generation step are performed by another AI agent.

Images