Program, apparatus, and method for creating survey questions

Abstract

We provide a program that obtains evaluable expectations and satisfaction levels by asking a variety of questions to users targeted in a survey. [Solution] In a program that causes a computer to function to collect questionnaires from users based on evaluation items, prompts are sent to a generation AI server instructing it to create questions about the future and questions about the present for each evaluation item. The program functions as a question creation means that receives the questions about the future and questions about the present from the generation AI server, and as a response collection means that sends the questions about the future and questions about the present to each user terminal, receives the answers to each question from the terminal, and stores the answers to the questions about the future as expectations and the answers to the questions about the present as satisfaction levels in a response database for each user.

Description

Technical Field

[0001] The present invention relates to a questionnaire technology for asking questions to multiple users.

Background Art

[0002] Conventionally, there is a technology for an organizational development support system that supports the strengthening and improvement of an organization (see, for example, Patent Document 1). According to this technology, scores for expectancy and satisfaction are obtained from the responses of multiple members to multiple survey questions, which are questions for which scores for expectancy and satisfaction are defined, and a score indicating the overall motivation of the members is calculated using the scores for the multiple expectancies and satisfactions as parameters.

Prior Art Documents

Non-Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when asking members using questions with predefined expectancy and satisfaction as in Patent Document 1, a variety of questions cannot be asked. That is, there is a problem in that it is necessary to obtain expectancy and satisfaction from fixed questions. In contrast, the inventor of the present application considered, firstly, whether it is possible to obtain evaluable expectancy and satisfaction by asking a variety of questions to the users who are the questionnaire targets. Secondly, the inventor considered whether it is possible to obtain an appropriate evaluation index from the response values of the users for each evaluation item. Furthermore, thirdly, the inventor considered whether it is possible to obtain an emotional evaluation index from the text of the qualitative responses from the users. Furthermore, fourthly, we considered whether it would be possible to identify nouns that are strongly emphasized emotionally from the qualitative responses provided by users.

[0005] Therefore, the first objective of the present invention is to provide a program, apparatus, and method that can obtain evaluable expectations and satisfaction levels by asking a variety of questions to users who are the target of a survey. [Means for solving the problem]

[0006] <First Invention> According to the first aspect of the present invention, in a program that causes a computer to function to collect questionnaires from users based on evaluation items, A question creation means that sends prompts to a generating AI (Artificial Intelligence) server instructing it to create questions about the future and questions about the present for each evaluation item, and receives the questions about the future and questions about the present from the generating AI server. A response collection method that sends questions about the future and questions about the present to each user terminal, receives answers to each question from the terminal, and stores the answers to the questions about the future as expectations and the answers to the questions about the present as satisfaction levels for each user in a response database. It is characterized by enabling the computer to function in this way.

[0007] According to another embodiment of the first program of the present invention, The evaluation items are survey questions for multiple users belonging to an organization or service, structured in a hierarchical manner. The question creation mechanism includes attribute information of the organization or service in the prompt in order to create questions based on that organization or service. It is also preferable to make the computer function in this way.

[0008] According to another embodiment of the first program of the present invention, The question creation mechanism includes, in order to create a question for each user, either user attribute information and / or past answer information, or a combination thereof, in the prompt. It is also preferable to make the computer function in this way.

[0009] According to another embodiment of the first program of the present invention, The question generation mechanism sends a prompt to the AI ​​generation server requesting that the generated question be a quantitative answer. It is also preferable to make the computer function in this way.

[0010] According to another embodiment of the first program of the present invention, The method for creating questions is, A prompt is sent to the generation AI server instructing it to create forward-sounding questions about the future, paradoxical questions about the future, forward-sounding questions about the present, and paradoxical questions about the present. The AI ​​server receives forward-sounding questions about the future, paradoxical questions about the future, forward-sounding questions about the present, and paradoxical questions about the present. The means of collecting responses is, Quantitative answers to forward-looking questions about the future are defined as positive expectations, and quantitative answers to paradoxical questions about the future are defined as negative expectations. Quantitative responses to forward-sounding questions about the present are treated as positive satisfaction levels, while quantitative responses to paradoxical questions about the present are treated as negative satisfaction levels, and these responses are stored in a response database. It is also preferable to make the computer function in this way.

[0011] According to another embodiment of the first program of the present invention, The question generation method sends a prompt to the generation AI server requesting that the generated question be a qualitative answer. The response collection method receives qualitative answers to questions about the future and qualitative answers to questions about the present from each user terminal. A qualitative answer prompt for a question about the future and a qualitative answer prompt for a question about the present are sent to a generative AI server to obtain a quantitative answer prompt for a question about the future and a quantitative answer prompt for a question about the present, and a qualitative answer estimation means for receiving, from the generative AI server, a quantitative answer for a question about the future and a quantitative answer for a question about the present It is also preferable to cause the computer to function.

[0012] According to another embodiment of the first program of the present invention, A qualitative answer is a user's text, Based on sentiment analysis of the text, the qualitative answer estimation means sends a prompt for obtaining a quantitative answer for a question about the future and a quantitative answer for a question about the present to the generative AI server It is also preferable to cause the computer to function.

[0013] According to another embodiment of the first program of the present invention, A prompt for instructing the creation of an evaluation comment from the expected degree and satisfaction degree of the answer database is sent to the generative AI server for each evaluation item or for each user, and an evaluation comment for the evaluation item or the user is further received from the generative AI server, and an evaluation comment acquisition means for displaying the evaluation comment on the administrator terminal It is also preferable to cause the computer to function.

[0014] According to another embodiment of the first program of the present invention, A prompt for instructing the derivation of an evaluation index from the expected degree and satisfaction degree of the answer database is sent to the generative AI server for each evaluation item or for each user, and an evaluation index for the evaluation item or the user is received from the generative AI server, and an evaluation index acquisition means for displaying the evaluation index on the administrator terminal It is also preferable to cause the computer to function.

[0015] According to the first aspect of the present invention, in a management device that collects questionnaires from users based on evaluation items, For each evaluation item, a prompt for instructing the creation of questions about the future and questions about the present is sent to a generation AI (Artificial Intelligence) server, and a question creation means for receiving questions about the future and questions about the present from the generation AI server; Questions about the future and questions about the present are each sent to each user terminal, and an answer for each question is received from the terminal. For each user, the answer to the question about the future is used as the degree of expectation, and the answer to the question about the present is used as the degree of satisfaction, and an answer collection means for accumulating the answers in an answer database characterized by comprising.

[0016] According to the first aspect of the present invention, in a management method of a device that collects questionnaires from users based on evaluation items, The device is For each evaluation item, a first step of sending a prompt for instructing the creation of questions about the future and questions about the present to a generation AI (Artificial Intelligence) server and receiving questions about the future and questions about the present from the generation AI server; A second step of sending questions about the future and questions about the present to each user terminal, receiving an answer for each question from the terminal, and accumulating, for each user, the answer to the question about the future as the degree of expectation and the answer to the question about the present as the degree of satisfaction in an answer database characterized by executing.

[0017] <Second aspect of the present invention> According to the second aspect of the present invention, in a program that causes a computer to function so as to derive an evaluation index from a predetermined range of answer values from users for a questionnaire based on evaluation items, A training list is created in which a row with the same answer values for all evaluation items is composed of a plurality of rows with different answer values, and an evaluation index is set for the answer value for each row. In the training phase, the model is trained by associating the response values ​​of all evaluation items with evaluation indices for each row of the training list, and in the estimation phase, the machine learning model is trained by inputting the response values ​​of all evaluation items that constitute the target data and estimating the evaluation indices. It is characterized by enabling the computer to function in this way.

[0018] According to another embodiment of the second program of the present invention, The training list is, For the central row, assign a median to all response values ​​for the evaluation items, and map the median to the evaluation index. For the highest-ranking row, assign the highest value to all response values ​​in the evaluation items, and associate the highest value with the evaluation index. For the lowest-ranking row, assign the lowest value to all response values ​​in the evaluation items, and associate the lowest value with the evaluation index. Starting from the central row to the highest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is incremented to create a correspondence. Starting from the middle row to the lowest-ranking row, all response values ​​for the evaluation items are reduced, and the evaluation index is reduced and matched accordingly. It is also preferable to make the computer function in this way.

[0019] According to another embodiment of the second program of the present invention, The response values ​​are pairs of expectation and satisfaction levels based on responses received from each user. The median of the response values ​​is the value where expectations and satisfaction levels match. The highest response value is the value where the expectation level is lowest and the satisfaction level is highest. The lowest response value represents the value where expectation is highest and satisfaction is lowest. It is also preferable to make the computer function in this way.

[0020] According to another embodiment of the second program of the present invention, The response value is calculated by dividing the satisfaction level, based on the responses received from each user, by the expected level. The median is 1.0. The evaluation index is a normalized value that falls within a predetermined range above and below the median. It is also preferable to make the computer function in this way.

[0021] According to another embodiment of the second program of the present invention, The training list is, For the central row, assign a median to all response values ​​for the evaluation items, and map the median to the evaluation index. For the highest-ranking row, assign the highest value to all response values ​​for the evaluation items, and associate the lowest value with the evaluation index. For the lowest-ranking row, assign the lowest value to all response values ​​for the evaluation items, and associate the highest value with the evaluation index. Starting from the central row to the highest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is decremented to create a correspondence. From the middle row to the lowest-ranking row, all response values ​​for the evaluation items are decreased, and the evaluation index is increased to create a corresponding relationship. It is also preferable to make the computer function in this way.

[0022] According to another embodiment of the second program of the present invention, The response values ​​are pairs of expectation and satisfaction levels based on responses received from each user. The median of the response values ​​is the value where expectations and satisfaction levels match. The highest response value is the value where the expectation level is highest and the satisfaction level is lowest. The lowest response value represents the lowest level of expectation and the highest level of satisfaction. It is also preferable to make the computer function in this way.

[0023] According to another embodiment of the second program of the present invention, The response value is calculated by dividing the expected value based on the responses received from each user by the satisfaction level. The median is 1.0. The evaluation index is a normalized value that falls within a predetermined range above and below the median. It is also preferable to make the computer function in this way.

[0024] According to another embodiment of the second program of the present invention, A machine learning model is a library or generative AI server that applies the decision tree algorithm. It is also preferable to have the computer function in this way.

[0025] <The Third Invention> According to the third aspect of the present invention, in a program that operates a computer that stores qualitative textual responses to questions in a user questionnaire, The questions were a mix of questions about the future and questions about the present. A qualitative response estimation method that uses a sentiment analysis model to estimate sentiment scores for texts, estimates higher evaluation indices for more positive sentiment scores, and estimates lower evaluation indices for more negative sentiment scores. It is characterized by enabling the computer to function in this way.

[0026] <Fourth Invention> According to the fourth aspect of the present invention, in a program that operates a computer that stores qualitative textual responses to questions in a user survey, The questions were a mix of questions about the future and questions about the present. A positive / negative classification method that uses a sentiment analysis model to classify texts as positive or negative, A noun extraction method that uses a morphological analysis model to extract nouns from text, A positive / negative counting method that counts the number of sentences containing each noun that are classified as positive, and the number of sentences that are classified as negative, A noun identification display means that visually highlights and displays nouns among multiple nouns that have a higher number of negatives than a predetermined amount relative to the number of positives. It is characterized by enabling the computer to function in this way. [Effects of the Invention]

[0027] According to the program, apparatus, and method of the present invention, firstly, by asking a variety of questions to the users being surveyed, it is possible to obtain evaluable expectations and satisfaction levels. [Brief explanation of the drawing]

[0028] [Figure 1] This is a functional configuration diagram of the control device according to the first invention. [Figure 2] This is a diagram illustrating the user attribute database. [Figure 3] This is an explanatory diagram of the answer database. [Figure 4] This is the first explanatory diagram for the question creation section. [Figure 5] These are user-specific questions created by the question creation team. [Figure 6] This is the second explanatory diagram for the question creation section. [Figure 7] This is an explanatory diagram of the qualitative response estimation unit. [Figure 8] This is a diagram illustrating the optional functional configuration of the control device in the first part of the present invention. [Figure 9] This is an explanatory diagram of the target user extraction unit, the item-specific response calculation unit, and the statistical database. [Figure 10] This is an explanatory diagram of the evaluation comment acquisition section. [Figure 11] This is an illustrative diagram showing the sub-items used as evaluation criteria. [Figure 12] This is an explanatory diagram of the unit for acquiring the evaluation index for the first training stage. [Figure 13] This is the first explanatory diagram of the evaluation index acquisition unit for the first estimation stage. [Figure 14] This is the second explanatory diagram of the evaluation index acquisition unit for the first estimation stage. [Figure 15] This is an explanatory diagram of the evaluation index acquisition unit for the second training stage. [Figure 16] This is an explanatory diagram of the evaluation index acquisition unit for the second estimation stage. [Figure 17] This is a functional configuration diagram of the control device in the third invention. [Figure 18] This is a functional configuration diagram of the control device according to the fourth invention. [Figure 19] This is an explanatory diagram of the positive / negative counting section and the noun identification display section. [Modes for carrying out the invention]

[0029] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0030] Figure 1 is a functional configuration diagram of the control device in the first invention.

[0031] According to Figure 1, the management device 1 collects questionnaires from users based on evaluation items. It then displays the questionnaire response results from multiple users to the administrator. In recent years, "engagement," which refers to "trust" and "favorability" towards an organization or service, has become increasingly important. If the organization is a company, for example, the managers would be executives and the users would be employees. If the service is a financial service, for example, the managers would be customer managers and the users would be customer users. For example, managers can identify areas for improvement within the company based on the results of employee surveys. Similarly, customer service managers can identify areas for improvement in their services based on the results of customer surveys. As a manager or customer service manager, it is desirable to operate in a way that ensures high levels of engagement among the members of the organization or service.

[0032] Let's consider a system where company managers evaluate employee engagement as an example. Managers are required to maintain a high level of employee engagement so that employees can work together as a team. However, even if managers improve the "work environment" for their employees, in reality, employees may prioritize "evaluation and compensation." In this case, improving the work environment becomes irrelevant to the employee. Furthermore, this is a problem that cannot be directly measured, as it varies depending on the employee's organizational attributes (department, position, work location, years of service) and personal attributes (gender, age, education level, family structure).

[0033] In contrast, there is a system that collects responses to survey questions based on members' "expectations" and "satisfaction levels." This system asks members to rate their expectations and satisfaction levels on a scale, for example, from 1 to 5. By analyzing member engagement-based expectations and satisfaction levels, it becomes possible to identify areas for improvement within the organization or its services. In particular, the management device 1 of the present invention can collect expectations and satisfaction levels by asking a variety of questions to users who are the target of the survey.

[0034] Management device 1 has a web interface that acts as a server for a website. Management device 1 also communicates with a generation AI server 3 that creates questions and a user terminal 4 that sends these questions to users for them to answer. Administrator terminal 2 accesses management device 1 from a pre-installed browser or application. User terminal 4 also accesses management device 1 from a pre-installed browser or application.

[0035] <First Invention> As shown in Figure 1, the management device 1 of the present invention includes a user attribute database 101, a response database 102, a question creation unit 11, a response collection unit 12, and a qualitative response estimation unit 13. These functional components may be realized by executing a program that makes the computer installed in the device function. Furthermore, the processing flow of these functional components can also be understood as a question creation method.

[0036] [Generating AI Server 3] Generative Artificial Intelligence (Generative AI) Server 3 is a system that understands the context of a given question and generates natural language text. Examples include OpenAI's ChatGPT (registered trademark), Anthropic's Claude (registered trademark), Google's Gemini (registered trademark), and Meta's Llama (registered trademark). These are pre-trained using general linguistic knowledge from large text corpora. Furthermore, Generative AI Server 3 can improve the accuracy of its texts through repeated fine-tuning.

[0037] [User Attribute Database 101] Figure 2 is an explanatory diagram of the user attribute database.

[0038] The user attribute database 101 stores attribute information for each user. For example, in the case of an organization such as a company, there are organizational attributes and / or individual attributes. According to Figure 6, for each user ID (Identifier), organizational attributes such as "department," "position," "work location," and "years of service" are recorded, and personal attributes such as "gender," "age," "educational background," and "family structure" are recorded. Examples of departments include sales, development, and general affairs. Examples of positions include department head, section chief, group leader, no position, and performance evaluation rank.

[0039] [Question Creation Section 11] The question creation unit 11 sends prompts to the generation AI server 3 instructing it to create "questions about the future" and "questions about the present" for each evaluation item. In response, the generation AI server 3 sends the "questions about the future" and "questions about the present" and stores them in the answer database 102.

[0040] Figure 3 is an explanatory diagram of the response database.

[0041] As shown in Figure 3, the response database 102 categorizes employee engagement in a company, for example, into major, medium, and minor items in a hierarchical structure. These items are survey items for multiple users belonging to an organization or service, arranged in a hierarchical manner. In the embodiment of the present invention, an example is described in which the "evaluation item" is a medium item, but of course, it may also be a major item or a minor item.

[0042] According to Figure 3, assuming an organization such as a company, the major categories are divided into three, for example, "Job Content / Environment," "Human Relations," and "Organizational Policies / Culture." Each major category is further divided into four subcategories, for a total of 12 categories. Each subcategory is further divided into five subcategories.

[0043] The administrator can edit item categories and item names as they see fit from administrator terminal 2. In other words, the administrator can decide at their discretion which aspects to focus on in order to improve the evaluation items. The content of the questions can also be changed as needed in conjunction with changes to the content and number of major, medium, and minor items. Figure 3 shows an example of items intended for company employees. Of course, it could also be an example of items intended for user members of a financial service. The administrator can edit the item categories and names as needed.

[0044] The response database 102 records the level of expectation and satisfaction for each evaluation item (e.g., sub-items) for each user. For example, based on major and minor categories, multiple questions are presented to the user regarding sub-categories, and the answers to those questions are collected. Then, in the response database 102, the expectation level and satisfaction level are derived and recorded from the collected responses for each evaluation item and / or sub-item. The expectation level and satisfaction level may be expressed as numerical values, for example, from 1 to 5. Furthermore, for example, regarding the major category "Job Content and Environment," the subcategory "Job Characteristics," and the minor category "Clarity of Roles and Responsibilities," multiple questions may be asked, and multiple answers may be obtained. Expectations and satisfaction levels can then be derived from these answers.

[0045] Figure 4 is the first explanatory diagram of the question creation section.

[0046] The question creation unit 11 creates prompts that instruct the creation of "questions about the future" and "questions about the present" for each combination of major items, evaluation items (medium items), and minor items in the answer database 102.

[0047] At this time, the question creation unit 11 includes attribute information of the organization or service in the prompt in order to create a question based on that organization or service. This makes it possible to create a question that is appropriate for that organization or service. Attribute information may include, for example, a company overview or IR (Investor Relations) information.

[0048] Furthermore, the question creation unit 11 may include, or a combination thereof, the user's user attribute information {user_profile}, the user's past answers {result}, or the user's past expectations and satisfaction levels {result} in the prompt in order to create each question for each user. This makes it possible to create questions that are appropriate for each user.

[0049] <Quantitative answer> The question creation unit 11 may also send a prompt to the generation AI server 3 requesting that the question to be created be a "quantitative answer." A quantitative answer may be one of several options (e.g., 1, 2, 3, 4, 5) for the question, or a numerical value entered from a predetermined range (e.g., 1.0 to 5.0). The numerical value may be entered by the user directly, or it may be specified by shifting a slider bar.

[0050] According to Figure 4, for example, the following prompt is sent to the generation AI server 3. -------------------------------------------------------------------------------- {company_profile} Based on {user_profile}, Based on the major category "Job Content and Environment," the subcategory "Job Characteristics," and the minor category "Clarity of Roles and Responsibilities," please create "Questions about the Future" and "Questions about the Present" for employees. Please ensure each question has a quantitative answer (1-5). -------------------------------------------------------------------------------- {company_profile}: Organization attribute information {user_profile}: User attribute information In response, the question creation unit 11 receives "questions about the future" and "questions about the present" from the generation AI server 3.

[0051] According to Figure 4, the following questions are generated by the AI ​​server 3. -------------------------------------------------------------------------------- (Questions about the future) "Do you want your role and responsibilities to become clearer?" Please answer using the scale from 1: Not desired to 5: Desired. (Questions about the present) "Do you feel that your role and responsibilities are clear?" Please answer using the scale from 1: Don't feel it to 5: Feel it. --------------------------------------------------------------------------------

[0052] Figure 5 shows the user-specific questions created by the question generation unit.

[0053] As shown in Figure 5, for each user, "questions about the future" and "questions about the present" corresponding to each sub-item are created. To obtain detailed answers, multiple "questions about the future" and multiple "questions about the present" may be mixed together.

[0054] <Qualitative answer> Figure 6 is the second explanatory diagram of the question creation section.

[0055] The question generation unit 11 sends a prompt to the generation AI server 3 requesting that the question to be generated be a "qualitative answer." A qualitative answer is, for example, a sentence entered by the user. According to Figure 6, for example, the following prompt is sent to the generation AI server 3. -------------------------------------------------------------------------------- {company_profile} Based on {user_profile}, Based on the major category "Job Content and Environment," the subcategory "Job Characteristics," and the minor category "Clarity of Roles and Responsibilities," please create "Questions about the Future" and "Questions about the Present" for employees. Please ensure your questions are in the form of qualitative answers (written responses). -------------------------------------------------------------------------------- In response, the question creation unit 11 receives a question that will be a qualitative answer from the generation AI server 3.

[0056] According to Figure 6, the following questions are generated by the AI ​​server 3. -------------------------------------------------------------------------------- (qualitative question) Please freely describe your future role and responsibilities, as well as your current role and responsibilities, below. --------------------------------------------------------------------------------

[0057] The question creation unit 11 outputs the sub-item and user-specific questions received from the generation AI server 3 to the answer collection unit 12.

[0058] [Response Collection Section 12] The response collection unit 12 sends "questions about the future" and "questions about the present" to each user terminal 4, and receives answers to each question from the user terminal 4. The questions may be quantitative or qualitative. The response collection unit 12 stores, for each user, the responses to questions about the future as "expectations" and the responses to questions about the present as "satisfaction levels" in the response database 102. Here, when the response collection unit 12 receives qualitative answers to questions about the future and qualitative answers to questions about the present from each user terminal 4, it outputs those qualitative answers to the qualitative response estimation unit 13. Furthermore, the text that constitutes a qualitative response may be text entered by the user themselves into user terminal 4, or text recognized by voice input via the microphone.

[0059] [Qualitative answer estimation part 13] Figure 7 is an explanatory diagram of the qualitative response estimation unit.

[0060] The qualitative response estimation unit 13 sends prompts to the generation AI server 3 to request "quantitative answers to questions about the future" and "quantitative answers to questions about the present" from "qualitative answers to questions about the future" and "qualitative answers to questions about the present". In this case, the prompt may request quantitative answers to questions about the future and quantitative answers to questions about the present, based on a sentiment analysis of the qualitative response text.

[0061] Here, the generating AI server 3 performs text mining on the text, dividing it into words and phrases, and analyzing their frequency of occurrence, co-occurrence correlations, occurrence trends, and time series. Based on this, it performs sentiment analysis on "descriptions in response to questions about the future" and "descriptions in response to questions about the present," and derives quantitative answers for each.

[0062] According to Figure 5, for example, the following prompt is sent to the generation AI server 3. -------------------------------------------------------------------------------- Based on {company_profile}, For the major category "Job Content and Environment," the subcategory "Job Characteristics," and the minor category "Clarity of Roles and Responsibilities," please estimate the level of expectation and satisfaction based on the sentiment analysis of the following qualitative responses (text). <Qualitative answer> --------------------------------------------------------------------------------

[0063] In response, the qualitative response estimation unit 13 receives from the generating AI server 3 a quantitative response (for example, one of 1 to 5) to a question about the future, and a quantitative response to a question about the present.

[0064] <Collection of answers to questions that present a logical argument and questions that present a counter-argument> (S1) The question creation unit 11 sends prompts to the generation AI server instructing it to create a forward-sounding question about the future, a backward-sounding question about the future, a forward-sounding question about the present, and a backward-sounding question about the present. (S2) The AI ​​server 3 receives forward-sounding questions about the future, backward-sounding questions about the future, forward-sounding questions about the present, and backward-sounding questions about the present. (S3) The response collection unit 12 stores the responses in the response database such that quantitative responses to forward-looking questions about the future are treated as positive expectations, quantitative responses to paradoxical questions about the future are treated as negative expectations, quantitative responses to forward-looking questions about the present are treated as positive satisfaction levels, and quantitative responses to paradoxical questions about the present are treated as negative satisfaction levels.

[0065] <Regarding the relationship between questions and responses to expectations and satisfaction levels> According to embodiments of the present invention, the user is not directly asked questions in which expectations or satisfaction levels are defined. In other words, the answers are not based on answers directly obtained through questions such as the following. Expectation Level: "What are your expectations regarding ~?" (Answer on a scale of 1 to 5) Satisfaction level: "Are you satisfied with ~?" (Answer with one of the following: 1-5) In contrast, according to the embodiment of the present invention, questions are created by sending prompts to the generating AI server 3 to create questions for the user (member). Furthermore, the expectation level and satisfaction level are obtained from the answers to these questions using a machine learning model and the generating AI server 3. This makes it possible to derive the expectation level and satisfaction level even when diverse answers are obtained from diverse questions.

[0066] Figure 8 is an optional functional configuration diagram of the control device in the first aspect of the present invention.

[0067] According to Figure 8, in addition to the functional components shown in Figure 1, the system further includes a target user extraction unit 14, an item-specific response calculation unit 15, a statistical database 103, an evaluation index acquisition unit 17, a machine learning model, and an evaluation comment acquisition unit 16. These functional components may be implemented by executing a program that enables the operation of a computer installed in the device. Furthermore, the processing flow of these functional components can also be understood as an evaluation acquisition method.

[0068] Figure 9 is an explanatory diagram of the target user extraction unit, the item-specific response calculation unit, and the statistical database.

[0069] [Target User Extraction Unit 14] The target user extraction unit 14 refers to the user attribute database 101 and extracts only the expected and satisfaction levels derived from target users who have the "attribute information" specified by the administrator. For example, the target users could be defined as "all employees." This would allow for the selection of improvement items for all employees in the company. Alternatively, the target users could be filtered to include, for example, "only employees with less than 3 years of service," or "only employees with the position of section manager." The extracted target users (e.g., IDs) are output to the item-specific answer calculation unit 15.

[0070] [Item-specific answer calculation section 15] The item-specific response calculation unit 15 calculates the expected level and satisfaction level for each item from the expected level and satisfaction level of multiple target users extracted by the target user extraction unit 14. As shown in Figure 9, a data set is created for only the target users, which will be stored in the statistical database 103. This makes it possible to derive only the expected level and satisfaction level for multiple evaluation items at the hierarchy specified by the administrator.

[0071] [Statistical Database 103] The statistical database 103, like the response database 102, is divided hierarchically into major, medium, and minor categories. It then records the expected and satisfaction levels derived by the item-specific response calculation unit 15 for each item.

[0072] According to Figure 9, the expectations and satisfaction levels in the statistical database 103 are based on responses to questions related to the organization or service. In other words, the expectations and satisfaction levels are statistical values ​​calculated from responses collected from multiple users. Specifically, for example, the expectation and satisfaction levels for sub-items are statistical values ​​such as the average or median of expectation and satisfaction levels collected from multiple users. Furthermore, the expected value and satisfaction level of a sub-item may be statistical values ​​such as the average and median expected value and satisfaction level of all sub-items belonging to that sub-item, respectively. Similarly, the expectations and satisfaction levels for major categories may be statistical values ​​such as the average and median of the expectations and satisfaction levels for all subcategories belonging to that major category.

[0073] [Evaluation Comment Acquisition Section 16] Figure 10 is an explanatory diagram of the evaluation comment acquisition unit.

[0074] The evaluation comment acquisition unit 16 sends a prompt to the generation AI server 3 instructing it to create an "evaluation comment" based on the expected value and satisfaction level from the statistical database 103, for each evaluation item or each user. The evaluation comment acquisition unit 16 then receives the "evaluation comment" for the evaluation item or user from the generation AI server 3 and displays the evaluation comment on the administrator terminal 2.

[0075] According to Figure 10, for example, the following prompt is sent to the generating AI server 3 for the entire company. -------------------------------------------------------------------------------- Based on {company_profile}, Main category: "Job content and environment" Sub-item: "Characteristics of the job" Clarity of roles and responsibilities: <Expectations> <Satisfaction> Meaning of the job: <Expectations><Satisfaction> Skill utilization: <Expectation level> <Satisfaction level> ····························· Medium item “Work environment” Comfort of the work environment: <Expectation level><Satisfaction level> Work-life balance: <Expectation level> <Satisfaction level> ····························· ····························· Main category: "Organizational policies and culture" Sub-item: "Vision and Mission" ····························· Please create your review comment. --------------------------------------------------------------------------------

[0076] In response, the evaluation comment acquisition unit 16 receives evaluation comments from the generating AI server 3 that provide advice about the organization of the company. By prompting the AI ​​to point out "good points" and "bad points," the evaluation comments can also receive text and images that describe the "good points" and "bad points" regarding the "characteristics of the work." The evaluation comments are presented to the administrator as AI insights to help them understand the current situation and encourage improvement.

[0077] [Evaluation Index Acquisition Unit 17] The evaluation index acquisition unit 17, like the evaluation comment acquisition unit 16, sends a prompt to the generation AI server 3 instructing it to create an "evaluation index" based on the expected value and satisfaction level from the statistical database 103, for each evaluation item or each user. The evaluation index acquisition unit 17 then receives the "evaluation index" for the evaluation item or user from the generation AI server 3 and displays the evaluation index on the administrator terminal 2. The evaluation index is a value that comprehensively assesses expectations and satisfaction, and can be used to determine whether the organization or service as a whole is in an ideal state for the user.

[0078] The evaluation index is mapped as follows based on expectations and satisfaction levels. High Expectations and High Satisfaction: Users have high expectations of the organization or service and are actually highly satisfied (an ideal positive item) = the evaluation index will be close to the median. High expectations, low satisfaction: Users have high expectations for the organization or service, but are dissatisfied due to a lack of those expectations (the area that needs the most improvement) = a high rating index. Low expectations, high satisfaction: Users had low expectations for the organization or service, but were more satisfied than expected (unnecessarily positive items) = low rating index. Low expectations / low satisfaction: Users have no expectations of the organization or service, and are not satisfied (items that no one pays attention to) = the evaluation index will be close to the median. According to these views, it is preferable that the plots for all evaluation items be as close as possible to the median (e.g., 100%).

[0079] Figure 11 is an illustrative diagram showing the sub-items used as evaluation criteria.

[0080] According to Figure 11, the following items are displayed hierarchically. Main category [Job content and environment] ->Subcategory [Job Characteristics] ->Sub-item [Clarity of roles and responsibilities] [The meaning of work] [Utilizing skills] [Opportunities for growth] Here, when an administrator selects a sub-item, multiple sub-items belonging to that sub-item are displayed. Each sub-item and sub-item displays "Expectation Level," "Satisfaction Level," and "Evaluation Index (Value Index)." The evaluation index displayed to the administrator does not include the difference between expectation level and satisfaction level; instead, it displays a normalized value within a predetermined range (e.g., 20% to 180%). A lower evaluation index indicates that the evaluation item needs improvement. Conversely, a higher evaluation index indicates that the evaluation item is unnecessarily extensive.

[0081] <The Second Invention> The evaluation index acquisition unit 17 can acquire the optimal evaluation index by training a machine learning model or a generating AI server 3. The evaluation index acquisition unit 17 derives the evaluation index from "a predetermined range of response values" from the user regarding a questionnaire based on evaluation items.

[0082] Figure 12 is an explanatory diagram of the evaluation index acquisition unit for the first training stage.

[0083] The evaluation index acquisition unit 17 may have a machine learning model. An external generation AI server 3 may be used as the machine learning model. A machine learning model is a library or generative AI server that applies the decision tree algorithm. Specifically, for example, XGBoost (eXtreme Gradient Boosting) can be used. This implements the Gradient Boosting Decision Tree algorithm. It works by creating decision trees one by one, weighting the incorrect data in the majority vote of the previous decision trees, and then training the next decision tree to compensate for those errors, thereby avoiding overfitting.

[0084] Figure 12 illustrates an example of deriving an evaluation index for a medium-sized item from the response values ​​of multiple sub-items. The "response value" will be a "pair of expectation and satisfaction" for each sub-item, based on the responses received from each user. First, create a training list. The training list consists of multiple rows, each with a different response value, where the same response value is listed for all evaluation items. An evaluation index is assigned to each response value in each row.

[0085] And the machine learning model works as follows: (Training phase) For each row in the training list, the response values ​​for all evaluation items are associated with the evaluation index during training. (Estimation stage) Input the response values ​​for all evaluation items that constitute the target data, and estimate the evaluation index.

[0086] According to Figure 12, rows are arranged for all combinations of expectation levels (5-1) and satisfaction levels (1-5). Here, the responses can be classified into two patterns based on the composition of the expectation and satisfaction levels, as follows: <Pattern 1> The median of the response values ​​is the value where expectations and satisfaction levels coincide. The highest possible response value is the value that represents both the lowest level of expectation (e.g., expectation level 1) and the highest level of satisfaction (e.g., satisfaction level 5). The lowest possible response value is the value that represents both the highest level of expectation (e.g., expectation level 5) and the lowest level of satisfaction (e.g., satisfaction level 1). • The evaluation index will be defined as "satisfaction level / expectation level". The evaluation index is a normalized value with a median of 1.0, so that the values ​​fall within a predetermined range above and below the median.

[0087] The training list is set up as follows: For the central row, assign a median (expected level = satisfaction level) to all response values ​​for the evaluation items, and associate the median (1.00) with the evaluation index. For the highest-ranking row, assign the highest value (expectation level 1, satisfaction level 5) to all response values ​​in the evaluation items, and associate the evaluation index with "highest value (satisfaction level / expectation level = 5.00 -> 180%)". For the lowest-ranking row, assign the lowest value (expectation level 5, satisfaction level 1) to all response values ​​in the evaluation items, and associate the evaluation index with "lowest value (satisfaction level / expectation level = 0.20 -> 20%)". From the central row to the highest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is "incremented" to create a corresponding relationship. • Starting from the middle row and moving towards the lowest-ranking row, all response values ​​for the evaluation items are reduced, and the evaluation index is "reduced" to match them.

[0088] According to Figure 12, for all sub-items, an evaluation index of 120% based on the evaluation index (satisfaction / expectation = 2.00) is associated with expectation level 1 and satisfaction level 2. Furthermore, for all sub-items, an evaluation index of 80% is associated with an expectation level of 5 and a satisfaction level of 4, based on an evaluation index (satisfaction level / expectation level = 0.80).

[0089] <Pattern 2> The median of the response values ​​is the value where expectations and satisfaction levels coincide. The highest possible response value is the value that represents both the highest level of expectation (e.g., expectation level 5) and the lowest level of satisfaction (e.g., satisfaction level 1). The lowest possible response value is the value that represents both the lowest level of expectation (e.g., expectation level 1) and the highest level of satisfaction (e.g., satisfaction level 5). • The evaluation index will be defined as "Expectation level / Satisfaction level". The evaluation index is a normalized value with a median of 1.0, so that the values ​​fall within a predetermined range above and below the median.

[0090] The training list is set up as follows: For the central row, assign a median (expected level = satisfaction level) to all response values ​​for the evaluation items, and associate the median (1.00) with the evaluation index. For the highest-ranking row, assign the highest value (expectation level 5, satisfaction level 1) to all response values ​​for the evaluation items, and associate the lowest value (expectation level / satisfaction level = 5.00 -> 20%) with the evaluation index. For the lowest-ranking row, assign the lowest value (expectation level 1, satisfaction level 5) to all response values ​​for the evaluation items, and associate the highest value (expectation level / satisfaction level = 0.20 -> 180%) with the evaluation index. From the central row to the highest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is "decremented" to create a corresponding correspondence. From the middle row to the lowest-ranking row, all response values ​​for the evaluation items are decreased, and the evaluation index is "incremented" to create a corresponding relationship.

[0091] According to Figure 12, for all sub-items, an evaluation index of 120% based on the evaluation index (expectation / satisfaction = 0.50) is associated with expectation level 1 and satisfaction level 2. Furthermore, for all sub-items, an evaluation index of 80% is associated with an expectation level of 5 and a satisfaction level of 4, based on an evaluation index (expectation level / satisfaction level = 1.25).

[0092] Figure 13 is the first explanatory diagram of the evaluation index acquisition unit in the first estimation stage. As shown in Figure 13, the response values ​​(expectations and satisfaction levels) for multiple sub-items based on the evaluation items of the target data item, "Job Characteristics," are input, and the evaluation index for that item, "Job Characteristics," is estimated.

[0093] Figure 14 is a second explanatory diagram of the evaluation index acquisition unit in the first estimation stage. As shown in Figure 14, the response values ​​(expectations and satisfaction levels) for multiple sub-items based on company-wide data are input, and an overall company-wide evaluation index is estimated.

[0094] Figure 15 is an explanatory diagram of the evaluation index acquisition unit for the second training stage.

[0095] According to Figure 15, rows for all combinations of response values ​​from 5.0 to 0.2 are placed. The training list is set up as follows: <Pattern 1> • Response value = Satisfaction level / Expectation level For the central row, assign the median (1.0) to all response values ​​for the evaluation items, and associate the median (100%) with the evaluation index. For the highest-ranking row, assign the maximum value (e.g., satisfaction level 5 / expectation level 1 = 5.00) to all response values ​​in the evaluation items, and associate the evaluation index with "maximum value (180%)". For the lowest-ranking row, assign a minimum value (e.g., satisfaction level 1 / expectation level 5 = 0.20) to all response values ​​in the evaluation items, and associate the evaluation index with "minimum value (20%)". From the central row to the highest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is "incremented" to create a corresponding relationship. • Starting from the middle row and moving towards the lowest-ranking row, all response values ​​for the evaluation items are reduced, and the evaluation index is "reduced" to match them.

[0096] <Pattern 2> • Response value = Expectation level / Satisfaction level For the central row, assign the median (1.0) to all response values ​​for the evaluation items, and associate the median (100%) with the evaluation index. For the highest-ranking row, assign the minimum value (e.g., expectation 1 / satisfaction 5 = 0.20) to all response values ​​for the evaluation items, and associate the evaluation index with "highest value (180%)". For the lowest-ranking row, assign the highest possible value (e.g., expectation level 5 / satisfaction level 1 = 5.00) to all response values ​​in the evaluation items, and associate the "lowest value (20%)" with the evaluation index. From the central row to the highest-ranking row, all response values ​​for the evaluation items are decremented, and the evaluation index is "incremented" to create a corresponding correspondence. • Starting from the middle row and moving towards the lowest-ranking row, all response values ​​for the evaluation items are incremented, and the evaluation index is "decremented" to create a corresponding correspondence.

[0097] Figure 16 is an explanatory diagram of the evaluation index acquisition unit in the second estimation stage. As shown in Figure 16, the response values ​​for multiple sub-items (satisfaction level / expectation level: pattern 1) based on company-wide data are input, and the overall company-wide evaluation index is estimated.

[0098] In other words, according to the embodiment of the present invention, the evaluation index is not calculated based on the difference between expectations and satisfaction. Nor is it simply calculated for each evaluation item, where a small difference between expectations and satisfaction results in a high evaluation index.

[0099] <The Third Invention> Figure 17 is a functional configuration diagram of the control device in the third aspect of the present invention.

[0100] According to Figure 17, the management device 1 includes a response collection unit 12 and a qualitative response estimation unit 13. These functional components are realized by executing a program that enables the computer installed in the management device 1 to function. Furthermore, the processing flow of these functional components can also be understood as an estimation method.

[0101] The response collection unit 13 sends a questionnaire to the user terminal 4, which includes a mix of questions about the future and questions about the present. In response, the user terminal 4 receives qualitative answers to these questions. The received text is output to the qualitative response estimation unit 13.

[0102] The qualitative response estimation unit 13 estimates the sentiment score for the text using the sentiment analysis model 5. Specifically, TextBlob or NTLK can be used. The output sentiment score is such that a positive score (0 to +1) indicates a positive emotion, and a negative score (-1 to 0) indicates a negative emotion. The qualitative response estimation unit 13 then estimates a higher evaluation index the more positive the sentiment score is, and a lower evaluation index the more negative the sentiment score is. Here, we note that the questions posed to the user are a mixture of questions about the future and questions about the present. If we define the answer to the question about the future as the expectation level and the answer to the question about the present as the satisfaction level, then a positive response in a mixture of these indicates that the expectation level and satisfaction level are close in value, and therefore the evaluation index is high. On the other hand, a negative response in a mixture of these indicates that the expectation level and satisfaction level are far apart, and therefore the evaluation index is low.

[0103] <Fourth Invention> Figure 18 is a functional configuration diagram of the control device according to the fourth aspect of the present invention.

[0104] According to Figure 18, the management device 1 includes a response collection unit 12, a positive / negative classification unit 181, a noun extraction unit 182, a positive / negative count unit 183, and a noun identification display unit 184. These functional components are realized by executing a program that makes the computer installed in the management device 1 function. Furthermore, the processing flow of these functional components can also be understood in terms of a display method.

[0105] The response collection unit 13, as described in Figure 17 above, sends a questionnaire to the user terminal 4 containing a mix of questions about the future and questions about the present. In response, the user terminal 4 sends qualitative answers to these questions in text form. The received text is output to the positive / negative classification unit 181.

[0106] The positive / negative classification unit 181 uses the sentiment analysis model 5 to classify texts as either positive or negative. Specifically, it can use the aforementioned TextBlob or NTLK.

[0107] The noun extraction unit 182 extracts nouns from the text using the morphological analysis model 6.

[0108] Figure 19 is an explanatory diagram of the positive / negative counting unit and the noun identification display unit.

[0109] The positive / negative counting unit 183 counts, for each noun, the number of sentences containing that noun that are classified as positive, and the number of sentences that are classified as negative.

[0110] The noun identification unit 184 visually highlights and displays nouns among several nouns in which the number of negatives is greater than or equal to a predetermined condition relative to the number of positives. The predetermined condition may be, for example, a number in which the number of negatives is greater than the number of positives, or a number in which the number of negatives is greater than or equal to a predetermined ratio.

[0111] As shown in Figure 19, the noun identification display unit 184 displays multiple nouns in relation to each other on the administrator terminal 2, and nouns with a negative count exceeding a predetermined condition are highlighted. By visualizing the highlighted nouns along with the evaluation index for each evaluation item, the administrator can consider measures to improve the system.

[0112] As described in detail above, according to the program, apparatus, and method of the present invention, firstly, it is possible to obtain evaluable expectations and satisfaction levels by asking a variety of questions to the users being surveyed. Secondly, an appropriate evaluation index can be obtained from the user's response values ​​for each evaluation item. Furthermore, thirdly, an emotional evaluation index can be obtained from the qualitative responses provided by users. Furthermore, fourthly, it is possible to identify nouns that are strongly emphasized emotionally from the qualitative responses provided by users.

[0113] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]

[0114] 1 Management device 101 User Attribute Database 102 Answer Database 103 Statistical Databases 11 Question Creation Department 12. Response Collection Department 13 Qualitative answer estimation section 14. Target User Extraction Unit 15 Item-specific answer calculation section 16. Evaluation Comment Acquisition Section 17 Evaluation Index Acquisition Section 2 Administrator terminal 3. Generation AI Server 4. User terminals 5. Sentiment Analysis Models 6. Morphological Analysis Model

Claims

1. In a program that uses a computer to collect questionnaires from users based on evaluation criteria, A question creation means sends prompts to a generating AI (Artificial Intelligence) server instructing it to create questions about the future and questions about the present for each evaluation item, and receives the questions about the future and questions about the present from the generating AI server. A response collection method that sends questions about the future and questions about the present to each user terminal, receives answers to each question from the terminal, and stores the answers to the questions about the future as expectations and the answers to the questions about the present as satisfaction levels for each user in a response database. A program characterized by its ability to make a computer function.

2. The evaluation items are survey questions for multiple users belonging to an organization or service, structured in a hierarchical manner. The question creation mechanism includes attribute information of the organization or service in the prompt in order to create questions based on that organization or service. The program according to claim 1, characterized in that it causes the computer to function in such a way.

3. The question creation mechanism includes, in order to create a question for each user, either user attribute information and / or past answer information, or a combination thereof, in the prompt. The program according to claim 1, characterized in that it causes the computer to function in such a way.

4. The question generation mechanism sends a prompt to the generation AI server requesting that the generated question be a quantitative answer. The program according to claim 1, characterized in that it causes the computer to function in such a way.

5. The method for creating questions is, A prompt is sent to the generation AI server instructing it to create a forward-sounding question about the future, a paradoxical question about the future, a forward-sounding question about the present, and a paradoxical question about the present. The AI ​​server receives forward-sounding questions about the future, backward-sounding questions about the future, forward-sounding questions about the present, and backward-sounding questions about the present. The means of collecting responses is, Quantitative answers to forward-looking questions about the future are defined as positive expectations, and quantitative answers to paradoxical questions about the future are defined as negative expectations. Quantitative responses to forward-sounding questions about the present are treated as positive satisfaction levels, while quantitative responses to paradoxical questions about the present are treated as negative satisfaction levels, and these responses are stored in a response database. The program according to claim 4, characterized in that it causes the computer to function in such a way.

6. The question generation mechanism sends a prompt to the generation AI server requesting that the generated question be a qualitative answer. The response collection method receives qualitative answers to questions about the future and qualitative answers to questions about the present from each user terminal. A qualitative response estimation means that sends prompts to a generating AI server requesting quantitative answers to questions about the future and questions about the present, based on qualitative answers to questions about the future and qualitative answers to questions about the present, and receives quantitative answers to questions about the future and questions about the present from the generating AI server. The program according to claim 1, characterized in that it causes a computer to function.

7. A qualitative response is a user's written response. The qualitative response estimation means sends prompts to the generating AI server, based on sentiment analysis of the text, requesting quantitative answers to questions about the future and quantitative answers to questions about the present. The program according to claim 6, characterized in that it causes the computer to function in such a way.

8. A means for acquiring evaluation comments that sends a prompt to a generation AI server instructing it to create evaluation comments based on the expected and satisfaction levels in the response database for each evaluation item or each user, receives the evaluation comments for that evaluation item or user from the generation AI server, and displays the evaluation comments on the administrator terminal. The program according to claim 1, characterized in that it causes a computer to function.

9. The evaluation index acquisition means sends a prompt to the generating AI server instructing it to derive an evaluation index from the expected and satisfaction levels in the response database for each evaluation item or each user, receives the evaluation index for the evaluation item or user from the generating AI server, and displays the evaluation index on the administrator terminal. The program according to claim 1, characterized in that it causes a computer to function.

10. In a management device that collects questionnaires from users based on evaluation criteria, A question creation means sends prompts to a generating AI (Artificial Intelligence) server instructing it to create questions about the future and questions about the present for each evaluation item, and receives the questions about the future and questions about the present from the generating AI server. A response collection method that sends questions about the future and questions about the present to each user terminal, receives answers to each question from the terminal, and stores the answers to the questions about the future as expectations and the answers to the questions about the present as satisfaction levels for each user in a response database. A control device characterized by having the following features.

11. In a method for managing a device that collects questionnaires from users based on evaluation criteria, The device is The first step involves sending prompts to a generation AI (Artificial Intelligence) server instructing it to create questions about the future and questions about the present for each evaluation item, and receiving the questions about the future and questions about the present from the generation AI server. The second step involves sending questions about the future and questions about the present to each user terminal, receiving answers for each question from the terminal, and for each user, storing the answer to the question about the future as the expectation level and the answer to the question about the present as the satisfaction level in the answer database. A management method characterized by performing the following actions.

Images